Did you stop development of this code?

Well, it works and there was no pressing reason to keep changing it, so there hasn't been any new development for a while.

HOWEVER, we recently became aware of a problem with the way TigerTree values are calculated which will require a change. The proper fixes are still being evaluated -- it will probably only require a line or two of added code, but it will change the output values.

Any update will be announced here, among other places, and perhaps within a week or two. So if you do start using this code for anything, keep in mind that a necessary change is coming...

...HOWEVER, we recently became aware of a problem...

Big thanks for mentioning this--I'll watch for the changes.

Seems my proposed data structure incorporating 'actual' difference bytes might run into 'legal' problems. I will need a more 'Bitzi' like approach.

Thoughts:

HashRite should be able to

1) Publish the TigerTreeHash for a given bitprint with a file name of the bitprint and an extension of "TTH'?:

7PJQCHA5IEDECEEQ4YPP2LZ2PQYQI2YJ.H4JFJ47NLNRPNASHYAPAN3RKA3VSXM2LN3EOPRQ.TTH

By using the bitprint as a file name and 'searching' for the file name, I'll be using the same concept as Bitzi--community votes on quality of the file.

This way many people can publish and confirm the 'correct' version of the TigerTree Hash files.

2) Search for HashRitePatch file name in format 'Good SHA1'.'Bad SHA1'.HRP:

7PJQCHA5IEDECEEQ4YPP2LZ2PQYQI2YJ.BRZ5J3R53Z3O3FSMT4B6QW45TPNLDESK.HRP

These are community published files which 'contain' the actual difference data to repair the file.

3) Search for HashRiteDifference file:

7PJQCHA5IEDECEEQ4YPP2LZ2PQYQI2YJ.BRZ5J3R53Z3O3FSMT4B6QW45TPNLDESK.HRD

These files do not contain the actual difference data, just offset and length of differences--should be legal/safe to store on a web site or to share. Requires that HashRite 'locate' the good file in the P2P community and download the 'smallest' amount of data required to correct the 'bad' file.

Will also offer to 'build' a .HRP file to publish for others to reduce BW required to repair files.

4) If no .HRP or HRD files exist, use .TTH to 'repair' local file (using 1k data blocks which have different hash values--again, like above using P2P shared files as source of data) and offer option to publish '.HRD' and/or '.HRP' files.

It makes sense for HasRite to 'reveal' how many bytes must be downloaded from a source to make the files match--this way a user can decide that this is not a good alternative source--This information can also be submitted as part of bitprint metatag info.

I'll prototype HashRite first without the P2P access (download popular variants/corrupted copies of the same file.) I'll publish a command line utility for testing.

I hope that P2P clients that support TTHash will 'offer' the entire hash tree for download--this way I can avoid using intermediate '.TTH' files--better yet, I hope they will directly support file repair.

It would be nice to add a metatag in the bitprint of the TTH data/file bitprint so I can use xml to find the TTH file.

To keep the TTH file small maybe it should only go as deep as 64k data hashes or would all the way down to 1k data hashes be better? Another choice might be to have two files top level hashes down to 64k data hashes then second file which contains hashes below 64k down to 1k?

Does this make sense or am I off base?

Forgot an alternative option is just to provide a file of hashes for every 1/4/64k block of a file since HashRite is really just sequential--it would keep the file small.

Sounds better than saving a binary tree of hashes, since I don't have a TTHash aware program at the other end of the P2P connection.

Files should be small and quick to download--then blocks with bad 'hashes' could be downloaded to fix the bad file and then a smaller HashRiteDifferece file or the larger HashRitePatch file could be generated.

.............HOWEVER, we recently became aware of a problem with the way TigerTree values are calculated which will require a change. The proper fixes are still being evaluated -- it will probably only require a line or two of added code, but it will change the output values......................

Can you please expound upon this topic?

I would like to be able to anticipate your changes.

Thanks

The process we've been using, which is also described by http://www.open-content.net/specs/draft-jchapweske-thex-01.html , allows you to create certain smaller files, filled with some of the internal tree hash data, which match the tree hash value of larger files.

For example, consider a 2048 byte file. Its two 1024 byte segments are each hashed, creating 2 24 byte hashes. Those hashes are then concatenated into a 48-byte segment, which is hashed to give the whole file's tree hash. However, another file which happens to have the same 48-bytes will have the same tree hash. So now you've got two files, of different lengths and contents, with the same tree hash value. (Similar collisions can be synthesized for longer files.)

If you know what file size you're expecting, this isn't a problem. Also, the collisions require the use of secure hash output from the larger file -- so they are essentially nonsense bits, and cannot be chosen for specific purposes.

But still, in its intended use as a secure hash, the TigerTree process should be resistant to any attempt to find two inputs with the same output.

A fix suggested by several experts has been prefixing the leaf data (the up-to-1024-byte file segments) with one flag value (probably a single zero byte) before feeding it to the Tiger hash, and prefixing the node data (the 48-byte concatenations) with a different flag value (probably a single '1' byte) before feeding it to the Tiger hash. This makes intentional node-to-leaf collisions like those highlighted above impossible. (As with any hash process, random collisions are still a theoretical possibility, but at so vanishingly small of a probability as to be of little concern.)

We and the other users of TigerTree hashes are testing this fix, ensuring our implementations match up on results, and preparing to track the new values (and discard the old). More info will appear here when the Bitcollider code is updated and patched TigerTree values start appearing in the Bitzi catalog.

Sandos indicated on another board that Shareaza 1.8.1.1 is out, and that it and draft 02 of THEX agree about the solution to the problem mentioned above. I'm not so interested in the tigertree project at sourceforge, since it won't compile cleanly under Visual Studio. ;) Any feedback on where the next version of bitcollider is in the pipeline?

- Todd

The updated THEX calculation (and other fixes and enhancements) are in our main 'bitcollider' CVS project, the 'bitcollider' module, and that should build cleanly in MSVC.

(The other modules under bitcollider -- such as bitprint and jbitprint -- haven't yet been updated, nor has the separate tigertree sourceforge project, though they eventually will be.)

An official new Bitcollider release is coming Real Soon Now.