sha1 hash using libgcrypt different from what returns sha1sum

[Denis Corbin]

Hi,

first, I hope that I do not sent my questions to the wrong mailing-list.

I maintain the dar (aka Disk ARchive) backup software that generates
sha1 or md5 sums of files it generates using libgcrypt. Recently a user
reported a hash failure on a large file when using sha1sum against the
hash generated by dar, while at the same time the generated archive was
not corrupted.

I could reproduce this problem but only using large files (larger than
200 GB and could not clearly find a threshold below which the problem
never exists and above which it always occurs).

dar is a somehow large C++ program thus I've extracted the pertinent
code into a more simple C program for easier review and criticism about
the way dar calls libgcrypt. This C code (hashsum.c [1]) also reproduces
the problem. I have re-read the libgcrypt documentation and could not
find anything wrong... but it's still possible I've missed something...

To avoid generating large testing files and to reduce execution time of
the tests the make_sparse_file.c program ([1]) let generate a so called
sparse file (if the underlying filesystem supports it). Note that using
dd to create a real plain equivalent file leads to the same difference
of hash for large files. Note also that if the generated file for
testing contains a long series of byte set to zero, the problem also
occurs for more random data pattern (like slice generated by dar).

The system used is Debian wheezy, on 64 bits AMD host:
    libgcrypt version based on 1.5.0 (with probable Debian patches)
    sha1sum (GNU coreutils) 8.13
    md5sum (GNU coreutils) 8.13

Using libgcrypt 1.5.3 gives the same result.

Well I wonder whether this is due to wrong usage of libgcrypt in dar or
here hashsum.c? Or is it a bug in libgcrypt or in sha1sum and md5sum?

Thanks for any help,
Regards,
Denis Corbin.

[1] source code and test results are available for download at:
	http://dar.linux.free.fr/libgcrypt-tests.tar.gz  (~ 3 KB)