sha1 hash using libgcrypt different from what returns sha1sum

[Yuriy Kaminskiy]

Werner Koch wrote:
> On Tue, 12 Nov 2013 00:44, yumkam at gmail.com said:
> 
>> I strongly believe this is a bug, I have not found any such behavior in standard
> 
> You are right.  This is a limitation of the code which was never hit in
> practice until now - at least I hope so.  The more disturbing fact is
> that this also affects GPG encrypted files: SHA-1 is used for an MDC to
> protect the encrtpted messages.  If both parties use GPG, this won't be
> a problem but it is not standard compliant.

I guess, same apply to signatures? (and it is not only sha1, it's most other
hashes [except of sha384 and sha512] too)

> Now, what shall we do with GPG? 
> 
>  - Fix the code and hope that no encrypted files larger than 256GB need
>    decryption?
> 
>  - Fix and print a warning for an MDC mismatch in case the file is too
>    long.

I'd vote for this as quick fix.

>  - Fix and add an option to use the unfixed SHA-1 code?  Takes a lot of
>    time for processing.

Actually, it won't add much time on *processing* - you can produce both versions
at nearly same cost as one, only *final method needs to be different to produce
both versions (one with counter truncation, one without).
(But I guess it will take lot of time on *designing proper API* for such hack;
and I'm not sure if it worth it.)
(However, while only printing warning feels "acceptable workaround" for MDC, it
feels totally wrong for signatures, so I guess in long run, there are no way
around it. Fortunately, very few users should be affected.)

> Anyone tested this with PGP?
> 
>> There are exactly same bug with sha256 and md5 implementations

... and more: md4, rmd160, tiger.

And I'm not sure, but cipher-ccm.c also feels suspicious in this respect (won't
it fail after SIZE_T_MAX bytes?).

>> (but, curiously, there are *no* similar problem in sha512).

> SHA-512 uses a 64 bit type for the counter because its implementation
> requires a 64 bit type anyway.

Strictly speaking, libgcrypt's sha512 and whirlpool are "broken" too - standard
limits message size with 2**128 bits (more for whirlpool), and so libgcrypt
sha512 will also fail after about 2**73 + 127 bytes. But I guess, unlike
*md*/sha*, it won't become practical problem for some quite time :-).

> p.s.
> Funny that Libgcrypt passes the FIPS validation.

(...And most likely fix will be in the not-yet-validated version faster.
"Validation" is nice and good if it is really able to catch all implementation
errors (which is, of course, pipedream). If bug was found after validation,
re-validation become additional hurdle on the path to employ fix on all affected
systems.)