[gnutls-dev] bug in _gnutls_pkcs1_rsa_encrypt
robey at danger.com
Tue Aug 17 02:46:23 CEST 2004
I just figured out what was causing a rare (once every 1000 or so)
failure in the TLS handshake in our tests.
In the "case 2" section of _gnutls_pkcs1_rsa_encrypt(), there's a big
loop that attempts to replace any zero bytes with a non-zero random
number. This line in particular:
if (i<2) ps[i] = rnd[i];
else ps[i] = GMAX( rnd + ps[i-1] + ps[i-2], rnd);
is wrong, because in some cases "rnd + ps[i-1] + ps[i-2]" is 256 or
512, which will be greater than the random byte, but end up being stored
After poking around in this function, I have to raise the question: Is
this loop's complexity absolutely necessary? For every byte in the
random buffer, 3 new random bytes are fetched from the random pool, and
almost always only the 3rd byte is used. This seems like a waste of the
random pool, and my hunch is that the fetch of 3 random bytes was meant
to go OUTSIDE the loop.
Attached is a patch against 1.0.19 which moves the 3-random-byte fetch
outside the loop, and adds a mask inside the GMAX so that only the lower
8 bits count.
This bug appears to be in gnutls 1.1.16 too, though the code has been
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
More information about the Gnutls-devel