[gnutls-dev] bug in _gnutls_pkcs1_rsa_encrypt

[Robey Pointer]

I just figured out what was causing a rare (once every 1000 or so) 
failure in the TLS handshake in our tests.

In the "case 2" section of _gnutls_pkcs1_rsa_encrypt(), there's a big 
loop that attempts to replace any zero bytes with a non-zero random 
number.   This line in particular:

                if (i<2) ps[i] = rnd[i];
                else ps[i] = GMAX( rnd[2] + ps[i-1] + ps[i-2], rnd[1]);

is wrong, because in some cases "rnd[2] + ps[i-1] + ps[i-2]" is 256 or 
512, which will be greater than the random byte, but end up being stored 
as zero.

After poking around in this function, I have to raise the question:  Is 
this loop's complexity absolutely necessary?  For every byte in the 
random buffer, 3 new random bytes are fetched from the random pool, and 
almost always only the 3rd byte is used.  This seems like a waste of the 
random pool, and my hunch is that the fetch of 3 random bytes was meant 
to go OUTSIDE the loop.

Attached is a patch against 1.0.19 which moves the 3-random-byte fetch 
outside the loop, and adds a mask inside the GMAX so that only the lower 
8 bits count.

This bug appears to be in gnutls 1.1.16 too, though the code has been 
reformatted.

robey

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: patch.txt
URL: </pipermail/attachments/20040816/24229502/attachment.txt>