sporadic failures in RSA code

[Christian Grothoff]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I wrote a testcase for GNUnet 0.5.1 which now supports libgcrypt for 
encryption. The testcase creates a secret RSA key and then performs 100 
encrypt/decrypt or 100 sign/verify operations. The test sometimes runs 
through smoothly and sometimes fails at a random iteration. The operation 
that fails is either the verification of the signature (invalid signature) or 
the decryption (result not in PKCS #1 format).

The testcase is available as part of GNUnet's main tar-ball at
ftp://ftp.gnu.org/gnu/GNUnet-0.5.1.tar.bz2

You need to run configure with --with-gcrypt=PATH-TO-GCRYPT,
and then do

# make 
# cd src/test
# make hostkeytest
# ./hostkeytest

You may have to run hostkeytest multiple times to get the problem. On any of 
my machines (Linux 2.4, i386, libgcrypt from CVS of today), I get a mixture 
of test runs that pass and fail:

>>>>>>
[grothoff@gecko test]$ ./hostkeytest
....................................................................................................100 
RSA encrypt/decrypt operations 11s
....................................................................................................100 
RSA sign/verify operations 11s
....................................................................................................100 
RSA encrypt/encode/decode/decrypt operations 11s
[grothoff@gecko test]$ ./hostkeytest
....................................................................................................100 
RSA encrypt/decrypt operations 11s
....................................................................................................100 
RSA sign/verify operations 10s
........................................ERROR: not a pkcs-1 block type 2 
(size=256, *tmp=100)!


1 TESTS FAILED!

[grothoff@gecko test]$ ./hostkeytest
....................................................................................................100 
RSA encrypt/decrypt operations 11s
....................................................................................................100 
RSA sign/verify operations 10s
....................................................................................................100 
RSA encrypt/encode/decode/decrypt operations 11s
[grothoff@gecko test]$ ./hostkeytest
....................................................................................................100 
RSA encrypt/decrypt operations 14s
....................................................................................................100 
RSA sign/verify operations 11s
..................ERROR: not a pkcs-1 block type 2 (size=256, *tmp=74)!


1 TESTS FAILED!

[grothoff@gecko test]$ ./hostkeytest
....................................................................................................100 
RSA encrypt/decrypt operations 13s
....................................................................................................100 
RSA sign/verify operations 12s
....................................................................................................100 
RSA encrypt/encode/decode/decrypt operations 11s
[grothoff@gecko test]$ ./hostkeytest
...................................................................................................ERROR: 
not a pkcs-1 block type 2 (size=256, *tmp=94)!
....................................................................................................100 
RSA sign/verify operations 11s
....................................................................................................100 
RSA encrypt/encode/decode/decrypt operations 11s


1 TESTS FAILED!

[grothoff@gecko test]$ ./hostkeytest
....................................................................................................100 
RSA encrypt/decrypt operations 11s
....................................................................................................100 
RSA sign/verify operations 11s
....................................................................................................100 
RSA encrypt/encode/decode/decrypt operations 11s
[grothoff@gecko test]$ ./hostkeytest
.................................................ERROR: not a pkcs-1 block 
type 2 (size=256, *tmp=54)!
....................................................................................................100 
RSA sign/verify operations 12s
....................................................................................................100 
RSA encrypt/encode/decode/decrypt operations 11s


1 TESTS FAILED!
<<<<<


Note that if you do not use --with-gcrypt, the configure magic may decide to 
use OpenSSL and the testcases will always pass. There is another interesting 
testcase (cry_comp.c) which tests compatibility of the RSA operations between 
OpenSSL and libgcrypt. Libgcrypt passes the compatibility test, except if the 
random decrypt/verify problem occurs.

best regards

Christian Grothoff [GNUnet maintainer]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE+QB039tNtMeXQLkIRAtIwAJ0dPVtXSnwzgo1kgAAKehWBWhfcygCdGu53
EYe91UVluVKhdJYongNZ0/4=
=wbP4
-----END PGP SIGNATURE-----