[Help-gnutls] OpenSSL <-> GnuTLS imcompatibilities

[David Herrmann]

Hi
I know that OpenSSL only implements SSL2/3 and partly TLS1 but is
there a way to connect with an OpenSSL
client to a simple GnuTLS server?
I used the following code to initialize my ssl listener:
    gnutls_dh_params_init(&ssl_dhparams);
    gnutls_dh_params_generate2(ssl_dhparams, SSL_DH_BITS);
    gnutls_certificate_allocate_credentials(&ssl_cred);
    gnutls_certificate_set_x509_key_file(ssl_cred, pemfile, pemfile,
GNUTLS_X509_FMT_PEM);
    gnutls_certificate_set_dh_params(ssl_cred, ssl_dhparams);

and later:
        gnutls_init(&ssl->session, GNUTLS_SERVER);
        gnutls_priority_set(ssl->session, ssl_priority);
        gnutls_credentials_set(ssl->session, GNUTLS_CRD_CERTIFICATE, ssl_cred);
        gnutls_certificate_server_set_request(ssl->session,
GNUTLS_CERT_REQUEST);
And then if I connect with a simple SSLv3 OpenSSL connection, my
gnutls_handshake returns either
GNUTLS_E_UNEXPECTED_PACKET_LENGTH or GNUTLS_E_UNKNOWN_CIPHER_SUITE.
However, the weird thing is, the OpenSSL handshake call blocks and
does not return any error.
Even when I kill the process of the GnuTLS listener the OpenSSL
handshake still blocks.
It would be nice to hear whether there are known compatibility
problems between GnuTLS and
OpenSSL and whether there are ways to force SSLv3 on GnuTLS to connect
to OpenSSL
without any problems.


Another problem is the following code:
static gnutls_rsa_params_t ssl_rsaparams;
gnutls_rsa_params_init(&ssl_rsaparams);
gnutls_rsa_params_generate2(ssl_rsaparams, 512);
The last function blocks and does not return. gdb gives me something like:
 (gdb) backtrace
#0  0xb800e430 in __kernel_vsyscall ()
#1  0xb7ea577d in select () from /lib/tls/i686/cmov/libc.so.6
#2  0xb7d7f782 in ?? () from /lib/libgcrypt.so.11
#3  0xb7d5334b in ?? () from /lib/libgcrypt.so.11
#4  0xb7d54946 in ?? () from /lib/libgcrypt.so.11
#5  0xb7d54cf9 in ?? () from /lib/libgcrypt.so.11
#6  0xb7d862dc in ?? () from /lib/libgcrypt.so.11
#7  0xb7d51bc7 in ?? () from /lib/libgcrypt.so.11
#8  0xb7d531c9 in ?? () from /lib/libgcrypt.so.11
#9  0xb7d719c4 in ?? () from /lib/libgcrypt.so.11
#10 0xb7d4920f in ?? () from /lib/libgcrypt.so.11
#11 0xb7d3dc24 in gcry_pk_genkey () from /lib/libgcrypt.so.11
#12 0xb7f6a545 in _gnutls_rsa_generate_params () from /usr/lib/libgnutls.so.26
#13 0xb7f8b6e9 in gnutls_x509_privkey_generate () from /usr/lib/libgnutls.so.26
#14 0xb7f6a3b4 in gnutls_rsa_params_generate2 () from /usr/lib/libgnutls.so.26
#15 0x080b878d in ssl_init () at ssl.c:73
 #16 0x0805fa28 in main (argc=4, argv=0xbfa29a64) at ircd.c:730
However, I had to interrupt the process myself. It looks like gcrypt calls some
function which actually is no function. The select() syscall may be some random
data in the memory.
I hope someone can help me

David Herrmann