Problem using the server name extension

[Simon Josefsson]

Sam Varshavchik <mrsam at courier-mta.com> writes:

> My client is compiled against gnutls 2.8.5. I am connecting to a
> server that's built against OpenSSL 1.0.0.
>
> The OpenSSL server is failing the handshake with the following error
> message:
>
> error:1408A0E3:SSL routines:SSL3_GET_CLIENT_HELLO:parse tlsext
>
> After some Googling around, I remove my client's call to
> gnutls_server_name_set( .. GNUTLS_NAME_DNS .. ), and that makes
> OpenSSL happy.
>
> If I do not invoke gnutls_server_name_set(), we have a happy
> conversation. If I invoke gnutls_server_name_set(), OpenSSL bombs out
> during the handshake.
>
> Has anyone seen this before?

We've seen it for very old implementations, notably some IBM-derived
variant of OpenSSL, that cannot handle any extensions.  But it is very
surprising to see it for a recent OpenSSL.  Are you sure OpenSSL 1.0.0
is used?  Can you reproduce this using 'openssl s_server'?  Maybe the
application server is requesting SSLv2 from OpenSSL?

/Simon