Problem using the server name extension

[Sam Varshavchik]

Sam Varshavchik writes:

> Simon Josefsson writes:
> 
>> Sam Varshavchik <mrsam at courier-mta.com> writes:
>> 
>>> Simon Josefsson writes:
>>>
>>>> Sam Varshavchik <mrsam at courier-mta.com> writes:
>>>>
>>>>> My client is compiled against gnutls 2.8.5. I am connecting to a
>>>>> server that's built against OpenSSL 1.0.0.
>>>>>
>>>>> The OpenSSL server is failing the handshake with the following error
>>>>> message:
>>>>>
>>>>> error:1408A0E3:SSL routines:SSL3_GET_CLIENT_HELLO:parse tlsext
>>>>>
>>>>> After some Googling around, I remove my client's call to
>>>>> gnutls_server_name_set( .. GNUTLS_NAME_DNS .. ), and that makes
>>>>> OpenSSL happy.
>>>>>
>>>>> If I do not invoke gnutls_server_name_set(), we have a happy
>>>>> conversation. If I invoke gnutls_server_name_set(), OpenSSL bombs out
>>>>> during the handshake.
>>>>>
>>>>> Has anyone seen this before?
>>>>
>>>> We've seen it for very old implementations, notably some IBM-derived
>>>> variant of OpenSSL, that cannot handle any extensions.  But it is very
>>>> surprising to see it for a recent OpenSSL.  Are you sure OpenSSL 1.0.0
>>>> is used?  Can you reproduce this using 'openssl s_server'?  Maybe the
>>>> application server is requesting SSLv2 from OpenSSL?
>>>
>>> The application is the client, and since the application is GnuTLS, it
>>> can't be asking for SSLv2.
>>>
>>> Yes, Fedora 12, OpenSSL 1.0.0 is the server side. It's configured to
>>> accept all protocols (SSLv23_method() in OpenSSL's API), but I also
>>> tried TLSv1_method() as well, no difference.
>>>
>>> On the GnuTLS client side, I'm specifying  GNUTLS_TLS1_1,
>>> GNUTLS_TLS1_0, and GNUTLS_SSL3 in that order. This is not a direct
>>> SSL/TLS connection, this is IMAP STARTTLS, so I can't easily drop in
>>> s_server in the server's place.
>>>
>>> I'll explore what debugging messages are available on the OpenSSL
>>> side. I gave up on the debugger. Debugging optimized code, on either
>>> the server or the client side, just doesn't work very well.
>> 
>> Maybe you can reproduce this using 'gnutls-cli'?  It supports STARTTLS
>> by using --starttls and entering ^D when you want to start the TLS
>> handshake.  Please post output from 'gnutls-cli -d 4711' if you can
>> reproduce it.
> 
> Well now, that seems to work. The handshake appears to complete successfully 
> with gnutls-cli. Perusing gnutls-cli's source, it does seem to use 
> gnutls_server_name_set() by default, so this is a valid, working baseline.
> 
> So, it appears that I'm doing something on the client side, with GnuTLS, 
> that's making OpenSSL on the server side crap out. Looks like I have 
> something that I can dig into. Stay tuned.
> 
>> Maybe the server name you provide is simply the wrong one, and that's
>> why the server refuses to talk with you?
> 
> I don't think that's it. It's "localhost" in both cases. I'm not certain 
> that OpenSSL implements this extension.

Well, thanks to the suggestion to use gnutls-cli, it set me on the right 
path to track down a bug in my code that passed zero for the name_length 
parameter to gnutls_server_name_set(). This apparently slipped past the 
library, which still sent a SERVER_NAME extension packet, but with a null 
hostname, which OpenSSL on the server side rejected with an alert, 
terminating the handshake.

Looks like OpenSSL either silently the server name extension which it 
doesn't recognize, or passes this down to the application (although a quick 
search doesn't seem to locate an OpenSSL API to retrieve the client's 
requested server name), however it rejects outright a 0-length requested 
server name. If you pass an empty string to gnutls_server_name_set() you'll 
see what I was seeing.

And that's all she wrote.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: </pipermail/attachments/20100501/b9b8a73f/attachment.pgp>