Handshake fails with Internal error in memory allocation

[Simon Josefsson]

Nikos Mavrogiannopoulos <nmav at gnutls.org> writes:

> Simon Josefsson wrote:
>> This error has come up lately, see:
>> 
>> http://bugs.debian.org/466477
>> http://bugs.debian.org/478191
>> 
>> The cause seems clear, the server sends a huge list of CA certs and
>> GnuTLS runs into some fixed size buffer or something.  This reproduces
>> it:
>> 
>> gnutls-cli -p 25 -s mail3.mclemente.net
>> ehlo foo
>> starttls
>> ^D
>> 
>> Nikos, do you have any idea?  I could look at it, but have little time
>> right now.
>
> There is this function that can be used to increase the default
> handshake packet size. The current is 16kb and is used to protect from
> denial of service.
>
> /**
>   * gnutls_handshake_set_max_packet_length - set the maximum length of a
> handshake message
>   * @session: is a #gnutls_session_t structure.
>   * @max: is the maximum number.
>   *
>   * This function will set the maximum size of a handshake message.
>   * Handshake messages over this size are rejected.  The default value
>   * is 16kb which is large enough. Set this to 0 if you do not want to
>   * set an upper limit.
>   *
>   **/

Ah, thanks for the pointer.  It seems this is also used for the _total_
handshake packet size, in gnutls_buffers.c:


/* Buffer for handshake packets. Keeps the packets in order
 * for finished messages to use them. Used in HMAC calculation
 * and finished messages.
 */
int
_gnutls_handshake_buffer_put (gnutls_session_t session, opaque * data,
			      size_t length)
{

  if (length == 0)
    return 0;

  if ((session->internals.max_handshake_data_buffer_size > 0) &&
      ((length + session->internals.handshake_hash_buffer.length) >
       session->internals.max_handshake_data_buffer_size))
    {
      gnutls_assert ();
      return GNUTLS_E_HANDSHAKE_TOO_LARGE;
    }

I've increased the default limit to 48kb.  One of the servers in these
two bug reports needed 25kb to do the handshake.

I've also updated the documentation for that function slightly.

/Simon