[gnutls-devel] gnutls-3.2.2 memory leak in handshake

[Richard Frith-Macdonald]

I have a server process using gnutls, and it's leaking substantial amounts of memory when sitting idle.
It  is receiving incoming HTTP connections from an OpenNMS system monitoring the port it's listening on, by establishing a connection to the port at fairly frequent intervals.
Running under valgrind reports as follows;

=10642== 24,341,716 bytes in 1,481 blocks are definitely lost in loss record 4,805 of 4,805
==10642==    at 0x4005B83: malloc (vg_replace_malloc.c:195)
==10642==    by 0x4F18D5A: _mbuffer_alloc (gnutls_mbuffers.c:287)
==10642==    by 0x4F19908: _gnutls_io_read_buffered (gnutls_buffers.c:272)
==10642==    by 0x4F14310: _gnutls_recv_in_buffers (gnutls_record.c:1038)
==10642==    by 0x4F1B92A: _gnutls_handshake_io_recv_int (gnutls_buffers.c:1223)
==10642==    by 0x4F1E963: _gnutls_recv_handshake (gnutls_handshake.c:1383)
==10642==    by 0x4F221FE: gnutls_handshake (gnutls_handshake.c:3046)
==10642==    by 0x4551F79: _i_GSTLSSession__handshake (GSTLS.m:1462)
==10642==    by 0x45D4039: _i_GSTLSHandle__sslHandshakeEstablished_outgoing_ (NSFileHandle.m:1026)
==10642==    by 0x41887B0: _i_WebServerConnection___timeout_ (WebServerConnection.m:1805)
==10642==    by 0x462838C: _i_NSObject__performSelector_withObject_ (NSObject.m:2029)
==10642==    by 0x469B047: _i_NSTimer(float, int, long double,...) (NSTimer.m:258)
==10642==    by 0x4667BF5: _i_NSRunLoop__limitDateForMode_ (NSRunLoop.m:1016)
==10642==    by 0x4031CDA: _i_EcProcess__ecRun (EcProcess.m:2137)
==10642==    by 0x804A647: main (main.m:54)
==10642== 


I think the issue is that gnutls_handshake() tries to read and immediately gets EOF, and in this case the memory allocated to buffer the input is leaked.

If you change _gnutls_io_read_buffered() in gnutls_buffers.c to free the buffer on EOF this should fix this case (though there may be other similar problems elsewhere, so perhaps a different fix, eg in _gnutls_read(), would make more sense).

      if (ret == 0) /* EOF */
        {
          _mbuffer_xfree (&bufel);
          return gnutls_assert_val(0);
        }