GNUTLS handshake errors and memory leaks (ECDHE related?)

Tue Jan 10 13:56:53 CET 2012

I've run valgrind against gnutls-serv and collected very similar results.
See logs (ab & gnutls-serv) attached.

==29724== 18,720 bytes in 390 blocks are definitely lost in loss record 67
of 70
==29724==    at 0x4C28F9F: malloc (vg_replace_malloc.c:236)
==29724==    by 0x5B03908: __gmp_default_allocate (in
/usr/lib/libgmp.so.10.0.1)
==29724==    by 0x5B17F39: __gmpz_mul (in /usr/lib/libgmp.so.10.0.1)
==29724==    by 0x4EE5444: ecc_projective_check_point
(ecc_projective_check_point.c:60)
==29724==    by 0x4EDE92A: _wrap_nettle_pk_derive (pk.c:144)
==29724==    by 0x4EC54A2: calc_ecdh_key (ecdh_common.c:64)
==29724==    by 0x4EC5569: _gnutls_proc_ecdh_common_client_kx
(ecdh_common.c:124)
==29724==    by 0x4E595E6: _gnutls_recv_client_kx_message (gnutls_kx.c:549)
==29724==    by 0x4E56D25: _gnutls_handshake_server
(gnutls_handshake.c:2801)
==29724==    by 0x4E57622: gnutls_handshake (gnutls_handshake.c:2368)
==29724==    by 0x405A99: main (serv.c:1270)

On Tue, Jan 10, 2012 at 2:44 PM, Yaroslav <yarosla at gmail.com> wrote:

> Hi!
>
> I have added SSL support to my web server (nxweb) using GNUTLS. The server
> has asynchronous architecture (based on epoll), where 4 threads run almost
> independently accepting and servicing client http connections. Therefore I
> am using GNUTLS in non-blocking mode.
>
> Platform: Ubuntu 11.10, x64, 4-core CPU.
> Linked with latest GNUTLS 3.0.11.
>
> Most of the time everything works fine. Benchmarks show significant speed
> advantage over nginx (which is build on OpenSSL). But sometimes there are
> handshake errors and also memory leaks, which might be bound to those
> errors.
>
> I am attaching full valgrind log. Here are some excerpts:
>
> % valgrind --leak-check=full --show-reachable=yes
> dist/Debug/GNU-Linux-x86/nxweb
> ...
> 2012-01-10 13:46:51 [28704:0x71b9700]: gnutls handshake error -24 fatal=1
> 2012-01-10 13:47:25 [28704:0x71b9700]: gnutls handshake error -110 fatal=1
> 2012-01-10 13:47:25 [28704:0x71b9700]: gnutls handshake error -10 fatal=1
> ...
> ==28704== 47,952 bytes in 999 blocks are definitely lost in loss record 11
> of 11
> ==28704==    at 0x4C28F9F: malloc (vg_replace_malloc.c:236)
> ==28704==    by 0x5F28908: __gmp_default_allocate (in
> /usr/lib/libgmp.so.10.0.1)
> ==28704==    by 0x5F3CF39: __gmpz_mul (in /usr/lib/libgmp.so.10.0.1)
> ==28704==    by 0x50ED444: ecc_projective_check_point
> (ecc_projective_check_point.c:60)
> ==28704==    by 0x50E692A: _wrap_nettle_pk_derive (pk.c:144)
> ==28704==    by 0x50CD4A2: calc_ecdh_key (ecdh_common.c:64)
> ==28704==    by 0x50CD569: _gnutls_proc_ecdh_common_client_kx
> (ecdh_common.c:124)
> ==28704==    by 0x50615E6: _gnutls_recv_client_kx_message (gnutls_kx.c:549)
> ==28704==    by 0x505ED25: _gnutls_handshake_server
> (gnutls_handshake.c:2801)
> ==28704==    by 0x505F622: gnutls_handshake (gnutls_handshake.c:2368)
> ==28704==    by 0x4088FD: do_handshake (nxd_ssl_socket.c:158)
> ==28704==    by 0x408A62: handshake_stub_is_do_write (nxd_ssl_socket.c:188)
>
> There are also other leaks related to gnutls_dh_params_generate2 and some
> error messages related to gnutls_certificate_set_x509_key_file but those
> are of less importance to me as they happen only once on server startup.
>
> The log has been collected while running apache-bench (ab) on the
> client-side. Here is the log:
>
> % ab -c 100 -n 1000 'https://test.domain.ru:8056/benchmark'
> This is ApacheBench, Version 2.3 <$Revision: 655654 $>
> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
> Licensed to The Apache Software Foundation, http://www.apache.org/
>
> Benchmarking test.domain.ru (be patient)
> Completed 100 requests
> Completed 200 requests
> Completed 300 requests
> Completed 400 requests
> Completed 500 requests
> Completed 600 requests
> Completed 700 requests
> SSL handshake failed (5).
> SSL handshake failed (5).
> SSL handshake failed (5).
> Completed 800 requests
> Completed 900 requests
> Completed 1000 requests
> Finished 1000 requests
>
>
> Server Software:        nxweb/3.0.0-dev
> Server Hostname:        test.domain.ru
> Server Port:            8056
> SSL/TLS Protocol:       TLSv1/SSLv3,ECDHE-RSA-AES256-SHA,2048,256
>
> Document Path:          /benchmark
> Document Length:        20 bytes
>
> Concurrency Level:      100
> Time taken for tests:   159.929 seconds
> Complete requests:      1000
> Failed requests:        3
>    (Connect: 0, Receive: 0, Length: 3, Exceptions: 0)
> Write errors:           0
> Total transferred:      164505 bytes
> HTML transferred:       19940 bytes
> Requests per second:    6.25 [#/sec] (mean)
> Time per request:       15992.885 [ms] (mean)
> Time per request:       159.929 [ms] (mean, across all concurrent requests)
> Transfer rate:          1.00 [Kbytes/sec] received
>
> Connection Times (ms)
>               min  mean[+/-sd] median   max
> Connect:        0 13464 1915.7  13581   16479
> Processing:   115 2526 1904.2   2416   15897
> Waiting:      115 2487 1758.1   2411   15785
> Total:      15870 15989 212.0  15912   16645
>
>
> The problems might be related to ECDHE algorithm. There were no handshake
> errors when 'PERFORMANCE:%SERVER_PRECEDENCE' priority string was used.
> There were no handshake errors when using GNUTLS 2.10.5 (from Ubuntu repo)
> -- I switched to 3.0 version as it supports significantly faster ECDHE
> algorithm, to be on par with nginx/openssl.
>
> And there seemed to be no such leaks when there were no handshake erros.
>
> And, yes, I do call gnutls_deinit(session) when closing every connection
> regardless of errors on it. I am ready to provide full source code (I plan
> to publish it soon as it is open source).
>
> Regards,
>
> Yaroslav Stavnichiy
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/attachments/20120110/2d987415/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: valgrind_log_120110-3-gnutls-serv.log
Type: text/x-log
Size: 43558 bytes
Desc: not available
URL: </pipermail/attachments/20120110/2d987415/attachment.bin>