GNUTLS handshake errors and memory leaks (ECDHE related?)

Tue Jan 10 14:13:57 CET 2012

Yet another log attached. This time gnutls-serv with --priority
'NORMAL:-ECDHE-RSA'.

SSL/TLS Protocol: TLSv1/SSLv3,DHE-RSA-AES256-SHA,2048,256

Still some handshake errors but no related memory leaks. Some minor leaks
still present.

On Tue, Jan 10, 2012 at 4:56 PM, Yaroslav <yarosla at gmail.com> wrote:

> I've run valgrind against gnutls-serv and collected very similar results.
> See logs (ab & gnutls-serv) attached.
>
> ==29724== 18,720 bytes in 390 blocks are definitely lost in loss record 67
> of 70
> ==29724==    at 0x4C28F9F: malloc (vg_replace_malloc.c:236)
> ==29724==    by 0x5B03908: __gmp_default_allocate (in
> /usr/lib/libgmp.so.10.0.1)
> ==29724==    by 0x5B17F39: __gmpz_mul (in /usr/lib/libgmp.so.10.0.1)
> ==29724==    by 0x4EE5444: ecc_projective_check_point
> (ecc_projective_check_point.c:60)
> ==29724==    by 0x4EDE92A: _wrap_nettle_pk_derive (pk.c:144)
> ==29724==    by 0x4EC54A2: calc_ecdh_key (ecdh_common.c:64)
> ==29724==    by 0x4EC5569: _gnutls_proc_ecdh_common_client_kx
> (ecdh_common.c:124)
> ==29724==    by 0x4E595E6: _gnutls_recv_client_kx_message (gnutls_kx.c:549)
> ==29724==    by 0x4E56D25: _gnutls_handshake_server
> (gnutls_handshake.c:2801)
> ==29724==    by 0x4E57622: gnutls_handshake (gnutls_handshake.c:2368)
> ==29724==    by 0x405A99: main (serv.c:1270)
>
>
> On Tue, Jan 10, 2012 at 2:44 PM, Yaroslav <yarosla at gmail.com> wrote:
>
>> Hi!
>>
>> I have added SSL support to my web server (nxweb) using GNUTLS. The
>> server has asynchronous architecture (based on epoll), where 4 threads run
>> almost independently accepting and servicing client http connections.
>> Therefore I am using GNUTLS in non-blocking mode.
>>
>> Platform: Ubuntu 11.10, x64, 4-core CPU.
>> Linked with latest GNUTLS 3.0.11.
>>
>> Most of the time everything works fine. Benchmarks show significant speed
>> advantage over nginx (which is build on OpenSSL). But sometimes there are
>> handshake errors and also memory leaks, which might be bound to those
>> errors.
>>
>> I am attaching full valgrind log. Here are some excerpts:
>>
>> % valgrind --leak-check=full --show-reachable=yes
>> dist/Debug/GNU-Linux-x86/nxweb
>> ...
>> 2012-01-10 13:46:51 [28704:0x71b9700]: gnutls handshake error -24 fatal=1
>> 2012-01-10 13:47:25 [28704:0x71b9700]: gnutls handshake error -110 fatal=1
>> 2012-01-10 13:47:25 [28704:0x71b9700]: gnutls handshake error -10 fatal=1
>> ...
>> ==28704== 47,952 bytes in 999 blocks are definitely lost in loss record
>> 11 of 11
>> ==28704==    at 0x4C28F9F: malloc (vg_replace_malloc.c:236)
>> ==28704==    by 0x5F28908: __gmp_default_allocate (in
>> /usr/lib/libgmp.so.10.0.1)
>> ==28704==    by 0x5F3CF39: __gmpz_mul (in /usr/lib/libgmp.so.10.0.1)
>> ==28704==    by 0x50ED444: ecc_projective_check_point
>> (ecc_projective_check_point.c:60)
>> ==28704==    by 0x50E692A: _wrap_nettle_pk_derive (pk.c:144)
>> ==28704==    by 0x50CD4A2: calc_ecdh_key (ecdh_common.c:64)
>> ==28704==    by 0x50CD569: _gnutls_proc_ecdh_common_client_kx
>> (ecdh_common.c:124)
>> ==28704==    by 0x50615E6: _gnutls_recv_client_kx_message
>> (gnutls_kx.c:549)
>> ==28704==    by 0x505ED25: _gnutls_handshake_server
>> (gnutls_handshake.c:2801)
>> ==28704==    by 0x505F622: gnutls_handshake (gnutls_handshake.c:2368)
>> ==28704==    by 0x4088FD: do_handshake (nxd_ssl_socket.c:158)
>> ==28704==    by 0x408A62: handshake_stub_is_do_write
>> (nxd_ssl_socket.c:188)
>>
>> There are also other leaks related to gnutls_dh_params_generate2 and some
>> error messages related to gnutls_certificate_set_x509_key_file but those
>> are of less importance to me as they happen only once on server startup.
>>
>> The log has been collected while running apache-bench (ab) on the
>> client-side. Here is the log:
>>
>> % ab -c 100 -n 1000 'https://test.domain.ru:8056/benchmark'
>> This is ApacheBench, Version 2.3 <$Revision: 655654 $>
>> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
>> Licensed to The Apache Software Foundation, http://www.apache.org/
>>
>> Benchmarking test.domain.ru (be patient)
>> Completed 100 requests
>> Completed 200 requests
>> Completed 300 requests
>> Completed 400 requests
>> Completed 500 requests
>> Completed 600 requests
>> Completed 700 requests
>> SSL handshake failed (5).
>> SSL handshake failed (5).
>> SSL handshake failed (5).
>> Completed 800 requests
>> Completed 900 requests
>> Completed 1000 requests
>> Finished 1000 requests
>>
>>
>> Server Software:        nxweb/3.0.0-dev
>> Server Hostname:        test.domain.ru
>> Server Port:            8056
>> SSL/TLS Protocol:       TLSv1/SSLv3,ECDHE-RSA-AES256-SHA,2048,256
>>
>> Document Path:          /benchmark
>> Document Length:        20 bytes
>>
>> Concurrency Level:      100
>> Time taken for tests:   159.929 seconds
>> Complete requests:      1000
>> Failed requests:        3
>>    (Connect: 0, Receive: 0, Length: 3, Exceptions: 0)
>> Write errors:           0
>> Total transferred:      164505 bytes
>> HTML transferred:       19940 bytes
>> Requests per second:    6.25 [#/sec] (mean)
>> Time per request:       15992.885 [ms] (mean)
>> Time per request:       159.929 [ms] (mean, across all concurrent
>> requests)
>> Transfer rate:          1.00 [Kbytes/sec] received
>>
>> Connection Times (ms)
>>               min  mean[+/-sd] median   max
>> Connect:        0 13464 1915.7  13581   16479
>> Processing:   115 2526 1904.2   2416   15897
>> Waiting:      115 2487 1758.1   2411   15785
>> Total:      15870 15989 212.0  15912   16645
>>
>>
>> The problems might be related to ECDHE algorithm. There were no handshake
>> errors when 'PERFORMANCE:%SERVER_PRECEDENCE' priority string was used.
>> There were no handshake errors when using GNUTLS 2.10.5 (from Ubuntu repo)
>> -- I switched to 3.0 version as it supports significantly faster ECDHE
>> algorithm, to be on par with nginx/openssl.
>>
>> And there seemed to be no such leaks when there were no handshake erros.
>>
>> And, yes, I do call gnutls_deinit(session) when closing every connection
>> regardless of errors on it. I am ready to provide full source code (I plan
>> to publish it soon as it is open source).
>>
>> Regards,
>>
>> Yaroslav Stavnichiy
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/attachments/20120110/6f934abc/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: valgrind_log_120110-gnutls-serv-2.log
Type: text/x-log
Size: 57609 bytes
Desc: not available
URL: </pipermail/attachments/20120110/6f934abc/attachment.bin>