[mod_gnutls-devel] [mod_gnutls] #136: Infinite loop while cleaning up GnuTLS session
mod_gnutls
webmaster at mod.gnutls.org
Sat May 21 14:54:01 CEST 2016
#136: Infinite loop while cleaning up GnuTLS session
-------------------------------------+-------------------------------------
Reporter: | Owner:
http://gtlsbug.livejournal.com/ | https://id.mayfirst.org/dkg
Type: defect | Status: new
Priority: critical | Component: code
Version: | Keywords:
-------------------------------------+-------------------------------------
I noticed that some apache2 processes take 100% CPU after a few requests.
Stack trace shows mod-gnutls as the problem. This problem causes embedded
devices to slow down to an unusable state after using them for a while.
A quick read tells us that cleanup_gnutls_session() getting stuck in a
loop due to GNUTLS_E_INTERRUPTED or GNUTLS_E_AGAIN being returned by
gnutls_bye(). After mgs_transport_write(), gnutls understands these
errnos even though errno is not set during that function?
Following is the debug information I collected. I am available to get
more information or try out patches if necessary. Thanks in advance.
----
'''How to Reproduce:'''
The problem is easily reproducible on my machines by sending a lot of
requests to Apache. Most requests don't cause a problem while some lead
to the problem. I believe the problem occurs when using mod-gnutls and
proxying connections. Also this is more observable on pages taking time
to load.
$ ab -v 1 -c 8 -n 800 -H 'Cookie: XXX' https://freedombox-
vm1/plinth/sys/users/
'''Stack traces:'''
I collected several stack traces by letting the process run for a while
and then interrupting it again.
{{{
#0 mgs_transport_write (ptr=0x7f785b260028, buffer=<optimized out>,
len=8023) at gnutls_io.c:871
#1 0x00007f78669d86d5 in _gnutls_writev_emu (session=<optimized out>,
session=<optimized out>, vec=<optimized out>, giovec_cnt=<optimized out>,
giovec=<optimized out>, fd=<optimized out>) at gnutls_buffers.c:447
#2 _gnutls_writev (total=8023, giovec_cnt=<optimized out>,
giovec=0x7fff2ce06a20, session=0x564c16d28710) at gnutls_buffers.c:505
#3 _gnutls_io_write_flush (session=session at entry=0x564c16d28710) at
gnutls_buffers.c:699
#4 0x00007f78669d4d5c in gnutls_bye (session=0x564c16d28710,
how=how at entry=GNUTLS_SHUT_WR) at gnutls_record.c:281
#5 0x00007f78641a329e in cleanup_gnutls_session (data=0x7f785b260028) at
gnutls_hooks.c:722
#6 0x00007f78693e1cfe in run_cleanups (cref=0x7f785b285098) at
/tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:2352
#7 apr_pool_clear (pool=0x7f785b285028) at
/tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:762
#8 0x00007f78609ab4ce in child_main (child_num_arg=child_num_arg at entry=7,
child_bucket=child_bucket at entry=0) at prefork.c:616
#9 0x00007f78609ab974 in make_child (s=0x7f7869c79470, slot=7, bucket=0)
at prefork.c:824
#10 0x00007f78609ac85d in perform_idle_server_maintenance (p=<optimized
out>) at prefork.c:932
#11 prefork_run (_pconf=<optimized out>, plog=<optimized out>,
s=<optimized out>) at prefork.c:1128
#12 0x0000564c15541eae in ap_run_mpm (pconf=0x7f7869ca9028,
plog=0x7f7869c76028, s=0x7f7869c79470) at mpm_common.c:94
#13 0x0000564c1553b516 in main (argc=3, argv=0x7fff2ce070c8) at main.c:778
#0 write_flush (ctxt=ctxt at entry=0x7f785b260028) at gnutls_io.c:624
#1 0x00007f786419ee32 in mgs_transport_write (ptr=0x7f785b260028,
buffer=<optimized out>, len=8023) at gnutls_io.c:871
#2 0x00007f78669d86d5 in _gnutls_writev_emu (session=<optimized out>,
session=<optimized out>, vec=<optimized out>, giovec_cnt=<optimized out>,
giovec=<optimized out>, fd=<optimized out>) at gnutls_buffers.c:447
#3 _gnutls_writev (total=8023, giovec_cnt=<optimized out>,
giovec=0x7fff2ce06a20, session=0x564c16d28710) at gnutls_buffers.c:505
#4 _gnutls_io_write_flush (session=session at entry=0x564c16d28710) at
gnutls_buffers.c:699
#5 0x00007f78669d4d5c in gnutls_bye (session=0x564c16d28710,
how=how at entry=GNUTLS_SHUT_WR) at gnutls_record.c:281
#6 0x00007f78641a329e in cleanup_gnutls_session (data=0x7f785b260028) at
gnutls_hooks.c:722
#7 0x00007f78693e1cfe in run_cleanups (cref=0x7f785b285098) at
/tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:2352
#8 apr_pool_clear (pool=0x7f785b285028) at
/tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:762
#9 0x00007f78609ab4ce in child_main (child_num_arg=child_num_arg at entry=7,
child_bucket=child_bucket at entry=0) at prefork.c:616
#10 0x00007f78609ab974 in make_child (s=0x7f7869c79470, slot=7, bucket=0)
at prefork.c:824
#11 0x00007f78609ac85d in perform_idle_server_maintenance (p=<optimized
out>) at prefork.c:932
#12 prefork_run (_pconf=<optimized out>, plog=<optimized out>,
s=<optimized out>) at prefork.c:1128
#13 0x0000564c15541eae in ap_run_mpm (pconf=0x7f7869ca9028,
plog=0x7f7869c76028, s=0x7f7869c79470) at mpm_common.c:94
#14 0x0000564c1553b516 in main (argc=3, argv=0x7fff2ce070c8) at main.c:778
#0 0x00007f78696022e0 in apr_bucket_alloc at plt () from /usr/lib/x86_64
-linux-gnu/libaprutil-1.so.0
#1 0x00007f7869604591 in apr_bucket_flush_create () from /usr/lib/x86_64
-linux-gnu/libaprutil-1.so.0
#2 0x00007f786419dc1c in write_flush (ctxt=ctxt at entry=0x7f785b260028) at
gnutls_io.c:642
#3 0x00007f786419ee32 in mgs_transport_write (ptr=0x7f785b260028,
buffer=<optimized out>, len=8023) at gnutls_io.c:871
#4 0x00007f78669d86d5 in _gnutls_writev_emu (session=<optimized out>,
session=<optimized out>, vec=<optimized out>, giovec_cnt=<optimized out>,
giovec=<optimized out>, fd=<optimized out>) at gnutls_buffers.c:447
#5 _gnutls_writev (total=8023, giovec_cnt=<optimized out>,
giovec=0x7fff2ce06a20, session=0x564c16d28710) at gnutls_buffers.c:505
#6 _gnutls_io_write_flush (session=session at entry=0x564c16d28710) at
gnutls_buffers.c:699
#7 0x00007f78669d4d5c in gnutls_bye (session=0x564c16d28710,
how=how at entry=GNUTLS_SHUT_WR) at gnutls_record.c:281
#8 0x00007f78641a329e in cleanup_gnutls_session (data=0x7f785b260028) at
gnutls_hooks.c:722
#9 0x00007f78693e1cfe in run_cleanups (cref=0x7f785b285098) at
/tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:2352
#10 apr_pool_clear (pool=0x7f785b285028) at
/tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:762
#11 0x00007f78609ab4ce in child_main (child_num_arg=child_num_arg at entry=7,
child_bucket=child_bucket at entry=0) at prefork.c:616
#12 0x00007f78609ab974 in make_child (s=0x7f7869c79470, slot=7, bucket=0)
at prefork.c:824
#13 0x00007f78609ac85d in perform_idle_server_maintenance (p=<optimized
out>) at prefork.c:932
#14 prefork_run (_pconf=<optimized out>, plog=<optimized out>,
s=<optimized out>) at prefork.c:1128
#15 0x0000564c15541eae in ap_run_mpm (pconf=0x7f7869ca9028,
plog=0x7f7869c76028, s=0x7f7869c79470) at mpm_common.c:94
#16 0x0000564c1553b516 in main (argc=3, argv=0x7fff2ce070c8) at main.c:778
#0 _mbuffer_head_get_next (cur=0x564c16d2c9a0,
msg=msg at entry=0x7fff2ce06a10) at gnutls_mbuffers.c:191
#1 0x00007f78669d83bd in _gnutls_io_write_flush
(session=session at entry=0x564c16d28710) at gnutls_buffers.c:682
#2 0x00007f78669d4d5c in gnutls_bye (session=0x564c16d28710,
how=how at entry=GNUTLS_SHUT_WR) at gnutls_record.c:281
#3 0x00007f78641a329e in cleanup_gnutls_session (data=0x7f785b260028) at
gnutls_hooks.c:722
#4 0x00007f78693e1cfe in run_cleanups (cref=0x7f785b285098) at
/tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:2352
#5 apr_pool_clear (pool=0x7f785b285028) at
/tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:762
#6 0x00007f78609ab4ce in child_main (child_num_arg=child_num_arg at entry=7,
child_bucket=child_bucket at entry=0) at prefork.c:616
#7 0x00007f78609ab974 in make_child (s=0x7f7869c79470, slot=7, bucket=0)
at prefork.c:824
#8 0x00007f78609ac85d in perform_idle_server_maintenance (p=<optimized
out>) at prefork.c:932
#9 prefork_run (_pconf=<optimized out>, plog=<optimized out>,
s=<optimized out>) at prefork.c:1128
#10 0x0000564c15541eae in ap_run_mpm (pconf=0x7f7869ca9028,
plog=0x7f7869c76028, s=0x7f7869c79470) at mpm_common.c:94
#11 0x0000564c1553b516 in main (argc=3, argv=0x7fff2ce070c8) at main.c:778
#0 0x00007f7869602caa in apr_brigade_cleanup () from /usr/lib/x86_64
-linux-gnu/libaprutil-1.so.0
#1 0x0000564c1555926a in ap_core_output_filter (f=0x7f785b2641e0,
new_bb=0x7f785b2640e8) at core_filters.c:385
#2 0x00007f786419dc4d in write_flush (ctxt=ctxt at entry=0x7f785b260028) at
gnutls_io.c:645
#3 0x00007f786419ee32 in mgs_transport_write (ptr=0x7f785b260028,
buffer=<optimized out>, len=8023) at gnutls_io.c:871
#4 0x00007f78669d86d5 in _gnutls_writev_emu (session=<optimized out>,
session=<optimized out>, vec=<optimized out>, giovec_cnt=<optimized out>,
giovec=<optimized out>, fd=<optimized out>) at gnutls_buffers.c:447
#5 _gnutls_writev (total=8023, giovec_cnt=<optimized out>,
giovec=0x7fff2ce06a20, session=0x564c16d28710) at gnutls_buffers.c:505
#6 _gnutls_io_write_flush (session=session at entry=0x564c16d28710) at
gnutls_buffers.c:699
#7 0x00007f78669d4d5c in gnutls_bye (session=0x564c16d28710,
how=how at entry=GNUTLS_SHUT_WR) at gnutls_record.c:281
#8 0x00007f78641a329e in cleanup_gnutls_session (data=0x7f785b260028) at
gnutls_hooks.c:722
#9 0x00007f78693e1cfe in run_cleanups (cref=0x7f785b285098) at
/tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:2352
#10 apr_pool_clear (pool=0x7f785b285028) at
/tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:762
#11 0x00007f78609ab4ce in child_main (child_num_arg=child_num_arg at entry=7,
child_bucket=child_bucket at entry=0) at prefork.c:616
#12 0x00007f78609ab974 in make_child (s=0x7f7869c79470, slot=7, bucket=0)
at prefork.c:824
#13 0x00007f78609ac85d in perform_idle_server_maintenance (p=<optimized
out>) at prefork.c:932
#14 prefork_run (_pconf=<optimized out>, plog=<optimized out>,
s=<optimized out>) at prefork.c:1128
#15 0x0000564c15541eae in ap_run_mpm (pconf=0x7f7869ca9028,
plog=0x7f7869c76028, s=0x7f7869c79470) at mpm_common.c:94
#16 0x0000564c1553b516 in main (argc=3, argv=0x7fff2ce070c8) at main.c:778
}}}
'''Software versions:'''
{{{
root at freedombox-vm1:~# dpkg -l libapache2-mod-gnutls* apache2* libgnutls*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-
pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version
Architecture Description
+++-===============================================-============================-============================-===================================================================================================
ii apache2 2.4.20-1
amd64 Apache HTTP Server
un apache2-api-20120211 <none>
<none> (no description available)
ii apache2-bin 2.4.20-1
amd64 Apache HTTP Server (modules and other binary
files)
ii apache2-data 2.4.20-1
all Apache HTTP Server (common files)
ii apache2-dbg 2.4.20-1
amd64 Apache debugging symbols
un apache2-doc <none>
<none> (no description available)
un apache2-suexec-custom <none>
<none> (no description available)
un apache2-suexec-pristine <none>
<none> (no description available)
ii apache2-utils 2.4.20-1
amd64 Apache HTTP Server (utility programs for web
servers)
un apache2.2-bin <none>
<none> (no description available)
un apache2.2-common <none>
<none> (no description available)
ii libapache2-mod-gnutls 0.7.4-2
amd64 Apache module for SSL and TLS encryption with
GnuTLS
ii libapache2-mod-gnutls-dbgsym 0.7.4-2
amd64 Debug symbols for libapache2-mod-gnutls
ii libgnutls-deb0-28:amd64 3.3.20-1
amd64 GNU TLS library - main runtime library
ii libgnutls-openssl27:amd64 3.4.11-4
amd64 GNU TLS library - OpenSSL wrapper
ii libgnutls30:amd64 3.4.11-4
amd64 GNU TLS library - main runtime library
un libgnutls30-dbg <none>
<none> (no description available)
ii libgnutls30-dbgsym:amd64 3.4.11-4
amd64 Debug symbols for libgnutls30
}}}
'''Apache MPM:'''
{{{
root at freedombox-vm1:~# a2query -M
prefork
}}}
'''Apache Site Configuration:'''
{{{
##
## On all sites, provide Plinth on a default path: /plinth
##
## Requires the following Apache modules to be enabled:
## mod_headers
## mod_proxy
## mod_proxy_http
##
<Location /plinth>
ProxyPass http://127.0.0.1:8000/plinth
## Send the scheme from user's request to enable Plinth to redirect
## URLs, set cookies, set absolute URLs (if any) properly.
RequestHeader set X-Forwarded-Proto 'https' env=HTTPS
## Allow traffic only from private networks
<RequireAny>
## IPv4 local addresses
Require ip 127.0.0.0/8
## IPv4 link local addresses
Require ip 169.254.0.0/16
## IPv4 class A private addresses
Require ip 10.0.0.0/8
## IPv4 class B private addresses
Require ip 172.16.0.0/12
## IPv4 class C private addresses
Require ip 192.168.0.0/16
## IPv6 local address
Require ip ::1
## IPv6 link local addresses
Require ip fe80::/10
## IPv6 private addresses
Require ip fc00::/7
</RequireAny>
</Location>
}}}
--
Ticket URL: <https://mod.gnutls.org/ticket/136>
mod_gnutls <https://mod.gnutls.org>
The apache httpd module for HTTPS using GnuTLS
More information about the mod_gnutls-devel
mailing list