[mod_gnutls-devel] [mod_gnutls] #136: Infinite loop while cleaning up GnuTLS session

mod_gnutls webmaster at mod.gnutls.org
Sat May 21 14:54:01 CEST 2016


#136: Infinite loop while cleaning up GnuTLS session
-------------------------------------+-------------------------------------
Reporter:                            |      Owner:
  http://gtlsbug.livejournal.com/    |  https://id.mayfirst.org/dkg
    Type:  defect                    |     Status:  new
Priority:  critical                  |  Component:  code
 Version:                            |   Keywords:
-------------------------------------+-------------------------------------
 I noticed that some apache2 processes take 100% CPU after a few requests.
 Stack trace shows mod-gnutls as the problem.  This problem causes embedded
 devices to slow down to an unusable state after using them for a while.

 A quick read tells us that cleanup_gnutls_session() getting stuck in a
 loop due to GNUTLS_E_INTERRUPTED or GNUTLS_E_AGAIN being returned by
 gnutls_bye().  After mgs_transport_write(), gnutls understands these
 errnos even though errno is not set during that function?

 Following is the debug information I collected.  I am available to get
 more information or try out patches if necessary.  Thanks in advance.

 ----

 '''How to Reproduce:'''

 The problem is easily reproducible on my machines by sending a lot of
 requests to Apache.  Most requests don't cause a problem while some lead
 to the problem.  I believe the problem occurs when using mod-gnutls and
 proxying connections.  Also this is more observable on pages taking time
 to load.

 $ ab -v 1 -c 8 -n 800 -H 'Cookie: XXX' https://freedombox-
 vm1/plinth/sys/users/

 '''Stack traces:'''

 I collected several stack traces by letting the process run for a while
 and then interrupting it again.

 {{{
 #0  mgs_transport_write (ptr=0x7f785b260028, buffer=<optimized out>,
 len=8023) at gnutls_io.c:871
 #1  0x00007f78669d86d5 in _gnutls_writev_emu (session=<optimized out>,
 session=<optimized out>, vec=<optimized out>, giovec_cnt=<optimized out>,
 giovec=<optimized out>, fd=<optimized out>) at gnutls_buffers.c:447
 #2  _gnutls_writev (total=8023, giovec_cnt=<optimized out>,
 giovec=0x7fff2ce06a20, session=0x564c16d28710) at gnutls_buffers.c:505
 #3  _gnutls_io_write_flush (session=session at entry=0x564c16d28710) at
 gnutls_buffers.c:699
 #4  0x00007f78669d4d5c in gnutls_bye (session=0x564c16d28710,
 how=how at entry=GNUTLS_SHUT_WR) at gnutls_record.c:281
 #5  0x00007f78641a329e in cleanup_gnutls_session (data=0x7f785b260028) at
 gnutls_hooks.c:722
 #6  0x00007f78693e1cfe in run_cleanups (cref=0x7f785b285098) at
 /tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:2352
 #7  apr_pool_clear (pool=0x7f785b285028) at
 /tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:762
 #8  0x00007f78609ab4ce in child_main (child_num_arg=child_num_arg at entry=7,
 child_bucket=child_bucket at entry=0) at prefork.c:616
 #9  0x00007f78609ab974 in make_child (s=0x7f7869c79470, slot=7, bucket=0)
 at prefork.c:824
 #10 0x00007f78609ac85d in perform_idle_server_maintenance (p=<optimized
 out>) at prefork.c:932
 #11 prefork_run (_pconf=<optimized out>, plog=<optimized out>,
 s=<optimized out>) at prefork.c:1128
 #12 0x0000564c15541eae in ap_run_mpm (pconf=0x7f7869ca9028,
 plog=0x7f7869c76028, s=0x7f7869c79470) at mpm_common.c:94
 #13 0x0000564c1553b516 in main (argc=3, argv=0x7fff2ce070c8) at main.c:778

 #0  write_flush (ctxt=ctxt at entry=0x7f785b260028) at gnutls_io.c:624
 #1  0x00007f786419ee32 in mgs_transport_write (ptr=0x7f785b260028,
 buffer=<optimized out>, len=8023) at gnutls_io.c:871
 #2  0x00007f78669d86d5 in _gnutls_writev_emu (session=<optimized out>,
 session=<optimized out>, vec=<optimized out>, giovec_cnt=<optimized out>,
 giovec=<optimized out>, fd=<optimized out>) at gnutls_buffers.c:447
 #3  _gnutls_writev (total=8023, giovec_cnt=<optimized out>,
 giovec=0x7fff2ce06a20, session=0x564c16d28710) at gnutls_buffers.c:505
 #4  _gnutls_io_write_flush (session=session at entry=0x564c16d28710) at
 gnutls_buffers.c:699
 #5  0x00007f78669d4d5c in gnutls_bye (session=0x564c16d28710,
 how=how at entry=GNUTLS_SHUT_WR) at gnutls_record.c:281
 #6  0x00007f78641a329e in cleanup_gnutls_session (data=0x7f785b260028) at
 gnutls_hooks.c:722
 #7  0x00007f78693e1cfe in run_cleanups (cref=0x7f785b285098) at
 /tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:2352
 #8  apr_pool_clear (pool=0x7f785b285028) at
 /tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:762
 #9  0x00007f78609ab4ce in child_main (child_num_arg=child_num_arg at entry=7,
 child_bucket=child_bucket at entry=0) at prefork.c:616
 #10 0x00007f78609ab974 in make_child (s=0x7f7869c79470, slot=7, bucket=0)
 at prefork.c:824
 #11 0x00007f78609ac85d in perform_idle_server_maintenance (p=<optimized
 out>) at prefork.c:932
 #12 prefork_run (_pconf=<optimized out>, plog=<optimized out>,
 s=<optimized out>) at prefork.c:1128
 #13 0x0000564c15541eae in ap_run_mpm (pconf=0x7f7869ca9028,
 plog=0x7f7869c76028, s=0x7f7869c79470) at mpm_common.c:94
 #14 0x0000564c1553b516 in main (argc=3, argv=0x7fff2ce070c8) at main.c:778

 #0  0x00007f78696022e0 in apr_bucket_alloc at plt () from /usr/lib/x86_64
 -linux-gnu/libaprutil-1.so.0
 #1  0x00007f7869604591 in apr_bucket_flush_create () from /usr/lib/x86_64
 -linux-gnu/libaprutil-1.so.0
 #2  0x00007f786419dc1c in write_flush (ctxt=ctxt at entry=0x7f785b260028) at
 gnutls_io.c:642
 #3  0x00007f786419ee32 in mgs_transport_write (ptr=0x7f785b260028,
 buffer=<optimized out>, len=8023) at gnutls_io.c:871
 #4  0x00007f78669d86d5 in _gnutls_writev_emu (session=<optimized out>,
 session=<optimized out>, vec=<optimized out>, giovec_cnt=<optimized out>,
 giovec=<optimized out>, fd=<optimized out>) at gnutls_buffers.c:447
 #5  _gnutls_writev (total=8023, giovec_cnt=<optimized out>,
 giovec=0x7fff2ce06a20, session=0x564c16d28710) at gnutls_buffers.c:505
 #6  _gnutls_io_write_flush (session=session at entry=0x564c16d28710) at
 gnutls_buffers.c:699
 #7  0x00007f78669d4d5c in gnutls_bye (session=0x564c16d28710,
 how=how at entry=GNUTLS_SHUT_WR) at gnutls_record.c:281
 #8  0x00007f78641a329e in cleanup_gnutls_session (data=0x7f785b260028) at
 gnutls_hooks.c:722
 #9  0x00007f78693e1cfe in run_cleanups (cref=0x7f785b285098) at
 /tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:2352
 #10 apr_pool_clear (pool=0x7f785b285028) at
 /tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:762
 #11 0x00007f78609ab4ce in child_main (child_num_arg=child_num_arg at entry=7,
 child_bucket=child_bucket at entry=0) at prefork.c:616
 #12 0x00007f78609ab974 in make_child (s=0x7f7869c79470, slot=7, bucket=0)
 at prefork.c:824
 #13 0x00007f78609ac85d in perform_idle_server_maintenance (p=<optimized
 out>) at prefork.c:932
 #14 prefork_run (_pconf=<optimized out>, plog=<optimized out>,
 s=<optimized out>) at prefork.c:1128
 #15 0x0000564c15541eae in ap_run_mpm (pconf=0x7f7869ca9028,
 plog=0x7f7869c76028, s=0x7f7869c79470) at mpm_common.c:94
 #16 0x0000564c1553b516 in main (argc=3, argv=0x7fff2ce070c8) at main.c:778

 #0  _mbuffer_head_get_next (cur=0x564c16d2c9a0,
 msg=msg at entry=0x7fff2ce06a10) at gnutls_mbuffers.c:191
 #1  0x00007f78669d83bd in _gnutls_io_write_flush
 (session=session at entry=0x564c16d28710) at gnutls_buffers.c:682
 #2  0x00007f78669d4d5c in gnutls_bye (session=0x564c16d28710,
 how=how at entry=GNUTLS_SHUT_WR) at gnutls_record.c:281
 #3  0x00007f78641a329e in cleanup_gnutls_session (data=0x7f785b260028) at
 gnutls_hooks.c:722
 #4  0x00007f78693e1cfe in run_cleanups (cref=0x7f785b285098) at
 /tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:2352
 #5  apr_pool_clear (pool=0x7f785b285028) at
 /tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:762
 #6  0x00007f78609ab4ce in child_main (child_num_arg=child_num_arg at entry=7,
 child_bucket=child_bucket at entry=0) at prefork.c:616
 #7  0x00007f78609ab974 in make_child (s=0x7f7869c79470, slot=7, bucket=0)
 at prefork.c:824
 #8  0x00007f78609ac85d in perform_idle_server_maintenance (p=<optimized
 out>) at prefork.c:932
 #9  prefork_run (_pconf=<optimized out>, plog=<optimized out>,
 s=<optimized out>) at prefork.c:1128
 #10 0x0000564c15541eae in ap_run_mpm (pconf=0x7f7869ca9028,
 plog=0x7f7869c76028, s=0x7f7869c79470) at mpm_common.c:94
 #11 0x0000564c1553b516 in main (argc=3, argv=0x7fff2ce070c8) at main.c:778

 #0  0x00007f7869602caa in apr_brigade_cleanup () from /usr/lib/x86_64
 -linux-gnu/libaprutil-1.so.0
 #1  0x0000564c1555926a in ap_core_output_filter (f=0x7f785b2641e0,
 new_bb=0x7f785b2640e8) at core_filters.c:385
 #2  0x00007f786419dc4d in write_flush (ctxt=ctxt at entry=0x7f785b260028) at
 gnutls_io.c:645
 #3  0x00007f786419ee32 in mgs_transport_write (ptr=0x7f785b260028,
 buffer=<optimized out>, len=8023) at gnutls_io.c:871
 #4  0x00007f78669d86d5 in _gnutls_writev_emu (session=<optimized out>,
 session=<optimized out>, vec=<optimized out>, giovec_cnt=<optimized out>,
 giovec=<optimized out>, fd=<optimized out>) at gnutls_buffers.c:447
 #5  _gnutls_writev (total=8023, giovec_cnt=<optimized out>,
 giovec=0x7fff2ce06a20, session=0x564c16d28710) at gnutls_buffers.c:505
 #6  _gnutls_io_write_flush (session=session at entry=0x564c16d28710) at
 gnutls_buffers.c:699
 #7  0x00007f78669d4d5c in gnutls_bye (session=0x564c16d28710,
 how=how at entry=GNUTLS_SHUT_WR) at gnutls_record.c:281
 #8  0x00007f78641a329e in cleanup_gnutls_session (data=0x7f785b260028) at
 gnutls_hooks.c:722
 #9  0x00007f78693e1cfe in run_cleanups (cref=0x7f785b285098) at
 /tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:2352
 #10 apr_pool_clear (pool=0x7f785b285028) at
 /tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:762
 #11 0x00007f78609ab4ce in child_main (child_num_arg=child_num_arg at entry=7,
 child_bucket=child_bucket at entry=0) at prefork.c:616
 #12 0x00007f78609ab974 in make_child (s=0x7f7869c79470, slot=7, bucket=0)
 at prefork.c:824
 #13 0x00007f78609ac85d in perform_idle_server_maintenance (p=<optimized
 out>) at prefork.c:932
 #14 prefork_run (_pconf=<optimized out>, plog=<optimized out>,
 s=<optimized out>) at prefork.c:1128
 #15 0x0000564c15541eae in ap_run_mpm (pconf=0x7f7869ca9028,
 plog=0x7f7869c76028, s=0x7f7869c79470) at mpm_common.c:94
 #16 0x0000564c1553b516 in main (argc=3, argv=0x7fff2ce070c8) at main.c:778
 }}}

 '''Software versions:'''

 {{{
 root at freedombox-vm1:~# dpkg -l libapache2-mod-gnutls* apache2* libgnutls*
 Desired=Unknown/Install/Remove/Purge/Hold
 | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-
 pend
 |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
 ||/ Name                                            Version
 Architecture                 Description
 +++-===============================================-============================-============================-===================================================================================================
 ii  apache2                                         2.4.20-1
 amd64                        Apache HTTP Server
 un  apache2-api-20120211                            <none>
 <none>                       (no description available)
 ii  apache2-bin                                     2.4.20-1
 amd64                        Apache HTTP Server (modules and other binary
 files)
 ii  apache2-data                                    2.4.20-1
 all                          Apache HTTP Server (common files)
 ii  apache2-dbg                                     2.4.20-1
 amd64                        Apache debugging symbols
 un  apache2-doc                                     <none>
 <none>                       (no description available)
 un  apache2-suexec-custom                           <none>
 <none>                       (no description available)
 un  apache2-suexec-pristine                         <none>
 <none>                       (no description available)
 ii  apache2-utils                                   2.4.20-1
 amd64                        Apache HTTP Server (utility programs for web
 servers)
 un  apache2.2-bin                                   <none>
 <none>                       (no description available)
 un  apache2.2-common                                <none>
 <none>                       (no description available)
 ii  libapache2-mod-gnutls                           0.7.4-2
 amd64                        Apache module for SSL and TLS encryption with
 GnuTLS
 ii  libapache2-mod-gnutls-dbgsym                    0.7.4-2
 amd64                        Debug symbols for libapache2-mod-gnutls
 ii  libgnutls-deb0-28:amd64                         3.3.20-1
 amd64                        GNU TLS library - main runtime library
 ii  libgnutls-openssl27:amd64                       3.4.11-4
 amd64                        GNU TLS library - OpenSSL wrapper
 ii  libgnutls30:amd64                               3.4.11-4
 amd64                        GNU TLS library - main runtime library
 un  libgnutls30-dbg                                 <none>
 <none>                       (no description available)
 ii  libgnutls30-dbgsym:amd64                        3.4.11-4
 amd64                        Debug symbols for libgnutls30
 }}}

 '''Apache MPM:'''

 {{{
 root at freedombox-vm1:~# a2query -M
 prefork
 }}}

 '''Apache Site Configuration:'''

 {{{
 ##
 ## On all sites, provide Plinth on a default path: /plinth
 ##
 ## Requires the following Apache modules to be enabled:
 ##   mod_headers
 ##   mod_proxy
 ##   mod_proxy_http
 ##
 <Location /plinth>
     ProxyPass        http://127.0.0.1:8000/plinth
     ## Send the scheme from user's request to enable Plinth to redirect
     ## URLs, set cookies, set absolute URLs (if any) properly.
     RequestHeader    set X-Forwarded-Proto 'https' env=HTTPS

     ## Allow traffic only from private networks
     <RequireAny>
         ## IPv4 local addresses
         Require ip 127.0.0.0/8

         ## IPv4 link local addresses
         Require ip 169.254.0.0/16

         ## IPv4 class A private addresses
         Require ip 10.0.0.0/8

         ## IPv4 class B private addresses
         Require ip 172.16.0.0/12

         ## IPv4 class C private addresses
         Require ip 192.168.0.0/16

         ## IPv6 local address
         Require ip ::1

         ## IPv6 link local addresses
         Require ip fe80::/10

         ## IPv6 private addresses
         Require ip fc00::/7
     </RequireAny>
 </Location>
 }}}

--
Ticket URL: <https://mod.gnutls.org/ticket/136>
mod_gnutls <https://mod.gnutls.org>
The apache httpd module for HTTPS using GnuTLS



More information about the mod_gnutls-devel mailing list