From rokclimb15 at gmail.com Mon Jan 2 23:42:58 2017 From: rokclimb15 at gmail.com (Brian Morton) Date: Mon, 2 Jan 2017 17:42:58 -0500 Subject: [mod_gnutls-devel] Segfault in 0.8.1 test 24 on i386 Message-ID: Hi mod_gnutls dev team, I've been working on diagnosing this FTBFS bug in Ubuntu https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1597450. Mod_gnutls fails to build on i386 due to some string format issues fixed in 0.8.1. Once those simple issues are fixed, test 24 fails due to a segfault in Apache. This seems to be true whether using Debian/Ubuntu sources or the latest from mod_gnutls. The crash appears to be due to a buffer overflow. Backtrace indicates several libs are involved including gnutls, softhsm2, and p11-kit. The issue very likely is within one of those libraries rather than mod_gnutls, but I'm trying to nail it down further so I thought I'd start here. I've been working on it for some time but I've reached the limit of my ability to diagnose the issue. I've managed to attach gdb to Apache during the test and can traverse the stack to observe execution, but I can't spot the issue. Running Apache with -X (no forking) gives a similar crash, but with GCC SSP being triggered. I also cannot break at the right point to observe the stack canary being overwritten since my breakpoint is unloaded by gdb due to module unload. Could anyone please point me in the right direction on how to track this down? I'm interested in learning more about using gdb to debug these types of tricky memory issues. I've attached the full backtrace. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- ?bmorton at ubuntu:~/apache-crash4$ gdb /usr/sbin/apache2 CoreDump GNU gdb (Ubuntu 7.12-0ubuntu3) 7.12 Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at: . For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/apache2...Reading symbols from /usr/lib/debug//us r/sbin/apache2...done. done. [New LWP 9005] warning: Unexpected size of section `.reg-xstate/9005' in core file. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/apache2 -f /home/bmorton/mod-gnutls-0.8.0/test/ tests/24_pkcs11_cert/a'. Program terminated with signal SIGSEGV, Segmentation fault. warning: Unexpected size of section `.reg-xstate/9005' in core file. #0 0xb7136747 in proxy_C_DecryptVerifyUpdate ( self=0xb64cb9f0 >::get() const+28>, handle=2175480616, enc_part=0xbfa945e8 "\bF\251\277?L\266\244\261T\266\330.\253\201\030F\251\27 7?L\266\060\243\252\201(3\253\201\030F\251\277?L\266\244\261T\266\234\253T\266\07 0F\251\277\363\264L\266\070\003\252\201\234\253T\266HF\251\277\342\264L\266\---T ype to continue, or q to quit--- 244\261T\266\234\253T\266XF\251\277\342\267L\266\070\003\252\201\001", enc_part_len=3058480322, part=0xb64cb968 ::_M_head(std::_He ad_base<0u, MutexFactory*, false> const&)+8> "\005\064\362\a", part_len=0x46505845) at p11-kit/proxy.c:1452 1452 p11-kit/proxy.c: No such file or directory. (gdb) bt full #0 0xb7136747 in proxy_C_DecryptVerifyUpdate (self=0xb64cb9f0 >::get() const+28>, handle=2175480616, enc_part=0xbfa945e8 "\bF\251\277?L\266\244\261T\266\330.\253\201\030F\251\277?L\266\060\243\252\201(3\253\201\030F\251\277?L\266\244\261T\266\234\253T\266\070F\251\277\363\264L\266\070\003\252\201\234\253T\266HF\251\277\342\264L\266\244\261T\266\234\253T\266XF\251\277\342\267L\266\070\003\252\201\001", enc_part_len=3058480322, part=0xb64cb968 ::_M_head(std::_Head_base<0u, MutexFactory*, false> const&)+8> "\005\064\362\a", part_len=0x46505845) at p11-kit/proxy.c:1452 state = 0x1 map = {wrap_slot = 2175480616, real_slot = 3059002268, funcs = 0xbfa94608} rv = 3058481387 #1 0xb64cb4cc in Mutex::~Mutex (this=0x81aa0338, __in_chrg=) at MutexFactory.cpp:53 No locals. #2 0xb64cb4f3 in Mutex::~Mutex (this=0x81aa0338, __in_chrg=) at MutexFactory.cpp:55 No locals. #3 0xb64cb7e2 in MutexFactory::recycleMutex (this=0x81aaa330, mutex=0x81aa0338) at MutexFactory.cpp:130 No locals. #4 0xb64eaea2 in HandleManager::~HandleManager (this=0x81ab32e8, __in_chrg=) at HandleManager.cpp:60 No locals. #5 0xb64eaeef in HandleManager::~HandleManager (this=0x81ab32e8, __in_chrg=) at HandleManager.cpp:61 No locals. #6 0xb649a4da in SoftHSM::~SoftHSM (this=0x81aaa310, __in_chrg=) at SoftHSM.cpp:335 No locals. #7 0xb649a5bd in SoftHSM::~SoftHSM (this=0x81aaa310, __in_chrg=) at SoftHSM.cpp:340 No locals. #8 0xb64c21b2 in std::default_delete::operator() (this=0xb654b1b0 , __ptr=0x81aaa310) at /usr/include/c++/6/bits/unique_ptr.h:76 No locals. #9 0xb64c1a61 in std::unique_ptr >::~unique_ptr (this=0xb654b1b0 , __in_chrg=) at /usr/include/c++/6/bits/unique_ptr.h:239 __ptr = @0xb654b1b0: 0x81aaa310 #10 0xb74daaab in __run_exit_handlers (status=0, listp=0xb76623dc <__exit_funcs>, run_list_atexit=true, run_dtors=true) at exit.c:83 atfct = onfct = cxafct = f = #11 0xb74dab11 in __GI_exit (status=0) at exit.c:105 No locals. #12 0xb76a765f in apr_proc_detach (daemonize=1) at ./threadproc/unix/procsup.c:32 x = #13 0xb7409d3f in worker_pre_config (pconf=0xb775d018, plog=0xb741b018, ptemp=0xb7417018) at worker.c:2151 no_detach = 0 debug = foreground = rv = userdata_key = 0xb740cf26 "mpm_worker_module" #14 0x80131f9f in ap_run_pre_config (pconf=0xb775d018, plog=0xb741b018, ptemp=0xb7417018) at config.c:89 pHook = n = 3 rv = 0 #15 0x8010fd70 in main (argc=, argv=) at main.c:739 c = 102 'f' showcompile = showdirectives = confname = def_server_root = temp_error_log = error = pconf = plog = 0xb741b018 ptemp = 0xb7417018 pcommands = 0xb7421018 opt = 0xb74210b8 rv = mod = opt_arg = 0xbfa95ac2 "/home/bmorton/mod-gnutls-0.8.0/test/tests/24_pkcs11_cert/apache.conf" signal_server = rc = From dkg at fifthhorseman.net Tue Jan 3 05:46:23 2017 From: dkg at fifthhorseman.net (Daniel Kahn Gillmor) Date: Mon, 02 Jan 2017 23:46:23 -0500 Subject: [mod_gnutls-devel] Segfault in 0.8.1 test 24 on i386 In-Reply-To: References: Message-ID: <87wpecoin4.fsf@alice.fifthhorseman.net> On Mon 2017-01-02 17:42:58 -0500, Brian Morton wrote: > I've been working on diagnosing this FTBFS bug in Ubuntu > https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1597450. > > Mod_gnutls fails to build on i386 due to some string format issues fixed in > 0.8.1. Once those simple issues are fixed, test 24 fails due to a segfault > in Apache. This seems to be true whether using Debian/Ubuntu sources or the > latest from mod_gnutls. The crash appears to be due to a buffer overflow. > Backtrace indicates several libs are involved including gnutls, softhsm2, > and p11-kit. The issue very likely is within one of those libraries rather > than mod_gnutls, but I'm trying to nail it down further so I thought I'd > start here. fwiw, we're not seeing these issues on debian unstable on i386 with the packaged version 0.8.1-1. It looks like some sort of failure in process cleanup related to softhsm2, but the versions of softhsm2 (2.2.0-1) look like they're the same in ubuntu zesty and debian unstable. i haven't been able to tease out any better diagnosis myself yet, sorry! --dkg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From rokclimb15 at gmail.com Tue Jan 3 06:23:54 2017 From: rokclimb15 at gmail.com (Brian Morton) Date: Tue, 3 Jan 2017 00:23:54 -0500 Subject: [mod_gnutls-devel] Segfault in 0.8.1 test 24 on i386 In-Reply-To: <87wpecoin4.fsf@alice.fifthhorseman.net> References: <87wpecoin4.fsf@alice.fifthhorseman.net> Message-ID: <26594742-7A57-411C-8E95-A6A6D26F0527@gmail.com> Great info! Apache2 and libgnutls30 are slightly newer in Sid currently. I'll see if I can pin some older versions in unstable and reproduce the crash. > On Jan 2, 2017, at 11:46 PM, Daniel Kahn Gillmor wrote: > >> On Mon 2017-01-02 17:42:58 -0500, Brian Morton wrote: >> I've been working on diagnosing this FTBFS bug in Ubuntu >> https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1597450. >> >> Mod_gnutls fails to build on i386 due to some string format issues fixed in >> 0.8.1. Once those simple issues are fixed, test 24 fails due to a segfault >> in Apache. This seems to be true whether using Debian/Ubuntu sources or the >> latest from mod_gnutls. The crash appears to be due to a buffer overflow. >> Backtrace indicates several libs are involved including gnutls, softhsm2, >> and p11-kit. The issue very likely is within one of those libraries rather >> than mod_gnutls, but I'm trying to nail it down further so I thought I'd >> start here. > > fwiw, we're not seeing these issues on debian unstable on i386 with the > packaged version 0.8.1-1. It looks like some sort of failure in process > cleanup related to softhsm2, but the versions of softhsm2 (2.2.0-1) look > like they're the same in ubuntu zesty and debian unstable. > > i haven't been able to tease out any better diagnosis myself yet, sorry! > > --dkg From rokclimb15 at gmail.com Fri Jan 6 05:58:37 2017 From: rokclimb15 at gmail.com (Brian Morton) Date: Thu, 5 Jan 2017 23:58:37 -0500 Subject: [mod_gnutls-devel] Segfault in 0.8.1 test 24 on i386 In-Reply-To: <26594742-7A57-411C-8E95-A6A6D26F0527@gmail.com> References: <87wpecoin4.fsf@alice.fifthhorseman.net> <26594742-7A57-411C-8E95-A6A6D26F0527@gmail.com> Message-ID: I'm fairly stuck on this one. Tried running old versions of packages on Sid and can't reproduce the crash. I also build apache2 and libgnutls30 from upstream sources and the crash still occurs on Zesty. Disabled apparmor, and compared lots of build logs between Debian and Ubuntu to see if there are any differences. I don't see any Debian patches in key dependencies that would explain the ability to build. At this point I'm starting to wonder if it's the toolchain or something in the kernel. Perhaps there's a simpler explanation? On Tue, Jan 3, 2017 at 12:23 AM, Brian Morton wrote: > Great info! > > Apache2 and libgnutls30 are slightly newer in Sid currently. I'll see if I > can pin some older versions in unstable and reproduce the crash. > > > On Jan 2, 2017, at 11:46 PM, Daniel Kahn Gillmor > wrote: > > > >> On Mon 2017-01-02 17:42:58 -0500, Brian Morton wrote: > >> I've been working on diagnosing this FTBFS bug in Ubuntu > >> https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1597450. > >> > >> Mod_gnutls fails to build on i386 due to some string format issues > fixed in > >> 0.8.1. Once those simple issues are fixed, test 24 fails due to a > segfault > >> in Apache. This seems to be true whether using Debian/Ubuntu sources or > the > >> latest from mod_gnutls. The crash appears to be due to a buffer > overflow. > >> Backtrace indicates several libs are involved including gnutls, > softhsm2, > >> and p11-kit. The issue very likely is within one of those libraries > rather > >> than mod_gnutls, but I'm trying to nail it down further so I thought I'd > >> start here. > > > > fwiw, we're not seeing these issues on debian unstable on i386 with the > > packaged version 0.8.1-1. It looks like some sort of failure in process > > cleanup related to softhsm2, but the versions of softhsm2 (2.2.0-1) look > > like they're the same in ubuntu zesty and debian unstable. > > > > i haven't been able to tease out any better diagnosis myself yet, sorry! > > > > --dkg > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas2.klute at uni-dortmund.de Sun Jan 8 17:57:01 2017 From: thomas2.klute at uni-dortmund.de (Thomas Klute) Date: Sun, 8 Jan 2017 17:57:01 +0100 Subject: [mod_gnutls-devel] Bugfix release: mod_gnutls 0.8.2 Message-ID: <80a05f1d-4649-ec39-e196-663d7c20dd31@uni-dortmund.de> Hi everyone, I've released mod_gnutls 0.8.2. It fixes two test suite bugs affecting builds using Apache version 2.4.24 (or newer) and on big endian architectures, respectively. As usual, the source archive and signature are available from https://mod.gnutls.org/downloads/ and the signed tag mod_gnutls/0.8.2 in the git repositories listed below. https://mod.gnutls.org/git/mod_gnutls https://github.com/airtower-luna/mod_gnutls.git Best regards, Thomas From thomas2.klute at uni-dortmund.de Sun Jan 8 18:48:41 2017 From: thomas2.klute at uni-dortmund.de (Thomas Klute) Date: Sun, 8 Jan 2017 18:48:41 +0100 Subject: [mod_gnutls-devel] Segfault in 0.8.1 test 24 on i386 In-Reply-To: References: <87wpecoin4.fsf@alice.fifthhorseman.net> <26594742-7A57-411C-8E95-A6A6D26F0527@gmail.com> Message-ID: <595d7815-0bc6-e1f1-a915-9b27d5b2929a@uni-dortmund.de> My observation from a Yakkety amd64 system: A segfault occurs reliably on Apache shutdown (so the test still passes) with libsofthsm2.so from the libsofthsm2 package, but not with libsofthsm2.so locally built from the SoftHSMv2 git repository at the 2.1.0 tag. Valgrind reports 4 lost blocks of memory while loading the Ubuntu SoftHSM, as opposed to 1 with the one built from Git. I also downloaded the Ubuntu source package and built that locally. Again: Segfault, and identical memory leaks reported by Valgrind. This excludes problems with the general system environment (it is the same) and in dependencies (according to ldd both libs link against the same system libs). Based on this I can only assume that there is an issue specific to the Ubuntu build process for SoftHSM. Regards, Thomas Am 06.01.2017 um 05:58 schrieb Brian Morton: > I'm fairly stuck on this one. Tried running old versions of packages on Sid > and can't reproduce the crash. I also build apache2 and libgnutls30 from > upstream sources and the crash still occurs on Zesty. Disabled apparmor, > and compared lots of build logs between Debian and Ubuntu to see if there > are any differences. I don't see any Debian patches in key dependencies > that would explain the ability to build. > > At this point I'm starting to wonder if it's the toolchain or something in > the kernel. Perhaps there's a simpler explanation? > > On Tue, Jan 3, 2017 at 12:23 AM, Brian Morton wrote: > >> Great info! >> >> Apache2 and libgnutls30 are slightly newer in Sid currently. I'll see if I >> can pin some older versions in unstable and reproduce the crash. >> >>> On Jan 2, 2017, at 11:46 PM, Daniel Kahn Gillmor >> wrote: >>> >>>> On Mon 2017-01-02 17:42:58 -0500, Brian Morton wrote: >>>> I've been working on diagnosing this FTBFS bug in Ubuntu >>>> https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1597450. >>>> >>>> Mod_gnutls fails to build on i386 due to some string format issues >> fixed in >>>> 0.8.1. Once those simple issues are fixed, test 24 fails due to a >> segfault >>>> in Apache. This seems to be true whether using Debian/Ubuntu sources or >> the >>>> latest from mod_gnutls. The crash appears to be due to a buffer >> overflow. >>>> Backtrace indicates several libs are involved including gnutls, >> softhsm2, >>>> and p11-kit. The issue very likely is within one of those libraries >> rather >>>> than mod_gnutls, but I'm trying to nail it down further so I thought I'd >>>> start here. >>> >>> fwiw, we're not seeing these issues on debian unstable on i386 with the >>> packaged version 0.8.1-1. It looks like some sort of failure in process >>> cleanup related to softhsm2, but the versions of softhsm2 (2.2.0-1) look >>> like they're the same in ubuntu zesty and debian unstable. >>> >>> i haven't been able to tease out any better diagnosis myself yet, sorry! >>> >>> --dkg >> > > > > _______________________________________________ > mod_gnutls-devel mailing list > mod_gnutls-devel at lists.gnutls.org > http://lists.gnupg.org/mailman/listinfo/mod_gnutls-devel > -- Dipl.-Ing. Thomas Klute E-Mail: thomas2.klute at uni-dortmund.de Tel.: +49 231 58680474 From rokclimb15 at gmail.com Sun Jan 8 18:57:53 2017 From: rokclimb15 at gmail.com (Brian Morton) Date: Sun, 8 Jan 2017 12:57:53 -0500 Subject: [mod_gnutls-devel] Segfault in 0.8.1 test 24 on i386 In-Reply-To: <595d7815-0bc6-e1f1-a915-9b27d5b2929a@uni-dortmund.de> References: <87wpecoin4.fsf@alice.fifthhorseman.net> <26594742-7A57-411C-8E95-A6A6D26F0527@gmail.com> <595d7815-0bc6-e1f1-a915-9b27d5b2929a@uni-dortmund.de> Message-ID: Thomas, This is very helpful, thank you. I noticed you are building on an amd64 system however, while this test failure only occurs on i386. My last test of Zesty sources on i386 with valgrind also reported one block lost but segfaults during the test. I'll definitely look more carefully at the softhsm2 package. Offhand, it's hard to think of how it might differ from Debian since the package is in sync with sid (2.2.0). I'll review the package build logs to see if the configure differs somehow. The biggest clue I have so far is Debian sid i386 passes this test and the only two direct dependencies that differ in version are gnutls and apache, but I've built those from latest release source and the crash still occurs. My idea at this point is some other second or third level dependency like openssl. Thanks, Brian On Sun, Jan 8, 2017 at 12:48 PM, Thomas Klute wrote: > My observation from a Yakkety amd64 system: A segfault occurs reliably > on Apache shutdown (so the test still passes) with libsofthsm2.so from > the libsofthsm2 package, but not with libsofthsm2.so locally built from > the SoftHSMv2 git repository at the 2.1.0 tag. > > Valgrind reports 4 lost blocks of memory while loading the Ubuntu > SoftHSM, as opposed to 1 with the one built from Git. I also downloaded > the Ubuntu source package and built that locally. Again: Segfault, and > identical memory leaks reported by Valgrind. > > This excludes problems with the general system environment (it is the > same) and in dependencies (according to ldd both libs link against the > same system libs). Based on this I can only assume that there is an > issue specific to the Ubuntu build process for SoftHSM. > > Regards, > Thomas > > Am 06.01.2017 um 05:58 schrieb Brian Morton: > > I'm fairly stuck on this one. Tried running old versions of packages on > Sid > > and can't reproduce the crash. I also build apache2 and libgnutls30 from > > upstream sources and the crash still occurs on Zesty. Disabled apparmor, > > and compared lots of build logs between Debian and Ubuntu to see if there > > are any differences. I don't see any Debian patches in key dependencies > > that would explain the ability to build. > > > > At this point I'm starting to wonder if it's the toolchain or something > in > > the kernel. Perhaps there's a simpler explanation? > > > > On Tue, Jan 3, 2017 at 12:23 AM, Brian Morton > wrote: > > > >> Great info! > >> > >> Apache2 and libgnutls30 are slightly newer in Sid currently. I'll see > if I > >> can pin some older versions in unstable and reproduce the crash. > >> > >>> On Jan 2, 2017, at 11:46 PM, Daniel Kahn Gillmor < > dkg at fifthhorseman.net> > >> wrote: > >>> > >>>> On Mon 2017-01-02 17:42:58 -0500, Brian Morton wrote: > >>>> I've been working on diagnosing this FTBFS bug in Ubuntu > >>>> https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1597450. > >>>> > >>>> Mod_gnutls fails to build on i386 due to some string format issues > >> fixed in > >>>> 0.8.1. Once those simple issues are fixed, test 24 fails due to a > >> segfault > >>>> in Apache. This seems to be true whether using Debian/Ubuntu sources > or > >> the > >>>> latest from mod_gnutls. The crash appears to be due to a buffer > >> overflow. > >>>> Backtrace indicates several libs are involved including gnutls, > >> softhsm2, > >>>> and p11-kit. The issue very likely is within one of those libraries > >> rather > >>>> than mod_gnutls, but I'm trying to nail it down further so I thought > I'd > >>>> start here. > >>> > >>> fwiw, we're not seeing these issues on debian unstable on i386 with the > >>> packaged version 0.8.1-1. It looks like some sort of failure in > process > >>> cleanup related to softhsm2, but the versions of softhsm2 (2.2.0-1) > look > >>> like they're the same in ubuntu zesty and debian unstable. > >>> > >>> i haven't been able to tease out any better diagnosis myself yet, > sorry! > >>> > >>> --dkg > >> > > > > > > > > _______________________________________________ > > mod_gnutls-devel mailing list > > mod_gnutls-devel at lists.gnutls.org > > http://lists.gnupg.org/mailman/listinfo/mod_gnutls-devel > > > > -- > Dipl.-Ing. Thomas Klute > > E-Mail: thomas2.klute at uni-dortmund.de > Tel.: +49 231 58680474 > -------------- next part -------------- An HTML attachment was scrubbed... URL: