GnuTLS/NSS interop in Exim 4.80 RC

[Phil Pennock]

Folks,

Short: NSS client to GnuTLS 2.12 (but not 2.8) fails TLS negotiation,
GnuTLS dropping connection after reporting receiving a phantom packet.

Long:

For the Exim 4.80 release, currently in Release Candidate, I re-did the
GnuTLS integration to stop using APIs which gave deprecation warnings.
As part of this, I removed the hard-coded lists of algorithms from Exim,
instead delegating that task to GnuTLS, and passing the Exim
tls_require_ciphers string to gnutls_priority_init().

Things had been going well in the Release Candidates, but we now have a
release blocker.  It seems that Thunderbird (NSS security library) can
not set up a TLS session with GnuTLS 2.12.18.  (I saw the .19
announcement, to me it doesn't look as though there's anything
relevant?)

Bug in 2.12.14 and 2.12.18, seen by two people (myself one of them); but
not in 2.8.5.  No problems observed with OpenSSL or GnuTLS clients, just
NSS.

Mail thread starts at:
  https://lists.exim.org/lurker/message/20120520.040118.edd7eecb.en.html
overview:
  https://lists.exim.org/lurker/thread/20120520.040118.edd7eecb.en.html#i20120520.040118.edd7eecb
Protocol dump in my mail:
  https://lists.exim.org/lurker/message/20120520.092423.0e38168b.en.html

It looks as though GnuTLS is claiming to have received a packet of
length 9, which ssltap does not see.  Could this be unconsumed data from
a previous packet?

Per ssltap, "openssl s_client" sent extensions:
  ec_point_formats
  elliptic_curves
  session_ticket
  signature_algorithms
  type 15
while Thunderbird sent extensions:
  server_name
  elliptic_curves
  ec_point_formats
  session_ticket

It's not server_name, we handle that just fine (and can now do SNI
dispatch for key/cert selection).  I see elliptic_curves and
ec_point_formats sent by both, so my "wild uneducated guess" was
probably wrong.

[source availability notes below]

Thanks for any help you can provide,
-Phil

On the off-chance that any GnuTLS devs run Exim themselves, I'll point
to our source; I don't expect you to debug our code, but I don't think
I'm being too arrogant in thinking someone may already be running Exim,
so I can point you at it to let you build yourselves to see the debug
messages.

You can add EXIM_GNUTLS_LIBRARY_LOG_LEVEL= to Local/Makefile with a
value accepted by gnutls_global_set_log_level() to have a callback
registered with gnutls_global_set_log_function(); "exim -d+tls" asks for
the TLS-specific debug messages, and the GnuTLS library messages will be
included.  My test builds just set "EXIM_GNUTLS_LIBRARY_LOG_LEVEL=9".

  4.80 RC2 announcement:
   https://lists.exim.org/lurker/message/20120519.075037.21283114.en.html
  git source:
    git://git.exim.org/exim.git
    The root of the distribution tarball is one-level deep inside git,
    in "src", so we do have src/src :( but if you "cd src" then you can
    create "Local/" inside there and proceed as per a normal release.