How to work with Libgcrypt in FIPS mode

[Simon Josefsson]

Simon Josefsson <simon at josefsson.org> writes:

> ** libgnutls-extra: Add function to work with Libgcrypt in FIPS mode.
> The function is gnutls_register_md5_handler.  When libgcrypt is in
> FIPS mode, MD5 is disabled, but TLS normally requires use of MD5 in
> the PRF.

Some more explanation related to this may be in order.  If you have
libgcrypt 1.4.3 or later, and create a file /etc/gcrypt/fips_enabled
libgcrypt will run in FIPS mode.  One consequence of this is that MD5 is
disabled...  alas, TLS typically requires MD5, so GnuTLS will not be
very useful with Libgcrypt in FIPS mode.

However, if you link your application to libgnutls-extra and call
gnutls_register_md5_handler, GnuTLS will begin to use an internal MD5
implementation instead of calling libgcrypt.  GnuTLS should then be
fully functional.

The command line tools gnutls-cli, gnutls-serv, and certtool use this
function, and I can successfully use libgcrypt in FIPS mode and connect
to various sites.

Note that this doesn't make GnuTLS FIPS certified, but it is a step
forward.  I believe it is possible to get an exception for MD5 as used
in TLS, and possible all of that code could be moved down into
libgcrypt.  In theory I don't see any reason why GnuTLS can't be FIPS
certified.  Someone needs to sponsor this though.

/Simon