new testsuite errors with gcrypt 1.5 and gnutls 2.17.1

[Andreas Metzler]

On 2011-07-04 Werner Koch <wk at gnupg.org> wrote:
> I see this in gnutls/lib/pk-libgcrypt.c:_wrap_gcry_pk_decrypt

>       bigint_t res;
>       res = gcry_sexp_nth_mpi (s_plain, 0, 0);
>       gcry_sexp_release (s_plain);

> This is wrong and worked only because of a bug in Libgcrypt < 1.5.0.
[...]
>      If you use this function to parse results of a public key function,
>      you most likely want to use `GCRYMPI_FMT_USG'.]

> If 0 is passed for MPIFMT a default is used, which is and has always
> been GCRYMPI_FMT_STD.  This introduces a leading zero byte so that the
> integer does not start with the MSB set.

> Note that some other code uses gcry_sexp_nth_data and is thus not
> affected by this bug fix.

Hello,

thanks.

For 2.12.7 [1] and 2.10.5 [2] this fixes one test failure
(x509self for 2.12 and x509dn for 2.10) while the other errors remain.
Sorry I am not more helpful than that, I am not a programmer.

> It is the same bug I introduced in GnuPG, thus it is not a surprise that
> you find it also in gnutls.  I did a web search to check the use of this
> function and found that most projects correctly specified the format
> they want.
[...]

If you were wondering what projects were using gcry_sexp_nth_mpi, here
is the list of packages build-depending on libgcrypt11-dev and using
the function:
gnome-keyring-3.0.3 libotr-3.2.0 openvas-libnasl-2.0.2 gnunet-0.8.1b
libspectrum-1.0.0 libgwenhywfar-4.1.0 gnupg2-2.0.17 forked-daapd-0.17
teleport-0.34 libdisplaymigration-0.28 and gnutls.

cu andreas

[1]
---------------
--- gnutls26-2.12.7.orig/lib/gcrypt/pk.c
+++ gnutls26-2.12.7/lib/gcrypt/pk.c
@@ -202,7 +202,7 @@ _wrap_gcry_pk_decrypt (gnutls_pk_algorit
       goto cleanup;
     }

-  res = gcry_sexp_nth_mpi (s_plain, 0, 0);
+  res = gcry_sexp_nth_mpi (s_plain, 0, GCRYMPI_FMT_USG);
   if (res == NULL)
     {
       gnutls_assert ();
---------------
[2]
---------------
--- gnutls26-2.10.5.orig/lib/pk-libgcrypt.c
+++ gnutls26-2.10.5/lib/pk-libgcrypt.c
@@ -202,7 +202,7 @@ _wrap_gcry_pk_decrypt (gnutls_pk_algorit
       goto cleanup;
     }

-  res = gcry_sexp_nth_mpi (s_plain, 0, 0);
+  res = gcry_sexp_nth_mpi (s_plain, 0, GCRYMPI_FMT_USG);
   if (res == NULL)
     {
       gnutls_assert ();
---------------