[gnutls-devel] Problem with proxied connections on 3.5.3

[Nikos Mavrogiannopoulos]

On Tue, Sep 20, 2016 at 8:51 PM, marcelomendes at gmail.com
<marcelomendes at gmail.com> wrote:
> Thanks, I'm attaching the logs.
> First I tried a git pull with libgnutls30 3.5.4, which gave me the
> handshaking error, then I downgraded to 3.5.2-3, and, as you can see
> at the end of the file, the pull ran well.

Thank you, that was very helpful.
gnutls[4]: HSK[0x2d932b0]: verify handshake data: using RSA-SHA256
gnutls[3]: ASSERT: pk.c[_wrap_nettle_pk_verify]:740

The lines above indicate that the failure is at the check:
"if (signature->size != pub.size)"
We don't have the values here, however, what changed since 3.5.2 is
that nettle is now calculating pub.size in rsa_public_key_prepare()
while previously it was directly calculated as
"nettle_mpz_sizeinbase_256_u(pub->n);"

According to the capture signature->size is 256 bytes and in the
certificate the modulus is 2048-bits (256 bytes).
The signature starts with "44c5...", while the modulus with "f245...".
The latter makes me suspect that rsa_public_key_prepare() could have
calculated size as 257 (assuming it had added a 00 ahead of f2 to make
it non-negative).

Let's see how rsa_public_key_prepare does calculate size. It sets size
as "(mpz_sizeinbase(n, 2) + 7) / 8;" . Interestingly
nettle_mpz_sizeinbase_256_u() also calculates that size as
"(mpz_sizeinbase(x,2) + 7) / 8", so they should have been equal. In
fact they are equal on every system I've tried but yours...
So I am wondering whether I have missed something here. Running the
example program I sent, as well as this under valgrind may provide
more information.

regards,
Nikos