2.5.7 gnutls_x509_privkey_generate() returns GNUTLS_E_INVALID_REQUEST

[Sam Varshavchik]

Nikos Mavrogiannopoulos writes:

> Sam Varshavchik wrote:
> 
>> Unfortunately, there appears to be more problems with private key
>> functions in 2.5.7. I'm now getting segfaults in
>> gnutls_x509_privkey_import(). My debugging results are as follows:
>> 
> [...]
>> ==3339== 5,886 (168 direct, 5,718 indirect) bytes in 3 blocks are
>> definitely lost in loss record 8 of 9
>> ==3339==    at 0x4A05174: calloc (vg_replace_malloc.c:397)
>> ==3339==    by 0x3158C0A211: (within /usr/lib64/libtasn1.so.3.0.14)
>> ==3339==    by 0x3158C0A3F2: (within /usr/lib64/libtasn1.so.3.0.14)
>> ==3339==    by 0x3158C0A7BD: asn1_create_element (in
>> /usr/lib64/libtasn1.so.3.0.14)
>> ==3339==    by 0x4C829B2: _gnutls_asn1_encode_rsa (privkey.c:1075)
>> ==3339==    by 0x4C83E0B: gnutls_x509_privkey_export (privkey.c:739)
> 
> Hello Sam,
>  I've commited fixes for these issues, so that latest code in git should
> not have these issues. All except the last one (quoted). It would be
> very helpful to give me some pointers on how to reproduce it.
> 
> As far as I understand it should happening when
> gnutls_rsa_params_export_raw() is called?

The following minimal program triggers the leak:

#include <gcrypt.h>
#include <gnutls/gnutls.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <pthread.h>

GCRY_THREAD_OPTION_PTHREAD_IMPL;

int main()
{
	int i;
	gnutls_rsa_params_t rsa;
	size_t buflen;

	gcry_control(GCRYCTL_SET_THREAD_CBS, &gcry_threads_pthread);
	gcry_control(GCRYCTL_ENABLE_QUICK_RANDOM);
	gnutls_global_init();

	gnutls_rsa_params_init(&rsa);

	printf("vers %s %s\n", LIBGNUTLS_VERSION, gnutls_check_version (NULL));

	gnutls_rsa_params_generate2(rsa, 1024);

	gnutls_rsa_params_export_pkcs1(rsa, GNUTLS_X509_FMT_DER, NULL, &buflen);

	char b[buflen];

	gnutls_rsa_params_export_pkcs1(rsa, GNUTLS_X509_FMT_DER, b, &buflen);

	gnutls_rsa_params_deinit(rsa);

	return(0);
}


The leak gets triggered only when gnutls_rsa_params_export_pkcs1() gets 
invoked twice, once with a NULL buffer pointer, to compute the buffer size, 
and a second time with the real buffer pointer. Invoking it just once is not 
sufficient.

What looks like is happening is that in gnutls_x509_privkey_export():

        case GNUTLS_PK_RSA:
          ret = _gnutls_asn1_encode_rsa (&key->key, key->params);

First time through, key->key is NULL, and _gnutls_asn1_encode_rsa() 
allocates it using asn1_create_element(). On the second call, key->key holds 
the previously allocated buffer, but _gnutls_asn1_encode_rsa() allocates a 
new one, without freeing the first one, and leaks it.

I also see similar logic for the GNUTLS_PK_DSA case, and 
_gnutls_asn1_encode_dsa(), so there may also be a similar leak there.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: </pipermail/attachments/20080921/95868500/attachment.pgp>