Fatal error: out of core in secure memory - during decrypt function call

[Warren, Tony]

Greetings all!

We are trying to develop a sample program in ANSI C running on Linux to =
understand how we can use libgcrypt functions to encrypt data via a web =
interface for secure storage. I have spent days searching through the =
libgcrypt documentation and the web, trying to find a reason/solution =
for this 'out of core' problem, but came up empty.
=20
I was having the same problems as Spencer Ogden=20
(http://lists.gnupg.org/pipermail/gcrypt-devel/2003-May/000276.html)
with the sexp_sscan delivering an error 201. To get around it, I've =
emulated the testsexp.c example and used the gcry_sexp_build function to =
generate an rsa key pair, encapsulate a short string into an sexp, and =
encrypt the sexp. So far all is good.

The problem is when I try to decrypt the encrypted sexp, I get an 'out =
of core in secure memory' fatal error.  This occurs whether running as =
root or not. Everyone in our shop is quite novice at using encryption =
software (why the senior engineers gave me the project, I suppose) but =
it doesn't make sense to me why the gcry_pki_decrypt function should =
fail if I give it a freshly created sKey, an encrypted sexp and a valid =
sexp* -- I'm stumped.

*****************************
We're running on:
*****************************
Red Hat Linux 7.2
Kernel 2.4.9-13smp on an i686
(Not sure if you need any other info)




*****************************
Source code follows...=20
*****************************


/* Sample libgcryp Program - v0.1 */
/* 08 MAY 03 */

#include <stdio.h>
#include "/usr/local/include/gcrypt.h"


int main (int argc, char *argv[])
{

GcrySexp data2;
char encryptMe[] =3D "";
const char *decrypted;
char name[20];
char fkey[3000];
FILE *fp;
char pubkey[]=3D"public-key";
char secKey[]=3D"private-key";
GcrySexp PARMS, result, data;
GCRY_MPI mData;
GcrySexp pKey, sKey, Key;
size_t n;
int i, rcode, nbits =3D 1024;

memset( &data2, 0, sizeof( data2 ) );
memset( &data, 0, sizeof( data ) );
memset( &PARMS, 0, sizeof( PARMS ) );
memset( &result, 0, sizeof( result ) );

printf( "Enter data to encode:" );
scanf( "%s", encryptMe );
printf( "data entered: %s", encryptMe);
n=3Dstrlen(encryptMe);

gcry_mpi_scan( &mData, GCRYMPI_FMT_HEX, encryptMe, NULL);
printf( "\n\n" );

gcry_control( GCRYCTL_INIT_SECMEM, 32768, 0 );

rcode =3D gcry_sexp_build (&data, NULL, "(data(flags raw)(value %d))", =
mData);
printf("\nreturn value for data sexp generation is [%d]", rcode);
printf( "\n" );


rcode =3D gcry_sexp_build (&PARMS, NULL, "(genkey(rsa(nbits %d)))", =
nbits);
printf("return code for sexp_new(PARMS) is [%d]", rcode);
printf("\n\n");

rcode =3D gcry_pk_genkey (&Key, PARMS);
printf("return code for genkey is [%d]", rcode);


printf("\n");
if (!rcode)
  {gcry_sexp_release( PARMS );}

pKey =3D gcry_sexp_find_token(Key, pubkey, strlen(pubkey));
sKey =3D gcry_sexp_find_token(Key, secKey, strlen(secKey));
printf( "\n\n" );
if (sKey&&pKey)
  {gcry_sexp_release( Key );}
printf( "\n\n" );

printf( "\n\n" );
rcode =3D gcry_pk_testkey (sKey);=20
printf("return code for testkey is [%d]", rcode);

rcode =3D gcry_pk_encrypt (&result, data, pKey);
    printf("Return value for encryption with pKey is [%d]", rcode);
    printf( "\n\n" );

rcode =3D gcry_pk_decrypt (&data2, result, sKey);        <-----Fatal =
error occurs here
printf("Return value for decryption: [%d]", rcode);          out of core =
in secure memory
if (!rcode)
  {gcry_sexp_dump (data2);}
printf( "\n\n" );

decrypted =3D gcry_sexp_nth_data(data, 2, &n);
printf( "\n\nDecrypted data:  %s", decrypted );
printf( "\n\n" );
    gcry_sexp_dump (data);

printf( "\n\n" );

/*************************************************
//  Cleaning House - memory-wise...
*/
    if (Key) { gcry_sexp_release( Key ); }
    if (result) { gcry_sexp_release( result ); }
    if (data2) { gcry_sexp_release( data2 ); }
    if (sKey) { gcry_sexp_release( sKey ); }
    if (pKey) { gcry_sexp_release( pKey ); }
    if (data) { gcry_sexp_release( data ); }

} // end main


********************
Results:
********************

Enter data to encode:12345678909876543210
data entered: 12345678909876543210


return value for data sexp generation is [0]
return code for sexp_new(PARMS) is [0]

return code for genkey is [0]

return code for testkey is [0]

Return value for encryption with pKey is [0]


Fatal error: out of core in secure memory
Aborted (core dumped)

********************
/Results
********************


Thanks for any assistance or information you can direct this way. =20


--=20
Tony Warren=20
Prairie Systems, Inc.
Omaha, NE USA
=20
<}-: