gpg encrypt hanging on AIX 5.1

[David Shaw]

On Thu, Nov 06, 2003 at 10:51:48AM -0500, Shelley Meredith McNeill wrote:

> Installed gpg 1.2.1 on AIX 5.1.  Imported public key of third-party
> company successfully.  Am now trying to encrypt a test file with the
> following command:
> 
> gpg --output hartford_test.gpg --encrypt --debug-all --always-trust --recipient ra at thehartford.com hartford_test.txt
> 
> No syntax problems, but the command just hangs.  With debugging on,
> I can see the last thing executed below:
> 
> gpg: DBG: mpi_free_limb_space of size 0
> gpg: DBG: fd_cache_open (hartford_test.txt) not cached
> gpg: DBG: iobuf-5.0: open `hartford_test.txt' fd=6
> File `hartford_test.gpg' exists. Overwrite (y/N)? y
> gpg: DBG: fd_cache_invalidate (hartford_test.gpg)
> gpg: DBG: iobuf-6.0: create `file_filter(fd)'
> 
> Any ideas on why it never finishes?

Try the attached patch to util/secmem.c

David
-------------- next part --------------
Index: util/secmem.c
===================================================================
RCS file: /cvs/gnupg/gnupg/util/secmem.c,v
retrieving revision 1.35.2.3
retrieving revision 1.35.2.4
diff -u -r1.35.2.3 -r1.35.2.4
--- util/secmem.c	22 May 2003 02:42:44 -0000	1.35.2.3
+++ util/secmem.c	23 Oct 2003 07:47:02 -0000	1.35.2.4
@@ -130,9 +130,20 @@
     /* ick. but at least we get secured memory. about to lock
        entire data segment. */
 #ifdef HAVE_PLOCK
+# ifdef _AIX
+    /* The configure for AIX returns broken mlock but the plock has
+       the strange requirement to somehow set the stack limit first.
+       The problem might turn out in indeterministic program behaviour
+       and hanging processes which can somehow be solved when enough
+       processes are clogging up the memory.  To get this problem out
+       of the way we simply don't try to lock the memory at all.
+       */    
+    err = EPERM;
+# else /*!_AIX*/
     err = plock( DATLOCK );
     if( err && errno )
         err = errno;
+# endif /*!_AIX*/
 #else /*!HAVE_PLOCK*/
     if( uid ) {
 	errno = EPERM;