bad system call on HP-UX 10.20

[David DeSimone]

Ralf Hildebrandt <R.Hildebrandt at tu-bs.de> wrote:
>
> Ok, but since we know that this fails as root on HP-UX (on your box
> too?), one could code around this...

The obvious fix is to assume that there is no mlock() function on HP-UX
10.20, since it does not appear to be implemented there.  I still
haven't been able to ascertain whether it can or cannot be enabled on
HP-UX 10.20.  It is documented in HP-UX 11.0, so it is useable on that
version of the OS.

The problem is that the library call exists on 10.20, but that it will
fail when an attempt is made to execute it.  So when configure checks to
see if it can link to a function called mlock(), it will succeed, so
configure thinks we "HAVE_MLOCK".  Then when configure tries to test the
function, it fails, so configures thinks we "HAVE_BROKEN_MLOCK".

Apparently, GPG assumes that a "broken mlock" is one that only works for
root.  That is not the case on HP-UX 10.20; it appears that on that OS,
a broken mlock is one that doesn't work... for any user.  It would be
better to undef "HAVE_MLOCK" in such a case, or to only use mlock if we
"HAVE_MLOCK" and not "HAVE_BROKEN_MLOCK".

At any rate, this code could also be rewritten to simply handle (ignore)
the SIGSYS system call that results from use of mlock on this OS.  That
means that, if the program is later run on an HP-UX 11.0 box, it will
correctly make use of secure memory, even if it was compiled on a 10.20
system.  Ignoring the SIGSYS signal for the duration of the mlock call
will cause the call to return with the error ENOSYS, which keeps the
program from dieing, and correctly notifies the program that secure
memory is not available.

An untested patch is attached here.

I think the code in util/secmem.c has some buglets here, because the
value of errno being tested is not always the result of the mlock() call
itself.  But that's another issue.

-- 
David DeSimone   | "The doctrine of human equality reposes on this:
fox at rsn.hp.com   |  that there is no man really clever who has not
Hewlett-Packard  |  found that he is stupid." -- Gilbert K. Chesterson
Richardson IT    |    PGP: 5B 47 34 9F 3B 9A B0 0D  AB A6 15 F1 BB BE 8C 44
-------------- next part --------------
*** util/secmem.c.orig	Sun Nov 14 12:55:27 1999
--- util/secmem.c	Fri Mar 24 17:55:42 2000
***************
*** 24,29 ****
--- 24,30 ----
  #include <string.h>
  #include <errno.h>
  #include <stdarg.h>
+ #include <signal.h>
  #include <unistd.h>
  #if defined(HAVE_MLOCK) || defined(HAVE_MMAP)
    #include <sys/mman.h>
***************
*** 109,114 ****
--- 110,123 ----
      uid_t uid;
      int err;
  
+     struct sigaction nact, oact;
+ 
+     sigemptyset( &nact.sa_mask );
+     nact.sa_handler = SIG_IGN;
+     nact.sa_flags = 0;
+ 
+     sigaction(SIGSYS, &nact, &oact);
+ 
      uid = getuid();
  
    #ifdef HAVE_BROKEN_MLOCK
***************
*** 126,131 ****
--- 135,142 ----
      if( err && errno )
  	err = errno;
    #endif
+ 
+     sigaction(SIGSYS, &oact, NULL);
  
      if( uid && !geteuid() ) {
  	if( setuid( uid ) || getuid() != geteuid()  )