GnuPG + FreeBSD 5.3 = intermitent memory warning

[David Shaw]

On Sun, Dec 12, 2004 at 08:24:17PM -0500, Atom 'Smasher' wrote:
> ** cross posted **
> 
> for those not familiar with GnuPG, read here for relevant background info: 
> http://www.gnupg.org/documentation/faqs.html#q6.1
> 
> if the binary is suid-root, it should not generate warnings about insecure 
> memory. my binary *is* suid-root, and whether it's run as a privileged or 
> unprivileged user i get intermittent warnings about insecure memory.

It took me a while to track this down, and thanks to Atom for helping
me run some FreeBSD tests.  It turns out that this isn't a GnuPG
specific problem.  The same problem can be duplicated by running any
program that calls mlock() on FreeBSD.

FreeBSD has a "1/3 of memory" hard limit for mlock().  What seems to
have happened is that for whatever reason, Atom's system was very
close to the 1/3 magic number, and so when GnuPG tried to get its
lock, it was sometimes refused.  This also explains why a busy system
seemed to aggravate the problem.

In terms of what to do about this in GnuPG, I'm not sure if there
should be anything done.  I think the the current GnuPG behavior is
pretty good: try to get locked memory, and if it can't, warn the user.

David