[PATCH] Fix broken mlock detection

Johannes Schindelin Johannes.Schindelin at gmx.de
Wed Jun 16 10:07:11 CEST 2021


We need to be careful when casting a pointer to a `long int`: the
highest bit might be set, in which case the result is a negative number.

In this instance, it is fatal: we now take the modulus of that negative
number with regards to the page size, and subtract it from the page
size. So what should be a number that is smaller than the page size is
now larger than the page size.

As a consequence, we do not try to lock a 4096-byte block that is at the
page size boundary inside a `malloc()`ed block, but we try to do that
_outside_ the block.

Which means that we are not at all detecting whether `mlock()` is
broken.

This actually happened here, in the i686 MSYS2 build of libgcrypt.

Let's be very careful to case the pointer to an _unsigned_ value
instead.

Note: technically, we should cast the pointer to a `size_t`. But since
we only need the remainder modulo the page size (which is a power of
two) anyway, it does not matter whether we clip, say, a 64-bit `size_t`
to a 32-bit `unsigned long`. It does matter, though, whether we
mistakenly turn the remainder into a negative one.

Signed-off-by: Johannes Schindelin <johannes.schindelin at gmx.de>
---
 acinclude.m4 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/acinclude.m4 b/acinclude.m4
index 3c8dfba7..4a2a83c0 100644
--- a/acinclude.m4
+++ b/acinclude.m4
@@ -236,7 +236,7 @@ int main()
     pool = malloc( 4096 + pgsize );
     if( !pool )
         return 2;
-    pool += (pgsize - ((long int)pool % pgsize));
+    pool += (pgsize - ((unsigned long int)pool % pgsize));

     err = mlock( pool, 4096 );
     if( !err || errno == EPERM || errno == EAGAIN)
--
2.31.1





More information about the Gcrypt-devel mailing list