[PATCH] Fix broken mlock detection
Johannes.Schindelin at gmx.de
Wed Jun 16 10:07:11 CEST 2021
We need to be careful when casting a pointer to a `long int`: the
highest bit might be set, in which case the result is a negative number.
In this instance, it is fatal: we now take the modulus of that negative
number with regards to the page size, and subtract it from the page
size. So what should be a number that is smaller than the page size is
now larger than the page size.
As a consequence, we do not try to lock a 4096-byte block that is at the
page size boundary inside a `malloc()`ed block, but we try to do that
_outside_ the block.
Which means that we are not at all detecting whether `mlock()` is
This actually happened here, in the i686 MSYS2 build of libgcrypt.
Let's be very careful to case the pointer to an _unsigned_ value
Note: technically, we should cast the pointer to a `size_t`. But since
we only need the remainder modulo the page size (which is a power of
two) anyway, it does not matter whether we clip, say, a 64-bit `size_t`
to a 32-bit `unsigned long`. It does matter, though, whether we
mistakenly turn the remainder into a negative one.
Signed-off-by: Johannes Schindelin <johannes.schindelin at gmx.de>
acinclude.m4 | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/acinclude.m4 b/acinclude.m4
index 3c8dfba7..4a2a83c0 100644
@@ -236,7 +236,7 @@ int main()
pool = malloc( 4096 + pgsize );
if( !pool )
- pool += (pgsize - ((long int)pool % pgsize));
+ pool += (pgsize - ((unsigned long int)pool % pgsize));
err = mlock( pool, 4096 );
if( !err || errno == EPERM || errno == EAGAIN)
More information about the Gcrypt-devel