[PATCH v2] mceliece6688128f: fix stack overflow crash on win64/wine

[Jacob Bachmeyer]

On 9/27/25 02:02, Jussi Kivilinna wrote:
> * cipher/mceliece6688128f.c (pk_gen): Remove 'mat' array allocation and
> rename function to ...
> (pk_gen_mat): ... this.
> (pk_gen): New wrapper for 'pk_gen_mat' with dynamic allocation of 'mat'
> array.
> --
>
> Huge array allocations from stack are not always guaranteed to work on
> every target platform, so avoid allocating multi-megabyte 'mat' array
> from stack.
>
> v2: zero 'mat' array after use.
>
> Signed-off-by: Jussi Kivilinna <jussi.kivilinna at iki.fi>
> ---
>   cipher/mceliece6688128f.c | 23 +++++++++++++++++------
>   1 file changed, 17 insertions(+), 6 deletions(-)
>
> diff --git a/cipher/mceliece6688128f.c b/cipher/mceliece6688128f.c
> index ca1952b5..bdc7f265 100644
> --- a/cipher/mceliece6688128f.c
> +++ b/cipher/mceliece6688128f.c
> [...]
> @@ -3460,6 +3459,18 @@ static int pk_gen(unsigned char * pk, const unsigned char * irr, uint32_t * perm
>   }
>   
>   
> +static int pk_gen(unsigned char * pk, const unsigned char * irr, uint32_t * perm, int16_t * pi, uint64_t * pivots)
> +{
> +	/* Allocate large array from heap to avoid stack overflow crash on Win32/Wine. */
> +	unsigned int sizeof_mat = sizeof(uint64_t) * PK_NROWS * nblocks_H;

Would this make sense as "const unsigned int sizeof_mat" to enable 
better compiler optimizations/inlining since PK_NROWS and nblocks_H seem 
to be constants?  Or is GCC able to deduce that sizeof_mat is only 
assigned once?

Also, would "size_t" be a more appropriate base type than unsigned int, 
yielding "const size_t sizeof_mat = [...]"?

> +	void *mat = xmalloc(sizeof_mat);
> +	int ret = pk_gen_mat(pk, irr, perm, pi, pivots, mat);
> +	wipememory(mat, sizeof_mat);
> +	xfree(mat);
> +	return ret;
> +}
> +
> +
>   /* from libmceliece-20230612/crypto_kem/6688128f/vec/sk_gen.c */
>   /*
>     This file is for secret-key generation


-- Jacob