[gnutls-devel] GnuTLS | certtool --generate-privkey Program received signal SIGILL, Illegal instruction when needle is built with x86-sha-ni (#1496)

Read-only notification of GnuTLS library development activities gnutls-devel at lists.gnutls.org
Mon Aug 7 23:02:44 CEST 2023

Giuseppe Foti created an issue: https://gitlab.com/gnutls/gnutls/-/issues/1496

## Description of problem:
I run the command `certtool --generate-privkey` on a LXD container with a gentoo image.
The program fails with:
`Illegal instruction (core dumped).`

Running certtool on gdb the error received is:

Program received signal SIGILL, Illegal instruction.
0x00007ffff784a8dd in _nettle_sha256_compress_n () from /usr/lib64/libnettle.so.8

libnettle.so.8 was built on the same container that built version 3.9.1 of dev-libs/nettle using this ebuild:
and with those CPU_FLAGS_X86 enabled: aes pclmul sha
(read here for hints about what CPU_FLAGS_ are in gentoo: https://wiki.gentoo.org/wiki/CPU_FLAGS_*)

After rebuilding dev-libs/nettle-3.9.1 disabling the "sha" CPU_FLAGS_X86 certtool works as expected
**The cpu_flags_x86_sha passes the x86-sha-ni arg to the compiler**

## Version of gnutls used:

## Distributor of gnutls (e.g., Ubuntu, Fedora, RHEL)

## How reproducible:

Steps to Reproduce:

 * build https://git.lysator.liu.se/nettle/nettle/-/tags/nettle_3.9.1_release_20230601 with x86-sha-ni ARG on configure
 * build gnutls-3.8.0
 * run `certtool --generate-privkey`

## Actual results:
The program fails with Illegal instruction (core dumped).

## Expected results:
Private Key generated

Reply to this email directly or view it on GitLab: https://gitlab.com/gnutls/gnutls/-/issues/1496
You're receiving this email because of your account on gitlab.com.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.gnupg.org/pipermail/gnutls-devel/attachments/20230807/a00d1447/attachment.html>

More information about the Gnutls-devel mailing list