[PATCH] Rinjdael: Fix use of SSE2 outside USE_AESNI/ctx->use_aesni

Jussi Kivilinna jussi.kivilinna at mbnet.fi
Mon Jan 28 10:11:33 CET 2013


* cipher/rijndael.c (_gcry_aes_cbc_enc): Check if AES-NI is enabled before
calling aesni_prepare() and aesni_cleanup().
--

aesni_cleanup() contains SSE2 instructions that are interpreted as MMX on CPUs
without SSE2 support (Pentium-III, etc). This causes x87 register state to be
poisoned, causing crashes later on when program tries to use floating point
registers.

Add '#ifdef USE_AESNI' and 'if (ctx->use_aesni)' for aesni_cleanup() and, while
at it, for aesni_prepare() too.

Reported-by: Mitsutoshi NAKANO <bkbin005 at rinku.zaq.ne.jp>
Signed-off-by: Jussi Kivilinna <jussi.kivilinna at mbnet.fi>

---

Note: this patch is for current development version of libgcrypt, it might not
apply cleanly to earlier stable releases.
---
 cipher/rijndael.c |   12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/cipher/rijndael.c b/cipher/rijndael.c
index 24372d9..a3080f5 100644
--- a/cipher/rijndael.c
+++ b/cipher/rijndael.c
@@ -1399,7 +1399,11 @@ _gcry_aes_cbc_enc (void *context, unsigned char *iv,
   unsigned char *outbuf = outbuf_arg;
   const unsigned char *inbuf = inbuf_arg;
 
-  aesni_prepare ();
+#ifdef USE_AESNI
+  if (ctx->use_aesni)
+    aesni_prepare ();
+#endif /*USE_AESNI*/
+
   for ( ;nblocks; nblocks-- )
     {
       if (0)
@@ -1449,7 +1453,11 @@ _gcry_aes_cbc_enc (void *context, unsigned char *iv,
       if (!cbc_mac)
         outbuf += BLOCKSIZE;
     }
-  aesni_cleanup ();
+
+#ifdef USE_AESNI
+  if (ctx->use_aesni)
+    aesni_cleanup ();
+#endif /*USE_AESNI*/
 
   _gcry_burn_stack (48 + 2*sizeof(int));
 }




More information about the Gcrypt-devel mailing list