[PATCH] Rinjdael: Fix use of SSE2 outside USE_AESNI/ctx->use_aesni

[Jussi Kivilinna]

Quoting 1xx <itsango at gmail.com>:

> Hi all .
> I applied this patch to the libgcrypt11-1.5.0.
> And I resumed tests of gnome-keyring-3.6.1 "make check" .
> The tests failed again .
> I investigated failure and found a leak of the patch .
> I wrote further patch .
> Please confirm it .

Yes, there is two functions in libgcrypt-1.5.0 were aesni_cleanup() is  
being called without use_aesni checks: _gcry_aes_cbc_dec and  
_gcry_aes_cbc_enc.

Current development version of libgcrypt only had this in  
_gcry_aes_cbc_enc and that's why patch I send didn't fix the problem  
for you.

-Jussi

>
> I made all "make check" tests of libgcrypt11-1.5.0
> and gnome-keyring successful now on Pentium III-M .
>
> Thanks .
>
>
> 2013/1/28 Jussi Kivilinna <jussi.kivilinna at mbnet.fi>:
>> * cipher/rijndael.c (_gcry_aes_cbc_enc): Check if AES-NI is enabled before
>> calling aesni_prepare() and aesni_cleanup().
>> --
>>
>> aesni_cleanup() contains SSE2 instructions that are interpreted as  
>> MMX on CPUs
>> without SSE2 support (Pentium-III, etc). This causes x87 register  
>> state to be
>> poisoned, causing crashes later on when program tries to use floating point
>> registers.
>>
>> Add '#ifdef USE_AESNI' and 'if (ctx->use_aesni)' for  
>> aesni_cleanup() and, while
>> at it, for aesni_prepare() too.
>>
>> Reported-by: Mitsutoshi NAKANO <bkbin005 at rinku.zaq.ne.jp>
>> Signed-off-by: Jussi Kivilinna <jussi.kivilinna at mbnet.fi>
>>
>> ---
>>
>> Note: this patch is for current development version of libgcrypt,  
>> it might not
>> apply cleanly to earlier stable releases.
>
> --
> 1xx <ItSANgo at gmail.com>
>
>
> -------- diff -u  rijndael.c.orig2 rijndael.c --------
> --- rijndael.c.orig2    2013-01-29 08:50:34.910273459 +0900
> +++ rijndael.c  2013-01-29 14:46:14.304715698 +0900
> @@ -1609,7 +1609,11 @@
>        inbuf += BLOCKSIZE;
>        outbuf += BLOCKSIZE;
>      }
> -  aesni_cleanup ();
> +
> +#ifdef USE_AESNI
> +  if (ctx->use_aesni)
> +    aesni_cleanup ();
> +#endif /*USE_AESNI*/
>
>    _gcry_burn_stack (48 + 2*sizeof(int) + BLOCKSIZE + 4*sizeof (char*));
>  }
>
>