gcry_cipher_encrypt() doesn't release all x87 FPU data registers .

[Jussi Kivilinna]

Quoting 1xx <itsango at gmail.com>:

> Hi all .
>
> I found a problem in aesni_cleanup() on rijndael.c .
> This macro uses MMX instruction, but doesn't call EMMS .

Those are SSE2 instructions, not MMX. So it does not touch x87  
registers at all.

> Therefore, all x87 FPU data registers remains used,
> and after calling gcry_cipher_encrypt(),
> programs cannot use float data .
> So, gnome-keyring, Chromium and Chrome crash on some PCs .
>
> I want you to correct it .
>
> I wrote a patch .
> And I wrote a reappearance program .
>
> If you want them, please see
> https://bugs.launchpad.net/ubuntu/+source/libgcrypt11/+bug/1105758 .

Your bug report shows that you have Pentium-III. It does not support SSE2.

Since PXOR for MMX is encoded with '0F EF' and for SSE2 with '66 0F  
EF', the SSE2 version runs as MMX version on CPUs without SSE2  
support. The 0x66 byte at front of SSE2-opcode is operand-size  
override prefix and is being ignored on your CPU.

Bug appears to be in _gcry_aes_cbc_enc(), aesni_cleanup() is called  
outside '#ifdef USE_AESNI' and/or 'if (ctr->use_aesni)'. This works on  
most (modern) CPUs, but not those without SSE2 support.

-Jussi

>
> Thanks .
>
> --
> 1xx <ItSANgo at gmail.com>
>
> _______________________________________________
> Gcrypt-devel mailing list
> Gcrypt-devel at gnupg.org
> http://lists.gnupg.org/mailman/listinfo/gcrypt-devel
>
>