SSSE3 problems on Nehalem?

[Jussi Kivilinna]

Hello,

On 03.01.2017 21:57, Werner Koch wrote:
> 
> The selftest should yield these values for rijndail.c:selftest_ctr_128
> around line 487 in _gcry_selftest_helper_ctr (with diff==0):
> 
> iv   : 00000800000000000000000000000008
> plain: 000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f \
>        202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f \
>        404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f \
>        606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f \
>        808182838485868788898a8b8c8d8e8f
> ciphr: eadf062f4bc843fe7662191a78dccd8011bea2ba43937fc63b66ddfaf902eb23 \
>        4585dcf111ea27c00ade03493a89ed6880a4bdc12f3ac0df9493db796266b611 \
>        e51cdbf3bb9be44981c2d4e6b7b34dd326d8676d1dd19949a848ba72343611fa \
>        6f636ddd8db82f0c17ed1bab5bfc1912082c87ff588404305ce8908d32f380c8 \
>        875ee5d348b357227991bf5f5d8f7186
> plain: 000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f \
>        202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f \
>        404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f \
>        606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f \
>        808182838485868788898a8b8c8d8e8f
> 
> All fine.  But on the E5520 I get this back after decryption:
> 
> plain: 000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f \
>        e5a9525c2fcb886698104111a6edaeb407f3b66338c43f35621b5e1bc4c33b9b \
>        ad1c9778f4694da7cbe11352030b156d99a857fc80e124250a358009af6b7ef8 \
>        5f6fc100ac3276af2d9670709718b43c96a62959bb48d623d21d1dedf32fcf0f \
>        da6405a4ba56eeb8e05e623acb304391
> 
> Thus _gcry_aes_ssse3_ctr_enc fails after one block (128 bits).

Bug is in _gcry_aes_ssse3_ctr_enc. 'ctrlow' is passed to assembly block
as read-only register when it should be read/write as assembly block does
64-bit increment on it. Whatever this ends up breaking depends on compiler
register allocation (thus version & flags).

So, on that machine, compiler passes 'ctrlow' to temporary register
before assembly and assembly part increments that register and
calculation is lost.

I'll push fix for this soon. Diff for rinjdael-ssse3 attached below.

-Jussi

---

diff --git a/cipher/rijndael-ssse3-amd64.c b/cipher/rijndael-ssse3-amd64.c
index a8e89d4..2adb73f 100644
--- a/cipher/rijndael-ssse3-amd64.c
+++ b/cipher/rijndael-ssse3-amd64.c
@@ -387,8 +387,8 @@ _gcry_aes_ssse3_ctr_enc (RIJNDAEL_context *ctx, unsigned char *outbuf,
                     ".Lno_carry%=:\n\t"
 
                     "pshufb %%xmm6, %%xmm7\n\t"
-                    :
-                    : [ctr] "r" (ctr), [ctrlow] "r" (ctrlow)
+                    : [ctrlow] "+r" (ctrlow)
+                    : [ctr] "r" (ctr)
                     : "cc", "memory");
 
       do_vpaes_ssse3_enc (ctx, nrounds, aes_const_ptr);


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 273 bytes
Desc: OpenPGP digital signature
URL: </pipermail/attachments/20170104/d6141724/attachment.sig>