SSSE3 problems on Nehalem?

[Werner Koch]

Hi!

Due to hardware failures on our old Jenkins server, we switched to an
E5520 box.  Although this box is older than the former Intel pre-release
Clarkdale box it is with its 8 cores more powerful and thus anyway
better for our purposes.

Now, here is the problem: We do not have AES-NI anymore and thus the
SSSE3 optimized AES implementation is used - which fails in the CTR mode
selftest.

I was not able to replicate this failure on other machines even when
forcing the use of SSSE3 for example by using

  tests/basic --disable-hwf intel-fast-shld:intel-pclmul:intel-aesni:intel-avx

(this works for master; you may need to use several --disable-hwf).

Disabling intel-ssse3 on the E5520 is possible (/etc/gcrypt/hwf.deny)
but not a proper fix. 

The selftest should yield these values for rijndail.c:selftest_ctr_128
around line 487 in _gcry_selftest_helper_ctr (with diff==0):

iv   : 00000800000000000000000000000008
plain: 000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f \
       202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f \
       404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f \
       606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f \
       808182838485868788898a8b8c8d8e8f
ciphr: eadf062f4bc843fe7662191a78dccd8011bea2ba43937fc63b66ddfaf902eb23 \
       4585dcf111ea27c00ade03493a89ed6880a4bdc12f3ac0df9493db796266b611 \
       e51cdbf3bb9be44981c2d4e6b7b34dd326d8676d1dd19949a848ba72343611fa \
       6f636ddd8db82f0c17ed1bab5bfc1912082c87ff588404305ce8908d32f380c8 \
       875ee5d348b357227991bf5f5d8f7186
plain: 000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f \
       202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f \
       404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f \
       606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f \
       808182838485868788898a8b8c8d8e8f

All fine.  But on the E5520 I get this back after decryption:

plain: 000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f \
       e5a9525c2fcb886698104111a6edaeb407f3b66338c43f35621b5e1bc4c33b9b \
       ad1c9778f4694da7cbe11352030b156d99a857fc80e124250a358009af6b7ef8 \
       5f6fc100ac3276af2d9670709718b43c96a62959bb48d623d21d1dedf32fcf0f \
       da6405a4ba56eeb8e05e623acb304391

Thus _gcry_aes_ssse3_ctr_enc fails after one block (128 bits).

Has anyone with an E5520 or another Nehalem CPU the same problem?


Shalom-Salam,

   Werner


-- 
Die Gedanken sind frei.  Ausnahmen regelt ein Bundesgesetz.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: </pipermail/attachments/20170103/79ac49c6/attachment.sig>