Possibly incorrect counter overflow handling for AES-GCM

[Clemens.Lang at bmw.de]

Hi,

> On 30. Jan 2018, at 03:43, NIIBE Yutaka <gniibe at fsij.org> wrote:
> 
> Thanks for your report.  I tried to test your example program with 20MB
> (not MiB, but MB) payload.  Given the IV, it works correctly for me.  I
> use libgcrypt version 1.8.1.

Note that you will also have to use the same key K to trigger the behavior. This is because the initial counter value J_0 is calculated from the given IV using the GHASH function, which uses the hash subkey H = CIPH_K(0^128).

Just for the record, I tested this with 1.8.2 and 1.7.6.

> No, it's your example program which must be wrong.
> 
> libgcrypt does the increment correctly by the function gcm_add32_be128
> in cipher/cipher-gcm.c.  Well, I admit that the function name is
> misleading, but it _does_ inc_32, indeed.

gcm_add32_be128 is only used for the first counter increment in _gcry_cipher_gcm_initiv, which is called from _gcry_cipher_gcm_setiv. The counter increase while encrypting is delegated from _gcry_cipher_gcm_encrypt to _gcry_cipher_ctr_encrypt, which is implemented in cipher/cipher-ctr.c. Ignoring the bulk methods for now, the code used to increase the counter in that function is

  for (i = blocksize; i > 0; i--)
    {
      c->u_ctr.ctr[i-1]++;
      if (c->u_ctr.ctr[i-1] != 0)
        break;
    }

where blocksize is the length of the entire counter, so it will not do inc_32 here. The bulk method for AES256-GCM would be _gcry_aes_ctr_enc from cipher/rijndael.c which uses counter increment code equivalent to the block cited above (see also do_aesni_ctr in cipher/rijndael-aesni.c).


> I think that your line:
> 
>    std::size_t ciphertextsize = ((plaintext.size() - 1) | (kBlockSize - 1)) + 1 + kAuthTagSize;
> 
> ... it doesn't work well for plaintext size not multiple of 16.

You’re right, that’s a bug in my reproducer. I’ve checked our original implementation – it does not have this issue. Regardless of that, we see the problem with plaintext sizes that are multiples of 16, where this code should have worked.


Clemens
-- 
Clemens Lang • Development Specialist
BMW Car IT GmbH • Lise-Meitner-Str. 14 • 89081 Ulm • http://bmw-carit.com
-------------------------------------------------------------------------
BMW Car IT GmbH
Geschäftsführer: Kai-Uwe Balszuweit und Christian Salzmann
Sitz und Registergericht: München HRB 134810
-------------------------------------------------------------------------