CMAC + SERPENT/IDEA/RC2 buffer overflow/crash with oversized key

[Guido Vranken]

In the program below, each of three calls to cmac() causes a different
crash (use AddressSanitizer to be sure). I think the correct approach is to
make gcry_mac_setkey() return an error code if the key has an inappropriate
size.

#include <gcrypt.h>

#define CF_CHECK_EQ(expr, res) if ( (expr) != (res) ) { goto end; }

static void cmac(const int mac, const int keysize) {
    unsigned char key[keysize];
    memset(key, 0, keysize);

    gcry_mac_hd_t h;
    CF_CHECK_EQ(gcry_mac_open(&h, mac, 0, NULL), GPG_ERR_NO_ERROR);
    CF_CHECK_EQ(gcry_mac_setkey(h, key, keysize), GPG_ERR_NO_ERROR);

end:
    /* noret */ gcry_mac_close(h);
}

int main(void)
{
    cmac(GCRY_MAC_CMAC_SERPENT, 64);
    cmac(GCRY_MAC_CMAC_IDEA, 32);
    cmac(GCRY_MAC_CMAC_RFC2268, 256);
    return 0;
}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.gnupg.org/pipermail/gcrypt-devel/attachments/20210331/1154e058/attachment.html>