From jussi.kivilinna at iki.fi Thu Oct 1 21:30:16 2026 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Thu, 1 Oct 2026 22:30:16 +0300 Subject: [PATCH 1/2] hwf-riscv: detect Zvkb also from Zvbb Message-ID: <20261001193017.3281823-1-jussi.kivilinna@iki.fi> * src/hwf-riscv.c (HWF_RISCV_HWPROBE_EXT_ZVBB): New. (hwprobe_features): Map Zvbb to HWF_RISCV_ZVKB. -- Signed-off-by: Jussi Kivilinna --- src/hwf-riscv.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/hwf-riscv.c b/src/hwf-riscv.c index 6c642a10..37690997 100644 --- a/src/hwf-riscv.c +++ b/src/hwf-riscv.c @@ -191,6 +191,7 @@ detect_riscv_at_hwcap(void) #define HWF_RISCV_HWPROBE_EXT_ZBB (1U << 4) #define HWF_RISCV_HWPROBE_EXT_ZBS (1U << 5) #define HWF_RISCV_HWPROBE_EXT_ZBC (1U << 7) +#define HWF_RISCV_HWPROBE_EXT_ZVBB (1U << 17) #define HWF_RISCV_HWPROBE_EXT_ZVKB (1U << 19) #define HWF_RISCV_HWPROBE_EXT_ZVKG (1U << 20) #define HWF_RISCV_HWPROBE_EXT_ZVKNED (1U << 21) @@ -221,6 +222,7 @@ static const struct hwprobe_feature_map_s hwprobe_features[] = | HWF_RISCV_HWPROBE_EXT_ZBB | HWF_RISCV_HWPROBE_EXT_ZBS, HWF_RISCV_B }, { HWF_RISCV_HWPROBE_EXT_ZVKB, HWF_RISCV_ZVKB }, + { HWF_RISCV_HWPROBE_EXT_ZVBB, HWF_RISCV_ZVKB }, { HWF_RISCV_HWPROBE_EXT_ZVKG, HWF_RISCV_ZVKG }, { HWF_RISCV_HWPROBE_EXT_ZVKNED, HWF_RISCV_ZVKNED }, { HWF_RISCV_HWPROBE_EXT_ZVKNHA, HWF_RISCV_ZVKNHA }, -- 2.53.0 From jussi.kivilinna at iki.fi Thu Oct 1 21:30:17 2026 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Thu, 1 Oct 2026 22:30:17 +0300 Subject: [PATCH 2/2] rijndael-riscv-zvkned: VLEN agnostic bulk block processing In-Reply-To: <20261001193017.3281823-1-jussi.kivilinna@iki.fi> References: <20261001193017.3281823-1-jussi.kivilinna@iki.fi> Message-ID: <20261001193017.3281823-2-jussi.kivilinna@iki.fi> * cipher/rijndael-riscv-zvkned.c (cast_u8m1_u64m1, cast_u64m1_i64m1) (cast_i64m1_u64m1, memory_barrier_with_vec, bswap128_u32m1) (unaligned_load_u64m1, merge_4x_u32m1_to_u32m4) (split_u32m4_to_4x_u32m1, xts_gfmul_byA): Remove. (broadcast128_u32m1_u32m1, nblocks_to_nblocks_per_m1, ocb_offsets_m1) (XTS_GFMUL_POW_MAX_SHIFT, DEFINE_XTS_GFMUL_POW): New. (ROUND_KEY_VARIABLES, PRELOAD_ROUND_KEYS): Move rk10-rk13 to ... (ROUND_KEY_VARIABLES_RK10_13, PRELOAD_ROUND_KEYS_RK10_13): ... these. (PRELOAD_ROUND_KEYS): Remove vl parameter. (LOAD_KEY, GET_PRELOADED_KEY): New. (AES_CRYPT): Add key loader and round key pointer parameters; split inline assembly variant to separate statements for rounds 0-9, 10-11, 12-13 and last round. (do_prepare_decryption): Read all round keys to registers before writing reversed key schedule. (aes_riscv_ocb_enc, aes_riscv_ocb_dec): Merge to ... (aes_riscv_ocb_crypt): ... this; handle authentication mode too. (_gcry_aes_riscv_zvkned_ocb_auth): Use 'aes_riscv_ocb_crypt'. (aes_riscv_xts_enc, aes_riscv_xts_dec): Merge to ... (aes_riscv_xts_crypt): ... this; build tweaks with doubling ladder. (aes_riscv_zvkned_ecb_crypt, _gcry_aes_riscv_zvkned_cbc_dec) (_gcry_aes_riscv_zvkned_cfb_dec, _gcry_aes_riscv_zvkned_ctr_enc) (_gcry_aes_riscv_zvkned_ctr32le_enc): Process m4/m2/m1 block groups sized by VLEN instead of fixed four blocks. (_gcry_aes_riscv_zvkned_cfb_enc): Use 'PRELOAD_ROUND_KEYS_RK10_13'; load next input block before storing current block. (_gcry_aes_riscv_zvkned_cbc_enc): Use 'PRELOAD_ROUND_KEYS_RK10_13'; load next input block before encrypting current block. * cipher/rijndael.c (do_setkey): Require Zvkb for Zvkned implementation. -- Instead of assuming fixed VLEN=128 and using m4 for four block bulk processing, reimplement Zvkned implementation to use full length RVV vectors. New implementation uses m4/m2/m1 in downwards ladder to bulk process with 4xVLEN/2xVLEN/1xVLEN blocks for parallelizable cipher modes. Tail handling is performed within last 1xVLEN bulk loop. For example, with VLEN=512b (4*128b), first m4 loop handles >=16 AES blocks in parallel, m2 loop >=8 AES blocks and m1 loop 1-7 AES blocks. Benchmark on SpacemiT K3, X100 core (2000 Mhz, out-of-order, VLEN=256): Before | AES128 cycles/byte AES256 cycles/byte ECB enc | 1.56 c/B 2.07 c/B ECB dec | 1.56 c/B 2.05 c/B CBC enc | 2.25 c/B 2.76 c/B CBC dec | 1.96 c/B 2.55 c/B CFB enc | 2.07 c/B 2.57 c/B CFB dec | 1.88 c/B 2.38 c/B CTR enc | 2.12 c/B 2.76 c/B CTR dec | 2.12 c/B 2.76 c/B XTS enc | 3.01 c/B 3.60 c/B XTS dec | 3.02 c/B 3.60 c/B OCB enc | 2.37 c/B 2.92 c/B OCB dec | 2.35 c/B 2.95 c/B OCB auth | 2.14 c/B 2.85 c/B After | AES128 cycles/byte AES256 cycles/byte ECB enc | 0.684 c/B 0.935 c/B ECB dec | 0.687 c/B 0.936 c/B CBC enc | 1.63 c/B 2.13 c/B CBC dec | 0.820 c/B 1.07 c/B CFB enc | 1.63 c/B 2.13 c/B CFB dec | 0.787 c/B 1.04 c/B CTR enc | 0.725 c/B 0.980 c/B CTR dec | 0.727 c/B 0.974 c/B XTS enc | 1.23 c/B 1.48 c/B XTS dec | 1.23 c/B 1.48 c/B OCB enc | 1.10 c/B 1.35 c/B OCB dec | 1.10 c/B 1.35 c/B OCB auth | 1.05 c/B 1.28 c/B ECB: AES128 ~2.3x, AES256 ~2.2x faster CBC enc: AES128 ~1.4x, AES256 ~1.3x faster CFB enc: AES128 ~1.3x, AES256 ~1.2x faster CBC/CFB dec: AES128 ~2.4x, AES256 ~2.3x faster CTR: AES128 ~2.9x, AES256 ~2.8x faster XTS: AES128 ~2.5x, AES256 ~2.4x faster OCB: AES128 ~2.1x, AES256 ~2.2x faster Benchmark on SpacemiT K3, A100 core (1800 Mhz, in-order, VLEN=1024): Before | AES128 cycles/byte AES256 cycles/byte ECB enc | 5.97 c/B 8.05 c/B ECB dec | 5.97 c/B 8.01 c/B CBC enc | 8.64 c/B 10.64 c/B CBC dec | 7.28 c/B 10.18 c/B CFB enc | 5.94 c/B 7.93 c/B CFB dec | 7.18 c/B 9.21 c/B CTR enc | 9.36 c/B 12.90 c/B CTR dec | 9.17 c/B 12.71 c/B XTS enc | 13.72 c/B 16.55 c/B XTS dec | 13.92 c/B 16.67 c/B OCB enc | 12.19 c/B 15.03 c/B OCB dec | 12.23 c/B 14.99 c/B OCB auth | 10.77 c/B 14.49 c/B After | AES128 cycles/byte AES256 cycles/byte ECB enc | 0.764 c/B 1.02 c/B ECB dec | 0.781 c/B 1.04 c/B CBC enc | 5.81 c/B 7.81 c/B CBC dec | 0.889 c/B 1.15 c/B CFB enc | 6.01 c/B 8.01 c/B CFB dec | 0.883 c/B 1.13 c/B CTR enc | 0.832 c/B 1.08 c/B CTR dec | 0.833 c/B 1.08 c/B XTS enc | 1.08 c/B 1.35 c/B XTS dec | 1.10 c/B 1.35 c/B OCB enc | 3.70 c/B 3.91 c/B OCB dec | 3.66 c/B 3.38 c/B OCB auth | 3.63 c/B 3.90 c/B ECB: AES128 ~7.7x, AES256 ~7.8x faster CBC enc: AES128 ~1.5x, AES256 ~1.4x faster CBC/CFB dec: AES128 ~8.2x, AES256 ~8.5x faster CTR: AES128 ~11.1x, AES256 ~11.9x faster XTS: AES128 ~12.7x, AES256 ~12.3x faster OCB: AES128 ~3.2x, AES256 ~4.0x faster Signed-off-by: Jussi Kivilinna --- cipher/rijndael-riscv-zvkned.c | 1762 ++++++++++++++++++-------------- cipher/rijndael.c | 1 + 2 files changed, 980 insertions(+), 783 deletions(-) diff --git a/cipher/rijndael-riscv-zvkned.c b/cipher/rijndael-riscv-zvkned.c index ee9c2de4..45fa4d67 100644 --- a/cipher/rijndael-riscv-zvkned.c +++ b/cipher/rijndael-riscv-zvkned.c @@ -1,5 +1,5 @@ /* rijndael-riscv-zvkned.c - RISC-V vector crypto implementation of AES - * Copyright (C) 2025 Jussi Kivilinna + * Copyright (C) 2025-2026 Jussi Kivilinna * * This file is part of Libgcrypt. * @@ -51,45 +51,37 @@ */ #define cast_u8m1_u32m1(a) __riscv_vreinterpret_v_u8m1_u32m1(a) -#define cast_u8m1_u64m1(a) __riscv_vreinterpret_v_u8m1_u64m1(a) #define cast_u32m1_u8m1(a) __riscv_vreinterpret_v_u32m1_u8m1(a) #define cast_u32m1_u64m1(a) __riscv_vreinterpret_v_u32m1_u64m1(a) #define cast_u64m1_u8m1(a) __riscv_vreinterpret_v_u64m1_u8m1(a) #define cast_u8m2_u32m2(a) __riscv_vreinterpret_v_u8m2_u32m2(a) #define cast_u32m2_u8m2(a) __riscv_vreinterpret_v_u32m2_u8m2(a) +#define cast_u32m2_u64m2(a) __riscv_vreinterpret_v_u32m2_u64m2(a) +#define cast_u64m2_u32m2(a) __riscv_vreinterpret_v_u64m2_u32m2(a) #define cast_u8m4_u32m4(a) __riscv_vreinterpret_v_u8m4_u32m4(a) #define cast_u32m4_u8m4(a) __riscv_vreinterpret_v_u32m4_u8m4(a) +#define cast_u32m4_u64m4(a) __riscv_vreinterpret_v_u32m4_u64m4(a) +#define cast_u64m4_u32m4(a) __riscv_vreinterpret_v_u64m4_u32m4(a) #define cast_u64m1_u32m1(a) __riscv_vreinterpret_v_u64m1_u32m1(a) #define cast_u32m1_u64m1(a) __riscv_vreinterpret_v_u32m1_u64m1(a) -#define cast_u64m1_i64m1(a) __riscv_vreinterpret_v_u64m1_i64m1(a) -#define cast_i64m1_u64m1(a) __riscv_vreinterpret_v_i64m1_u64m1(a) - -#define memory_barrier_with_vec(a) __asm__("" : "+vr"(a) :: "memory") - static ASM_FUNC_ATTR_INLINE vuint32m1_t -bswap128_u32m1(vuint32m1_t vec, size_t vl_u32) -{ - static const byte bswap128_arr[16] = - { 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 }; - size_t vl_bytes = vl_u32 * 4; - vuint8m1_t bswap128 = __riscv_vle8_v_u8m1(bswap128_arr, vl_bytes); - - return cast_u8m1_u32m1( - __riscv_vrgather_vv_u8m1(cast_u32m1_u8m1(vec), bswap128, vl_bytes)); -} - -static ASM_FUNC_ATTR_INLINE vuint64m1_t -unaligned_load_u64m1(const void *ptr, size_t vl_u64) +broadcast128_u32m1_u32m1(vuint32m1_t vec, size_t vl_u32) { -#ifdef RVV_UNALIGNED_NOT_ALLOWED - return cast_u8m1_u64m1(__riscv_vle8_v_u8m1(ptr, vl_u64 * 8)); + vuint32m1_t vdst = __riscv_vmv_v_x_u32m1(0, vl_u32); +#ifdef HAVE_BROKEN_VAES_VS_INTRINSIC + asm ( "vsetvli zero,%[vl],e32,m1,ta,ma;\n\t" + "vaesz.vs %[dst],%[src];\n\t" + : [dst] "+vr" (vdst) + : [vl] "r" (vl_u32), [src] "vr" (vec) + : "vl", "vtype"); + return vdst; #else - return __riscv_vle64_v_u64m1(ptr, vl_u64); + return __riscv_vaesz_vs_u32m1_u32m1(vdst, vec, vl_u32); #endif } @@ -153,44 +145,10 @@ unaligned_store_u32m4(void *ptr, vuint32m4_t vec, size_t vl_u32) #endif } -static ASM_FUNC_ATTR_INLINE vuint32m4_t -merge_4x_u32m1_to_u32m4(vuint32m1_t v0, vuint32m1_t v1, vuint32m1_t v2, - vuint32m1_t v3) -{ - vuint32m2_t v01, v23, tmp2; - vuint32m4_t out, tmp4; - size_t vl = 4; - - v01 = __riscv_vlmul_ext_v_u32m1_u32m2(v0); - tmp2 = __riscv_vlmul_ext_v_u32m1_u32m2(v1); - v01 = __riscv_vslideup_vx_u32m2(v01, tmp2, vl, vl * 2); - v23 = __riscv_vlmul_ext_v_u32m1_u32m2(v2); - tmp2 = __riscv_vlmul_ext_v_u32m1_u32m2(v3); - v23 = __riscv_vslideup_vx_u32m2(v23, tmp2, vl, vl * 2); - out = __riscv_vlmul_ext_v_u32m2_u32m4(v01); - tmp4 = __riscv_vlmul_ext_v_u32m2_u32m4(v23); - return __riscv_vslideup_vx_u32m4(out, tmp4, vl * 2, vl * 4); -} - -static ASM_FUNC_ATTR_INLINE vuint32m1x4_t -split_u32m4_to_4x_u32m1(vuint32m4_t v0123) +static ASM_FUNC_ATTR_INLINE size_t +nblocks_to_nblocks_per_m1(size_t max_blocks_m1, size_t nblocks) { - size_t vl = 4; - vuint32m2_t v01 = __riscv_vlmul_trunc_v_u32m4_u32m2(v0123); - vuint32m2_t v23 = __riscv_vlmul_trunc_v_u32m4_u32m2( - __riscv_vslidedown_vx_u32m4(v0123, vl * 2, vl * 4)); - vuint32m1_t v0 = __riscv_vlmul_trunc_v_u32m2_u32m1(v01); - vuint32m1_t v1 = __riscv_vlmul_trunc_v_u32m2_u32m1( - __riscv_vslidedown_vx_u32m2(v01, vl, vl * 2)); - vuint32m1_t v2 = __riscv_vlmul_trunc_v_u32m2_u32m1(v23); - vuint32m1_t v3 = __riscv_vlmul_trunc_v_u32m2_u32m1( - __riscv_vslidedown_vx_u32m2(v23, vl, vl * 2)); - vuint32m1x4_t out = __riscv_vundefined_u32m1x4(); - out = __riscv_vset_v_u32m1_u32m1x4(out, 0, v0); - out = __riscv_vset_v_u32m1_u32m1x4(out, 1, v1); - out = __riscv_vset_v_u32m1_u32m1x4(out, 2, v2); - out = __riscv_vset_v_u32m1_u32m1x4(out, 3, v3); - return out; + return nblocks < max_blocks_m1 ? nblocks : max_blocks_m1; } @@ -396,39 +354,64 @@ do_prepare_decryption(RIJNDAEL_context *ctx) u32 *dkey = (u32 *)(void *)ctx->keyschdec; int rounds = ctx->rounds; size_t vl = 4; - int rr; - int r; - -#define COPY_KEY() \ - __riscv_vse32_v_u32m1(dkey + r * 4, \ - __riscv_vle32_v_u32m1(ekey + rr * 4, vl), \ - vl) - - r = 0; - rr = rounds; - COPY_KEY(); r++; rr--; - COPY_KEY(); r++; rr--; - COPY_KEY(); r++; rr--; - COPY_KEY(); r++; rr--; - COPY_KEY(); r++; rr--; - COPY_KEY(); r++; rr--; - COPY_KEY(); r++; rr--; - COPY_KEY(); r++; rr--; - COPY_KEY(); r++; rr--; - COPY_KEY(); r++; rr--; - if (rr > 0) + vuint32m1_t k0, k1, k2, k3, k4, k5, k6, k7; + vuint32m1_t k8, k9, k10, k11, k12, k13, k14; + +#define READ_KEY(n) (k##n = __riscv_vle32_v_u32m1(ekey + (rounds - n) * 4, vl)) +#define WRITE_KEY(n) __riscv_vse32_v_u32m1(dkey + n * 4, k##n, vl) + + k11 = __riscv_vundefined_u32m1(); + k12 = __riscv_vundefined_u32m1(); + k13 = __riscv_vundefined_u32m1(); + k14 = __riscv_vundefined_u32m1(); + READ_KEY(0); READ_KEY(1); + READ_KEY(2); READ_KEY(3); + READ_KEY(4); READ_KEY(5); + READ_KEY(6); READ_KEY(7); + READ_KEY(8); READ_KEY(9); + if (LIKELY(rounds >= 12)) { - COPY_KEY(); r++; rr--; - COPY_KEY(); r++; rr--; - if (rr > 0) + READ_KEY(10); READ_KEY(11); + if (LIKELY(rounds > 12)) { - COPY_KEY(); r++; rr--; - COPY_KEY(); r++; rr--; + READ_KEY(12); READ_KEY(13); + READ_KEY(14); + } + else + { + READ_KEY(12); } } - COPY_KEY(); + else + { + READ_KEY(10); + } -#undef COPY_KEY + WRITE_KEY(0); WRITE_KEY(1); + WRITE_KEY(2); WRITE_KEY(3); + WRITE_KEY(4); WRITE_KEY(5); + WRITE_KEY(6); WRITE_KEY(7); + WRITE_KEY(8); WRITE_KEY(9); + if (LIKELY(rounds >= 12)) + { + WRITE_KEY(10); WRITE_KEY(11); + if (LIKELY(rounds > 12)) + { + WRITE_KEY(12); WRITE_KEY(13); + WRITE_KEY(14); + } + else + { + WRITE_KEY(12); + } + } + else + { + WRITE_KEY(10); + } + +#undef READ_KEY +#undef WRITE_KEY } void ASM_FUNC_ATTR_NOINLINE FUNC_ATTR_OPT_O2 @@ -444,30 +427,42 @@ _gcry_aes_riscv_zvkned_prepare_decryption(RIJNDAEL_context *ctx) */ #define ROUND_KEY_VARIABLES \ - vuint32m1_t rk0, rk1, rk2, rk3, rk4, rk5, rk6, rk7, rk8; \ - vuint32m1_t rk9, rk10, rk11, rk12, rk13, rk_last; - -#define PRELOAD_ROUND_KEYS(rk, nrounds, vl) \ - do { \ - rk0 = __riscv_vle32_v_u32m1(rk + 0 * 4, vl); \ - rk1 = __riscv_vle32_v_u32m1(rk + 1 * 4, vl); \ - rk2 = __riscv_vle32_v_u32m1(rk + 2 * 4, vl); \ - rk3 = __riscv_vle32_v_u32m1(rk + 3 * 4, vl); \ - rk4 = __riscv_vle32_v_u32m1(rk + 4 * 4, vl); \ - rk5 = __riscv_vle32_v_u32m1(rk + 5 * 4, vl); \ - rk6 = __riscv_vle32_v_u32m1(rk + 6 * 4, vl); \ - rk7 = __riscv_vle32_v_u32m1(rk + 7 * 4, vl); \ - rk8 = __riscv_vle32_v_u32m1(rk + 8 * 4, vl); \ - rk9 = __riscv_vle32_v_u32m1(rk + 9 * 4, vl); \ - if (UNLIKELY(nrounds >= 12)) \ + vuint32m1_t rk0, rk1, rk2, rk3, rk4, rk5, rk6, rk7, rk8, rk9, rk_last +#define ROUND_KEY_VARIABLES_RK10_13 \ + vuint32m1_t rk10, rk11, rk12, rk13 + +#define LOAD_KEY(tempvec, roundnum) \ + tempvec = __riscv_vle32_v_u32m1(rk + roundnum * 4, BLOCKSIZE / 4) + +#define GET_PRELOADED_KEY(tempvec, roundnum) \ + tempvec = rk ## roundnum + +#define PRELOAD_ROUND_KEYS(rk, nrounds) \ + ({ \ + LOAD_KEY(rk0, 0); \ + LOAD_KEY(rk1, 1); \ + LOAD_KEY(rk2, 2); \ + LOAD_KEY(rk3, 3); \ + LOAD_KEY(rk4, 4); \ + LOAD_KEY(rk5, 5); \ + LOAD_KEY(rk6, 6); \ + LOAD_KEY(rk7, 7); \ + LOAD_KEY(rk8, 8); \ + LOAD_KEY(rk9, 9); \ + LOAD_KEY(rk_last, nrounds); \ + }) + +#define PRELOAD_ROUND_KEYS_RK10_13(rk, nrounds) \ + ({ \ + if (LIKELY((nrounds) >= 12)) \ { \ - rk10 = __riscv_vle32_v_u32m1(rk + 10 * 4, vl); \ - rk11 = __riscv_vle32_v_u32m1(rk + 11 * 4, vl); \ - if (LIKELY(nrounds > 12)) \ - { \ - rk12 = __riscv_vle32_v_u32m1(rk + 12 * 4, vl); \ - rk13 = __riscv_vle32_v_u32m1(rk + 13 * 4, vl); \ - } \ + LOAD_KEY(rk10, 10); \ + LOAD_KEY(rk11, 11); \ + if (LIKELY((nrounds) > 12)) \ + { \ + LOAD_KEY(rk12, 12); \ + LOAD_KEY(rk13, 13); \ + } \ else \ { \ rk12 = __riscv_vundefined_u32m1(); \ @@ -481,41 +476,65 @@ _gcry_aes_riscv_zvkned_prepare_decryption(RIJNDAEL_context *ctx) rk12 = __riscv_vundefined_u32m1(); \ rk13 = __riscv_vundefined_u32m1(); \ } \ - rk_last = __riscv_vle32_v_u32m1(rk + nrounds * 4, vl); \ - } while (0) + }) #ifdef HAVE_BROKEN_VAES_VS_INTRINSIC -#define AES_CRYPT(e_d, mx, nrounds, blk, vlen) \ - asm ( "vsetvli zero,%[vl],e32,"#mx",ta,ma;\n\t" \ - "vaesz.vs %[block],%[rk0];\n\t" \ - "vaes"#e_d"m.vs %[block],%[rk1];\n\t" \ - "vaes"#e_d"m.vs %[block],%[rk2];\n\t" \ - "vaes"#e_d"m.vs %[block],%[rk3];\n\t" \ - "vaes"#e_d"m.vs %[block],%[rk4];\n\t" \ - "vaes"#e_d"m.vs %[block],%[rk5];\n\t" \ - "vaes"#e_d"m.vs %[block],%[rk6];\n\t" \ - "vaes"#e_d"m.vs %[block],%[rk7];\n\t" \ - "vaes"#e_d"m.vs %[block],%[rk8];\n\t" \ - "vaes"#e_d"m.vs %[block],%[rk9];\n\t" \ - "blt %[rounds],%[num12],.Lcryptlast%=;\n\t" \ - "vaes"#e_d"m.vs %[block],%[rk10];\n\t" \ - "vaes"#e_d"m.vs %[block],%[rk11];\n\t" \ - "beq %[rounds],%[num12],.Lcryptlast%=;\n\t" \ - "vaes"#e_d"m.vs %[block],%[rk12];\n\t" \ - "vaes"#e_d"m.vs %[block],%[rk13];\n\t" \ - ".Lcryptlast%=:\n\t" \ - "vaes"#e_d"f.vs %[block],%[rk_last];\n\t" \ - : [block] "+vr" (blk) \ - : [vl] "r" (vlen), [rounds] "r" (nrounds), [num12] "r" (12), \ - [rk0] "vr" (rk0), [rk1] "vr" (rk1), [rk2] "vr" (rk2), \ - [rk3] "vr" (rk3), [rk4] "vr" (rk4), [rk5] "vr" (rk5), \ - [rk6] "vr" (rk6), [rk7] "vr" (rk7), [rk8] "vr" (rk8), \ - [rk9] "vr" (rk9), [rk10] "vr" (rk10), [rk11] "vr" (rk11), \ - [rk12] "vr" (rk12), [rk13] "vr" (rk13), \ - [rk_last] "vr" (rk_last) \ - : "vl") +#define AES_CRYPT(e_d, mx, load_key, rk, nrounds, blk, vlen) \ + ({ \ + asm ( "vsetvli zero,%[vl],e32,"#mx",ta,ma;\n\t" \ + "vaesz.vs %[block],%[rk0];\n\t" \ + "vaes"#e_d"m.vs %[block],%[rk1];\n\t" \ + "vaes"#e_d"m.vs %[block],%[rk2];\n\t" \ + "vaes"#e_d"m.vs %[block],%[rk3];\n\t" \ + "vaes"#e_d"m.vs %[block],%[rk4];\n\t" \ + "vaes"#e_d"m.vs %[block],%[rk5];\n\t" \ + "vaes"#e_d"m.vs %[block],%[rk6];\n\t" \ + "vaes"#e_d"m.vs %[block],%[rk7];\n\t" \ + "vaes"#e_d"m.vs %[block],%[rk8];\n\t" \ + "vaes"#e_d"m.vs %[block],%[rk9];\n\t" \ + : [block] "+vr" (blk) \ + : [vl] "r" (vlen), \ + [rk0] "vr" (rk0), [rk1] "vr" (rk1), [rk2] "vr" (rk2), \ + [rk3] "vr" (rk3), [rk4] "vr" (rk4), [rk5] "vr" (rk5), \ + [rk6] "vr" (rk6), [rk7] "vr" (rk7), [rk8] "vr" (rk8), \ + [rk9] "vr" (rk9) \ + : "vl", "vtype"); \ + if (LIKELY((nrounds) >= 12)) \ + { \ + vuint32m1_t tmp_rk10; \ + vuint32m1_t tmp_rk11; \ + load_key(tmp_rk10, 10); \ + load_key(tmp_rk11, 11); \ + asm ( "vsetvli zero,%[vl],e32,"#mx",ta,ma;\n\t" \ + "vaes"#e_d"m.vs %[block],%[rk10];\n\t" \ + "vaes"#e_d"m.vs %[block],%[rk11];\n\t" \ + : [block] "+vr" (blk) \ + : [vl] "r" (vlen), \ + [rk10] "vr" (tmp_rk10), [rk11] "vr" (tmp_rk11) \ + : "vl", "vtype"); \ + if (LIKELY((nrounds) > 12)) \ + { \ + vuint32m1_t tmp_rk12; \ + vuint32m1_t tmp_rk13; \ + load_key(tmp_rk12, 12); \ + load_key(tmp_rk13, 13); \ + asm ( "vsetvli zero,%[vl],e32,"#mx",ta,ma;\n\t" \ + "vaes"#e_d"m.vs %[block],%[rk12];\n\t" \ + "vaes"#e_d"m.vs %[block],%[rk13];\n\t" \ + : [block] "+vr" (blk) \ + : [vl] "r" (vlen), \ + [rk12] "vr" (tmp_rk12), [rk13] "vr" (tmp_rk13) \ + : "vl", "vtype"); \ + } \ + } \ + asm ( "vsetvli zero,%[vl],e32,"#mx",ta,ma;\n\t" \ + "vaes"#e_d"f.vs %[block],%[rk_last];\n\t" \ + : [block] "+vr" (blk) \ + : [vl] "r" (vlen), [rk_last] "vr" (rk_last) \ + : "vl", "vtype"); \ + }) #else -#define AES_CRYPT(e_d, mx, rounds, block, vl) \ +#define AES_CRYPT(e_d, mx, load_key, rk, nrounds, block, vl) \ ({ \ (block) = __riscv_vaesz_vs_u32m1_u32##mx((block), rk0, (vl)); \ (block) = __riscv_vaes##e_d##m_vs_u32m1_u32##mx((block), rk1, (vl)); \ @@ -527,14 +546,22 @@ _gcry_aes_riscv_zvkned_prepare_decryption(RIJNDAEL_context *ctx) (block) = __riscv_vaes##e_d##m_vs_u32m1_u32##mx((block), rk7, (vl)); \ (block) = __riscv_vaes##e_d##m_vs_u32m1_u32##mx((block), rk8, (vl)); \ (block) = __riscv_vaes##e_d##m_vs_u32m1_u32##mx((block), rk9, (vl)); \ - if (UNLIKELY((rounds) >= 12)) \ + if (LIKELY((nrounds) >= 12)) \ { \ - (block) = __riscv_vaes##e_d##m_vs_u32m1_u32##mx((block), rk10, (vl)); \ - (block) = __riscv_vaes##e_d##m_vs_u32m1_u32##mx((block), rk11, (vl)); \ - if (LIKELY((rounds) > 12)) \ + vuint32m1_t tmp_rk10; \ + vuint32m1_t tmp_rk11; \ + load_key(tmp_rk10, 10); \ + load_key(tmp_rk11, 11); \ + (block) = __riscv_vaes##e_d##m_vs_u32m1_u32##mx((block), tmp_rk10, (vl)); \ + (block) = __riscv_vaes##e_d##m_vs_u32m1_u32##mx((block), tmp_rk11, (vl)); \ + if (LIKELY((nrounds) > 12)) \ { \ - (block) = __riscv_vaes##e_d##m_vs_u32m1_u32##mx((block), rk12, (vl)); \ - (block) = __riscv_vaes##e_d##m_vs_u32m1_u32##mx((block), rk13, (vl)); \ + vuint32m1_t tmp_rk12; \ + vuint32m1_t tmp_rk13; \ + load_key(tmp_rk12, 12); \ + load_key(tmp_rk13, 13); \ + (block) = __riscv_vaes##e_d##m_vs_u32m1_u32##mx((block), tmp_rk12, (vl)); \ + (block) = __riscv_vaes##e_d##m_vs_u32m1_u32##mx((block), tmp_rk13, (vl)); \ } \ } \ (block) = __riscv_vaes##e_d##f_vs_u32m1_u32##mx((block), rk_last, (vl)); \ @@ -551,11 +578,11 @@ _gcry_aes_riscv_zvkned_encrypt (const RIJNDAEL_context *ctx, unsigned char *out, vuint32m1_t block; ROUND_KEY_VARIABLES; - PRELOAD_ROUND_KEYS (rk, rounds, vl); + PRELOAD_ROUND_KEYS (rk, rounds); block = unaligned_load_u32m1(in, vl); - AES_CRYPT(e, m1, rounds, block, vl); + AES_CRYPT(e, m1, LOAD_KEY, rk, rounds, block, vl); unaligned_store_u32m1(out, block, vl); @@ -574,11 +601,11 @@ _gcry_aes_riscv_zvkned_decrypt (const RIJNDAEL_context *ctx, unsigned char *out, vuint32m1_t block; ROUND_KEY_VARIABLES; - PRELOAD_ROUND_KEYS (rk, rounds, vl); + PRELOAD_ROUND_KEYS (rk, rounds); block = unaligned_load_u32m1(in, vl); - AES_CRYPT(d, m1, rounds, block, vl); + AES_CRYPT(d, m1, LOAD_KEY, rk, rounds, block, vl); unaligned_store_u32m1(out, block, vl); @@ -596,49 +623,79 @@ aes_riscv_zvkned_ecb_crypt (void *context, void *outbuf_arg, const unsigned char *inbuf = inbuf_arg; const u32 *rk = encrypt ? ctx->keyschenc32[0] : ctx->keyschdec32[0]; int rounds = ctx->rounds; - size_t vl = 4; + size_t max_blocks_m1 = __riscv_vsetvlmax_e32m1() / 4; + size_t max_blocks_m2 = max_blocks_m1 * 2; + size_t max_blocks_m4 = max_blocks_m1 * 4; ROUND_KEY_VARIABLES; + ROUND_KEY_VARIABLES_RK10_13; - if (!encrypt && !ctx->decryption_prepared) + if (!encrypt && UNLIKELY(!ctx->decryption_prepared)) { do_prepare_decryption(ctx); ctx->decryption_prepared = 1; } - PRELOAD_ROUND_KEYS (rk, rounds, vl); + PRELOAD_ROUND_KEYS (rk, rounds); - for (; nblocks >= 4; nblocks -= 4) + while (nblocks >= max_blocks_m4) { + size_t vl_m4 = max_blocks_m4 * 4; vuint32m4_t blocks; - blocks = unaligned_load_u32m4(inbuf, vl * 4); + blocks = unaligned_load_u32m4(inbuf, vl_m4); if (encrypt) - AES_CRYPT(e, m4, rounds, blocks, vl * 4); + AES_CRYPT(e, m4, LOAD_KEY, rk, rounds, blocks, vl_m4); else - AES_CRYPT(d, m4, rounds, blocks, vl * 4); + AES_CRYPT(d, m4, LOAD_KEY, rk, rounds, blocks, vl_m4); - unaligned_store_u32m4(outbuf, blocks, vl * 4); + unaligned_store_u32m4(outbuf, blocks, vl_m4); - inbuf += 4 * BLOCKSIZE; - outbuf += 4 * BLOCKSIZE; + nblocks -= max_blocks_m4; + inbuf += BLOCKSIZE * max_blocks_m4; + outbuf += BLOCKSIZE * max_blocks_m4; } - for (; nblocks; nblocks--) + if (nblocks) + PRELOAD_ROUND_KEYS_RK10_13 (rk, rounds); + + if (nblocks && nblocks >= max_blocks_m2) { - vuint32m1_t block; + size_t vl_m2 = max_blocks_m2 * 4; + vuint32m2_t blocks; - block = unaligned_load_u32m1(inbuf, vl); + blocks = unaligned_load_u32m2(inbuf, vl_m2); if (encrypt) - AES_CRYPT(e, m1, rounds, block, vl); + AES_CRYPT(e, m2, GET_PRELOADED_KEY, rk, rounds, blocks, vl_m2); else - AES_CRYPT(d, m1, rounds, block, vl); + AES_CRYPT(d, m2, GET_PRELOADED_KEY, rk, rounds, blocks, vl_m2); - unaligned_store_u32m1(outbuf, block, vl); + unaligned_store_u32m2(outbuf, blocks, vl_m2); - inbuf += BLOCKSIZE; - outbuf += BLOCKSIZE; + nblocks -= max_blocks_m2; + inbuf += BLOCKSIZE * max_blocks_m2; + outbuf += BLOCKSIZE * max_blocks_m2; + } + + while (nblocks) + { + size_t curr_nblks = nblocks_to_nblocks_per_m1(max_blocks_m1, nblocks); + size_t vl_m1 = curr_nblks * 4; + vuint32m1_t blocks; + + blocks = unaligned_load_u32m1(inbuf, vl_m1); + + if (encrypt) + AES_CRYPT(e, m1, GET_PRELOADED_KEY, rk, rounds, blocks, vl_m1); + else + AES_CRYPT(d, m1, GET_PRELOADED_KEY, rk, rounds, blocks, vl_m1); + + unaligned_store_u32m1(outbuf, blocks, vl_m1); + + nblocks -= curr_nblks; + inbuf += BLOCKSIZE * curr_nblks; + outbuf += BLOCKSIZE * curr_nblks; } clear_vec_regs(); @@ -682,23 +739,35 @@ _gcry_aes_riscv_zvkned_cfb_enc (void *context, unsigned char *iv_arg, size_t vl = 4; vuint32m1_t iv; ROUND_KEY_VARIABLES; + ROUND_KEY_VARIABLES_RK10_13; - PRELOAD_ROUND_KEYS (rk, rounds, vl); + PRELOAD_ROUND_KEYS (rk, rounds); + PRELOAD_ROUND_KEYS_RK10_13 (rk, rounds); iv = __riscv_vle32_v_u32m1((void *)iv_arg, vl); - for (; nblocks; nblocks--) + if (nblocks) { vuint32m1_t data = unaligned_load_u32m1(inbuf, vl); - AES_CRYPT(e, m1, rounds, iv, vl); + while (--nblocks) + { + AES_CRYPT(e, m1, GET_PRELOADED_KEY, rk, rounds, iv, vl); + + iv = __riscv_vxor_vv_u32m1(iv, data, vl); + + inbuf += BLOCKSIZE; + data = unaligned_load_u32m1(inbuf, vl); + + unaligned_store_u32m1(outbuf, iv, vl); + outbuf += BLOCKSIZE; + } + + AES_CRYPT(e, m1, GET_PRELOADED_KEY, rk, rounds, iv, vl); - data = __riscv_vxor_vv_u32m1(iv, data, vl); - unaligned_store_u32m1(outbuf, data, vl); - iv = data; + iv = __riscv_vxor_vv_u32m1(iv, data, vl); - outbuf += BLOCKSIZE; - inbuf += BLOCKSIZE; + unaligned_store_u32m1(outbuf, iv, vl); } __riscv_vse32_v_u32m1((void *)iv_arg, iv, vl); @@ -720,22 +789,35 @@ _gcry_aes_riscv_zvkned_cbc_enc (void *context, unsigned char *iv_arg, size_t vl = 4; vuint32m1_t iv; ROUND_KEY_VARIABLES; + ROUND_KEY_VARIABLES_RK10_13; - PRELOAD_ROUND_KEYS (rk, rounds, vl); + PRELOAD_ROUND_KEYS (rk, rounds); + PRELOAD_ROUND_KEYS_RK10_13 (rk, rounds); iv = __riscv_vle32_v_u32m1((void *)iv_arg, vl); - for (; nblocks; nblocks--) + if (nblocks) { vuint32m1_t data = unaligned_load_u32m1(inbuf, vl); + + while (--nblocks) + { + iv = __riscv_vxor_vv_u32m1(data, iv, vl); + + inbuf += BLOCKSIZE; + data = unaligned_load_u32m1(inbuf, vl); + + AES_CRYPT(e, m1, GET_PRELOADED_KEY, rk, rounds, iv, vl); + + unaligned_store_u32m1(outbuf, iv, vl); + outbuf += outbuf_add; + } + iv = __riscv_vxor_vv_u32m1(data, iv, vl); - AES_CRYPT(e, m1, rounds, iv, vl); + AES_CRYPT(e, m1, GET_PRELOADED_KEY, rk, rounds, iv, vl); unaligned_store_u32m1(outbuf, iv, vl); - - inbuf += BLOCKSIZE; - outbuf += outbuf_add; } __riscv_vse32_v_u32m1((void *)iv_arg, iv, vl); @@ -748,137 +830,149 @@ _gcry_aes_riscv_zvkned_ctr_enc (void *context, unsigned char *ctr_arg, void *outbuf_arg, const void *inbuf_arg, size_t nblocks) { - static const byte add_u8_array[4][16] __attribute__ ((aligned (16))) = - { - { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1 }, - { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2 }, - { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3 }, - { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4 } - }; - static const u32 le_add[4] = { 1, 0, 0, 0 }; RIJNDAEL_context *ctx = context; unsigned char *outbuf = outbuf_arg; const unsigned char *inbuf = inbuf_arg; const u32 *rk = ctx->keyschenc32[0]; int rounds = ctx->rounds; - size_t vl = 4; - u64 ctrlow; + size_t blk_vl32 = BLOCKSIZE / 4; + size_t max_blocks_m1 = __riscv_vsetvlmax_e32m1() / 4; + size_t max_blocks_m2 = max_blocks_m1 * 2; + size_t max_blocks_m4 = max_blocks_m1 * 4; + size_t vl32m1 = max_blocks_m1 * 4; + size_t vl32m2 = max_blocks_m2 * 4; + size_t vl32m4 = max_blocks_m4 * 4; vuint32m1_t ctr; - vuint32m1_t add1; + vbool32_t lane3_mask_m1; + vuint32m1_t ctrle_m1; + u32 ctrlow; ROUND_KEY_VARIABLES; + ROUND_KEY_VARIABLES_RK10_13; - PRELOAD_ROUND_KEYS (rk, rounds, vl); + PRELOAD_ROUND_KEYS (rk, rounds); - add1 = __riscv_vle32_v_u32m1((const void *)add_u8_array[0], vl); - ctr = __riscv_vle32_v_u32m1((void *)ctr_arg, vl); - ctrlow = __riscv_vmv_x_s_u64m1_u64(cast_u32m1_u64m1(bswap128_u32m1(ctr, vl))); + ctrlow = ctr_arg[15] + (ctr_arg[14] << 8); + ctrlow += nblocks; - memory_barrier_with_vec(add1); + /* Prepare full m1 wide counter */ + lane3_mask_m1 = __riscv_vmseq_vx_u32m1_b32( + __riscv_vand_vx_u32m1(__riscv_vid_v_u32m1(vl32m1), 3, vl32m1), 3, + vl32m1); + ctr = __riscv_vle32_v_u32m1((void *)ctr_arg, blk_vl32); + ctrle_m1 = __riscv_vrev8_v_u32m1(ctr, blk_vl32); + if (max_blocks_m1 > 1) + { + ctrle_m1 = broadcast128_u32m1_u32m1(ctrle_m1, vl32m1); + ctrle_m1 = __riscv_vadd_vv_u32m1_mu(lane3_mask_m1, ctrle_m1, ctrle_m1, + __riscv_vsrl_vx_u32m1(__riscv_vid_v_u32m1(vl32m1), 2, vl32m1), + vl32m1); + } - if (nblocks >= 4) + if (nblocks >= max_blocks_m2) { - vuint32m1_t add2 = __riscv_vle32_v_u32m1((const void *)add_u8_array[1], vl); - vuint32m1_t add3 = __riscv_vle32_v_u32m1((const void *)add_u8_array[2], vl); - vuint32m1_t add4 = __riscv_vle32_v_u32m1((const void *)add_u8_array[3], vl); + vbool16_t lane3_mask_m2 = __riscv_vmseq_vx_u32m2_b16( + __riscv_vand_vx_u32m2(__riscv_vid_v_u32m2(vl32m2), 3, vl32m2), 3, + vl32m2); + vuint32m2_t ctrle_m2 = __riscv_vset_v_u32m1_u32m2( + __riscv_vundefined_u32m2(), 0, ctrle_m1); + + /* Double counter width from m1 to m2 */ + ctrle_m2 = __riscv_vset_v_u32m1_u32m2(ctrle_m2, 1, + __riscv_vadd_vx_u32m1_mu(lane3_mask_m1, ctrle_m1, ctrle_m1, + max_blocks_m1, vl32m1)); + + if (nblocks >= max_blocks_m4) + { + vbool8_t lane3_mask_m4 = __riscv_vmseq_vx_u32m4_b8( + __riscv_vand_vx_u32m4(__riscv_vid_v_u32m4(vl32m4), 3, vl32m4), + 3, vl32m4); + vuint32m4_t ctrle_m4 = __riscv_vset_v_u32m2_u32m4( + __riscv_vundefined_u32m4(), 0, ctrle_m2); + + /* Double counter width from m2 to m4 */ + ctrle_m4 = __riscv_vset_v_u32m2_u32m4(ctrle_m4, 1, + __riscv_vadd_vx_u32m2_mu(lane3_mask_m2, ctrle_m2, ctrle_m2, + max_blocks_m2, vl32m2)); + + do + { + vuint32m4_t ctr_m4 = __riscv_vrev8_v_u32m4(ctrle_m4, vl32m4); + vuint32m4_t data_blks; - memory_barrier_with_vec(add2); - memory_barrier_with_vec(add3); - memory_barrier_with_vec(add4); + ctrle_m4 = __riscv_vadd_vx_u32m4_mu(lane3_mask_m4, ctrle_m4, + ctrle_m4, max_blocks_m4, + vl32m4); - for (; nblocks >= 4; nblocks -= 4) - { - vuint32m4_t data4blks; - vuint32m4_t ctr4blks; + data_blks = unaligned_load_u32m4((const void *)inbuf, vl32m4); - /* detect if 8-bit carry handling is needed */ - if (UNLIKELY(((ctrlow += 4) & 0xff) <= 3)) - { - vuint32m1_t add_1 = __riscv_vle32_v_u32m1(le_add, vl); - vuint32m1_t add_2 = __riscv_vadd_vv_u32m1(add_1, add_1, vl); - vuint32m1_t ctr_le; - vuint32m1_t ctr_1; - vuint32m1_t ctr_2; - vuint32m1_t ctr_3; - vuint32m1_t ctr_4; - - /* Byte swap counter */ - ctr_le = bswap128_u32m1(ctr, vl); - - /* Addition with carry handling */ - ctr_1 = __riscv_vadd_vv_u32m1(ctr_le, add_1, vl); - ctr_2 = __riscv_vadd_vv_u32m1(ctr_le, add_2, vl); - ctr_3 = __riscv_vadd_vv_u32m1(ctr_1, add_2, vl); - ctr_4 = __riscv_vadd_vv_u32m1(ctr_2, add_2, vl); - - /* Byte swap counters */ - ctr_1 = bswap128_u32m1(ctr_1, vl); - ctr_2 = bswap128_u32m1(ctr_2, vl); - ctr_3 = bswap128_u32m1(ctr_3, vl); - ctr_4 = bswap128_u32m1(ctr_4, vl); - - ctr4blks = merge_4x_u32m1_to_u32m4(ctr, ctr_1, ctr_2, ctr_3); - ctr = ctr_4; - } - else - { - /* Fast path addition without carry handling */ - vuint32m1_t ctr0 = ctr; - vuint32m1_t ctr1 = __riscv_vadd_vv_u32m1(ctr, add1, vl); - vuint32m1_t ctr2 = __riscv_vadd_vv_u32m1(ctr, add2, vl); - vuint32m1_t ctr3 = __riscv_vadd_vv_u32m1(ctr, add3, vl); + AES_CRYPT(e, m4, LOAD_KEY, rk, rounds, ctr_m4, vl32m4); - ctr = __riscv_vadd_vv_u32m1(ctr, add4, vl); + data_blks = __riscv_vxor_vv_u32m4(ctr_m4, data_blks, vl32m4); + unaligned_store_u32m4((void *)outbuf, data_blks, vl32m4); - ctr4blks = merge_4x_u32m1_to_u32m4(ctr0, ctr1, ctr2, ctr3); + inbuf += max_blocks_m4 * BLOCKSIZE; + outbuf += max_blocks_m4 * BLOCKSIZE; + nblocks -= max_blocks_m4; } + while (nblocks >= max_blocks_m4); + + ctrle_m2 = __riscv_vget_v_u32m4_u32m2(ctrle_m4, 0); + } + + if (nblocks) + PRELOAD_ROUND_KEYS_RK10_13 (rk, rounds); + + while (nblocks && nblocks >= max_blocks_m2) + { + vuint32m2_t ctr_m2 = __riscv_vrev8_v_u32m2(ctrle_m2, vl32m2); + vuint32m2_t data_blks; + + ctrle_m2 = __riscv_vadd_vx_u32m2_mu(lane3_mask_m2, ctrle_m2, + ctrle_m2, max_blocks_m2, vl32m2); - data4blks = unaligned_load_u32m4(inbuf, vl * 4); + data_blks = unaligned_load_u32m2((const void *)inbuf, vl32m2); - AES_CRYPT(e, m4, rounds, ctr4blks, vl * 4); + AES_CRYPT(e, m2, GET_PRELOADED_KEY, rk, rounds, ctr_m2, vl32m2); - data4blks = __riscv_vxor_vv_u32m4(ctr4blks, data4blks, vl * 4); - unaligned_store_u32m4(outbuf, data4blks, vl * 4); + data_blks = __riscv_vxor_vv_u32m2(ctr_m2, data_blks, vl32m2); + unaligned_store_u32m2((void *)outbuf, data_blks, vl32m2); - inbuf += 4 * BLOCKSIZE; - outbuf += 4 * BLOCKSIZE; + inbuf += max_blocks_m2 * BLOCKSIZE; + outbuf += max_blocks_m2 * BLOCKSIZE; + nblocks -= max_blocks_m2; } + + ctrle_m1 = __riscv_vget_v_u32m2_u32m1(ctrle_m2, 0); } + else if (nblocks) + PRELOAD_ROUND_KEYS_RK10_13 (rk, rounds); - for (; nblocks; nblocks--) + while (nblocks) { - vuint32m1_t block = ctr; - vuint32m1_t data = unaligned_load_u32m1(inbuf, vl); - - /* detect if 8-bit carry handling is needed */ - if (UNLIKELY((++ctrlow & 0xff) == 0)) - { - vuint32m1_t add_val = __riscv_vle32_v_u32m1(le_add, vl); + size_t curr_nblks = nblocks_to_nblocks_per_m1(max_blocks_m1, nblocks); + vuint32m1_t ctr_m1; + vuint32m1_t data_blks; - /* Byte swap counter */ - ctr = bswap128_u32m1(ctr, vl); + vl32m1 = curr_nblks * 4; - /* Addition with carry handling */ - ctr = __riscv_vadd_vv_u32m1(ctr, add_val, vl); + ctr_m1 = __riscv_vrev8_v_u32m1(ctrle_m1, vl32m1); + ctrle_m1 = __riscv_vadd_vx_u32m1_mu(lane3_mask_m1, ctrle_m1, ctrle_m1, + curr_nblks, vl32m1); - /* Byte swap counter */ - ctr = bswap128_u32m1(ctr, vl); - } - else - { - /* Fast path addition without carry handling */ - ctr = __riscv_vadd_vv_u32m1(ctr, add1, vl); - } + data_blks = unaligned_load_u32m1((const void *)inbuf, vl32m1); - AES_CRYPT(e, m1, rounds, block, vl); + AES_CRYPT(e, m1, GET_PRELOADED_KEY, rk, rounds, ctr_m1, vl32m1); - data = __riscv_vxor_vv_u32m1(block, data, vl); - unaligned_store_u32m1(outbuf, data, vl); + data_blks = __riscv_vxor_vv_u32m1(ctr_m1, data_blks, vl32m1); + unaligned_store_u32m1((void *)outbuf, data_blks, vl32m1); - inbuf += BLOCKSIZE; - outbuf += BLOCKSIZE; + inbuf += curr_nblks * BLOCKSIZE; + outbuf += curr_nblks * BLOCKSIZE; + nblocks -= curr_nblks; } - __riscv_vse32_v_u32m1((void *)ctr_arg, ctr, vl); + ctr_arg[15] = ctrlow; + ctr_arg[14] = ctrlow >> 8; clear_vec_regs(); } @@ -888,77 +982,145 @@ _gcry_aes_riscv_zvkned_ctr32le_enc (void *context, unsigned char *ctr_arg, void *outbuf_arg, const void *inbuf_arg, size_t nblocks) { - static const u32 add_u32_array[4][16] = - { - { 1, }, { 2, }, { 3, }, { 4, } - }; RIJNDAEL_context *ctx = context; unsigned char *outbuf = outbuf_arg; const unsigned char *inbuf = inbuf_arg; const u32 *rk = ctx->keyschenc32[0]; int rounds = ctx->rounds; - size_t vl = 4; - vuint32m1_t ctr; - vuint32m1_t add1; + size_t blk_vl32 = BLOCKSIZE / 4; + size_t max_blocks_m1 = __riscv_vsetvlmax_e32m1() / 4; + size_t max_blocks_m2 = max_blocks_m1 * 2; + size_t max_blocks_m4 = max_blocks_m1 * 4; + size_t vl32m1 = max_blocks_m1 * 4; + size_t vl32m2 = max_blocks_m2 * 4; + size_t vl32m4 = max_blocks_m4 * 4; + vbool32_t lane0_mask_m1; + vuint32m1_t ctrle_m1; + u32 ctrlow; ROUND_KEY_VARIABLES; + ROUND_KEY_VARIABLES_RK10_13; - PRELOAD_ROUND_KEYS (rk, rounds, vl); + PRELOAD_ROUND_KEYS (rk, rounds); - add1 = __riscv_vle32_v_u32m1(add_u32_array[0], vl); - ctr = __riscv_vle32_v_u32m1((void *)ctr_arg, vl); + ctrlow = *(u32 *)((void *)ctr_arg); + ctrlow += nblocks; - memory_barrier_with_vec(add1); + /* Prepare full m1 wide counter */ + ctrle_m1 = __riscv_vle32_v_u32m1((void *)ctr_arg, blk_vl32); + lane0_mask_m1 = __riscv_vmseq_vx_u32m1_b32( + __riscv_vand_vx_u32m1(__riscv_vid_v_u32m1(vl32m1), 3, vl32m1), 0, + vl32m1); + if (max_blocks_m1 > 1) + { + ctrle_m1 = broadcast128_u32m1_u32m1(ctrle_m1, vl32m1); + ctrle_m1 = __riscv_vadd_vv_u32m1_mu(lane0_mask_m1, ctrle_m1, ctrle_m1, + __riscv_vsrl_vx_u32m1(__riscv_vid_v_u32m1(vl32m1), 2, vl32m1), + vl32m1); + } - if (nblocks >= 4) + if (nblocks >= max_blocks_m2) { - vuint32m1_t add2 = __riscv_vle32_v_u32m1(add_u32_array[1], vl); - vuint32m1_t add3 = __riscv_vle32_v_u32m1(add_u32_array[2], vl); - vuint32m1_t add4 = __riscv_vle32_v_u32m1(add_u32_array[3], vl); + vbool16_t lane0_mask_m2 = __riscv_vmseq_vx_u32m2_b16( + __riscv_vand_vx_u32m2(__riscv_vid_v_u32m2(vl32m2), 3, vl32m2), 0, + vl32m2); + vuint32m2_t ctrle_m2 = __riscv_vset_v_u32m1_u32m2( + __riscv_vundefined_u32m2(), 0, ctrle_m1); + + /* Double counter width from m1 to m2 */ + ctrle_m2 = __riscv_vset_v_u32m1_u32m2(ctrle_m2, 1, + __riscv_vadd_vx_u32m1_mu(lane0_mask_m1, ctrle_m1, ctrle_m1, + max_blocks_m1, vl32m1)); + + if (nblocks >= max_blocks_m4) + { + vbool8_t lane0_mask_m4 = __riscv_vmseq_vx_u32m4_b8( + __riscv_vand_vx_u32m4(__riscv_vid_v_u32m4(vl32m4), 3, vl32m4), + 0, vl32m4); + vuint32m4_t ctrle_m4 = __riscv_vset_v_u32m2_u32m4( + __riscv_vundefined_u32m4(), 0, ctrle_m2); + + /* Double counter width from m2 to m4 */ + ctrle_m4 = __riscv_vset_v_u32m2_u32m4(ctrle_m4, 1, + __riscv_vadd_vx_u32m2_mu(lane0_mask_m2, ctrle_m2, ctrle_m2, + max_blocks_m2, vl32m2)); + + do + { + vuint32m4_t ctr_m4 = ctrle_m4; + vuint32m4_t data_blks; + + ctrle_m4 = __riscv_vadd_vx_u32m4_mu(lane0_mask_m4, ctrle_m4, + ctrle_m4, max_blocks_m4, + vl32m4); + + data_blks = unaligned_load_u32m4((const void *)inbuf, vl32m4); + + AES_CRYPT(e, m4, LOAD_KEY, rk, rounds, ctr_m4, vl32m4); - memory_barrier_with_vec(add2); - memory_barrier_with_vec(add3); - memory_barrier_with_vec(add4); + data_blks = __riscv_vxor_vv_u32m4(ctr_m4, data_blks, vl32m4); + unaligned_store_u32m4((void *)outbuf, data_blks, vl32m4); - for (; nblocks >= 4; nblocks -= 4) + inbuf += max_blocks_m4 * BLOCKSIZE; + outbuf += max_blocks_m4 * BLOCKSIZE; + nblocks -= max_blocks_m4; + } + while (nblocks >= max_blocks_m4); + + ctrle_m2 = __riscv_vget_v_u32m4_u32m2(ctrle_m4, 0); + } + + if (nblocks) + PRELOAD_ROUND_KEYS_RK10_13 (rk, rounds); + + while (nblocks && nblocks >= max_blocks_m2) { - vuint32m1_t ctr1 = __riscv_vadd_vv_u32m1(ctr, add1, vl); - vuint32m1_t ctr2 = __riscv_vadd_vv_u32m1(ctr, add2, vl); - vuint32m1_t ctr3 = __riscv_vadd_vv_u32m1(ctr, add3, vl); - vuint32m4_t ctr4blks; - vuint32m4_t data4blks; + vuint32m2_t ctr_m2 = ctrle_m2; + vuint32m2_t data_blks; - ctr4blks = merge_4x_u32m1_to_u32m4(ctr, ctr1, ctr2, ctr3); - ctr = __riscv_vadd_vv_u32m1(ctr, add4, vl); + ctrle_m2 = __riscv_vadd_vx_u32m2_mu(lane0_mask_m2, ctrle_m2, + ctrle_m2, max_blocks_m2, vl32m2); - data4blks = unaligned_load_u32m4(inbuf, vl * 4); + data_blks = unaligned_load_u32m2((const void *)inbuf, vl32m2); - AES_CRYPT(e, m4, rounds, ctr4blks, vl * 4); + AES_CRYPT(e, m2, GET_PRELOADED_KEY, rk, rounds, ctr_m2, vl32m2); - data4blks = __riscv_vxor_vv_u32m4(ctr4blks, data4blks, vl * 4); - unaligned_store_u32m4(outbuf, data4blks, vl * 4); + data_blks = __riscv_vxor_vv_u32m2(ctr_m2, data_blks, vl32m2); + unaligned_store_u32m2((void *)outbuf, data_blks, vl32m2); - inbuf += 4 * BLOCKSIZE; - outbuf += 4 * BLOCKSIZE; + inbuf += max_blocks_m2 * BLOCKSIZE; + outbuf += max_blocks_m2 * BLOCKSIZE; + nblocks -= max_blocks_m2; } + + ctrle_m1 = __riscv_vget_v_u32m2_u32m1(ctrle_m2, 0); } + else if (nblocks) + PRELOAD_ROUND_KEYS_RK10_13 (rk, rounds); - for (; nblocks; nblocks--) + while (nblocks) { - vuint32m1_t block = ctr; - vuint32m1_t data = unaligned_load_u32m1(inbuf, vl); + size_t curr_nblks = nblocks_to_nblocks_per_m1(max_blocks_m1, nblocks); + vuint32m1_t ctr_m1 = ctrle_m1; + vuint32m1_t data_blks; + + vl32m1 = curr_nblks * 4; + + ctrle_m1 = __riscv_vadd_vx_u32m1_mu(lane0_mask_m1, ctrle_m1, ctrle_m1, + curr_nblks, vl32m1); - ctr = __riscv_vadd_vv_u32m1(ctr, add1, vl); + data_blks = unaligned_load_u32m1((const void *)inbuf, vl32m1); - AES_CRYPT(e, m1, rounds, block, vl); + AES_CRYPT(e, m1, GET_PRELOADED_KEY, rk, rounds, ctr_m1, vl32m1); - data = __riscv_vxor_vv_u32m1(block, data, vl); - unaligned_store_u32m1(outbuf, data, vl); + data_blks = __riscv_vxor_vv_u32m1(ctr_m1, data_blks, vl32m1); + unaligned_store_u32m1((void *)outbuf, data_blks, vl32m1); - inbuf += BLOCKSIZE; - outbuf += BLOCKSIZE; + inbuf += curr_nblks * BLOCKSIZE; + outbuf += curr_nblks * BLOCKSIZE; + nblocks -= curr_nblks; } - __riscv_vse32_v_u32m1((void *)ctr_arg, ctr, vl); + *(u32 *)(void *)ctr_arg = ctrlow; clear_vec_regs(); } @@ -973,49 +1135,94 @@ _gcry_aes_riscv_zvkned_cfb_dec (void *context, unsigned char *iv_arg, const unsigned char *inbuf = inbuf_arg; const u32 *rk = ctx->keyschenc32[0]; int rounds = ctx->rounds; - size_t vl = 4; + size_t blk_vl32 = BLOCKSIZE / 4; + size_t max_blocks_m1 = __riscv_vsetvlmax_e32m1() / 4; + size_t max_blocks_m2 = max_blocks_m1 * 2; + size_t max_blocks_m4 = max_blocks_m1 * 4; vuint32m1_t iv; + vuint32m2_t iv_m2; + vuint32m4_t iv_m4; ROUND_KEY_VARIABLES; + ROUND_KEY_VARIABLES_RK10_13; - PRELOAD_ROUND_KEYS (rk, rounds, vl); + PRELOAD_ROUND_KEYS (rk, rounds); - iv = __riscv_vle32_v_u32m1((void *)iv_arg, vl); + iv = __riscv_vle32_v_u32m1((void *)iv_arg, blk_vl32); - for (; nblocks >= 4; nblocks -= 4) + iv_m4 = __riscv_vundefined_u32m4(); + while (nblocks >= max_blocks_m4) { - vuint32m4_t data4blks = unaligned_load_u32m4(inbuf, vl * 4); - vuint32m1_t new_iv = __riscv_vlmul_trunc_v_u32m4_u32m1( - __riscv_vslidedown_vx_u32m4(data4blks, 12, 16)); - vuint32m4_t iv_m4 = __riscv_vlmul_ext_v_u32m1_u32m4(iv); - vuint32m4_t iv4blks = __riscv_vslideup_vx_u32m4(iv_m4, data4blks, 4, 16); + size_t vl_m4 = max_blocks_m4 * 4; + size_t vl_m1 = max_blocks_m1 * 4; + vuint32m4_t data_blks = unaligned_load_u32m4(inbuf, vl_m4); + vuint32m1_t new_iv = __riscv_vslidedown_vx_u32m1( + __riscv_vget_v_u32m4_u32m1(data_blks, 3), vl_m1 - blk_vl32, vl_m1); + + iv_m4 = __riscv_vset_v_u32m1_u32m4(iv_m4, 0, iv); + iv_m4 = __riscv_vslideup_vx_u32m4(iv_m4, data_blks, blk_vl32, vl_m4); iv = new_iv; - AES_CRYPT(e, m4, rounds, iv4blks, vl * 4); + AES_CRYPT(e, m4, LOAD_KEY, rk, rounds, iv_m4, vl_m4); - data4blks = __riscv_vxor_vv_u32m4(iv4blks, data4blks, vl * 4); - unaligned_store_u32m4(outbuf, data4blks, vl * 4); + data_blks = __riscv_vxor_vv_u32m4(iv_m4, data_blks, vl_m4); + unaligned_store_u32m4(outbuf, data_blks, vl_m4); - inbuf += 4 * BLOCKSIZE; - outbuf += 4 * BLOCKSIZE; + inbuf += max_blocks_m4 * BLOCKSIZE; + outbuf += max_blocks_m4 * BLOCKSIZE; + nblocks -= max_blocks_m4; } - for (; nblocks; nblocks--) + if (nblocks) + PRELOAD_ROUND_KEYS_RK10_13 (rk, rounds); + + iv_m2 = __riscv_vundefined_u32m2(); + while (nblocks && nblocks >= max_blocks_m2) { - vuint32m1_t data = unaligned_load_u32m1(inbuf, vl); - vuint32m1_t new_iv = data; + size_t vl_m2 = max_blocks_m2 * 4; + size_t vl_m1 = max_blocks_m1 * 4; + vuint32m2_t data_blks = unaligned_load_u32m2(inbuf, vl_m2); + vuint32m1_t new_iv = __riscv_vslidedown_vx_u32m1( + __riscv_vget_v_u32m2_u32m1(data_blks, 1), vl_m1 - blk_vl32, vl_m1); - AES_CRYPT(e, m1, rounds, iv, vl); + iv_m2 = __riscv_vset_v_u32m1_u32m2(iv_m2, 0, iv); + iv_m2 = __riscv_vslideup_vx_u32m2(iv_m2, data_blks, blk_vl32, vl_m2); - data = __riscv_vxor_vv_u32m1(iv, data, vl); - unaligned_store_u32m1(outbuf, data, vl); iv = new_iv; - inbuf += BLOCKSIZE; - outbuf += BLOCKSIZE; + AES_CRYPT(e, m2, GET_PRELOADED_KEY, rk, rounds, iv_m2, vl_m2); + + data_blks = __riscv_vxor_vv_u32m2(iv_m2, data_blks, vl_m2); + unaligned_store_u32m2(outbuf, data_blks, vl_m2); + + inbuf += max_blocks_m2 * BLOCKSIZE; + outbuf += max_blocks_m2 * BLOCKSIZE; + nblocks -= max_blocks_m2; } - __riscv_vse32_v_u32m1((void *)iv_arg, iv, vl); + while (nblocks) + { + size_t curr_nblks = nblocks_to_nblocks_per_m1(max_blocks_m1, nblocks); + size_t vl_m1 = curr_nblks * 4; + vuint32m1_t data_blks = unaligned_load_u32m1(inbuf, vl_m1); + vuint32m1_t new_iv = __riscv_vslidedown_vx_u32m1(data_blks, + vl_m1 - blk_vl32, vl_m1); + vuint32m1_t iv_m1 = __riscv_vslideup_vx_u32m1(iv, data_blks, blk_vl32, + vl_m1); + + iv = new_iv; + + AES_CRYPT(e, m1, GET_PRELOADED_KEY, rk, rounds, iv_m1, vl_m1); + + data_blks = __riscv_vxor_vv_u32m1(iv_m1, data_blks, vl_m1); + unaligned_store_u32m1(outbuf, data_blks, vl_m1); + + inbuf += curr_nblks * BLOCKSIZE; + outbuf += curr_nblks * BLOCKSIZE; + nblocks -= curr_nblks; + } + + __riscv_vse32_v_u32m1((void *)iv_arg, iv, blk_vl32); clear_vec_regs(); } @@ -1030,594 +1237,583 @@ _gcry_aes_riscv_zvkned_cbc_dec (void *context, unsigned char *iv_arg, const unsigned char *inbuf = inbuf_arg; const u32 *rk = ctx->keyschdec32[0]; int rounds = ctx->rounds; - size_t vl = 4; + size_t blk_vl32 = BLOCKSIZE / 4; + size_t max_blocks_m1 = __riscv_vsetvlmax_e32m1() / 4; + size_t max_blocks_m2 = max_blocks_m1 * 2; + size_t max_blocks_m4 = max_blocks_m1 * 4; vuint32m1_t iv; + vuint32m2_t iv_m2; + vuint32m4_t iv_m4; ROUND_KEY_VARIABLES; + ROUND_KEY_VARIABLES_RK10_13; - if (!ctx->decryption_prepared) + if (UNLIKELY(!ctx->decryption_prepared)) { do_prepare_decryption(ctx); ctx->decryption_prepared = 1; } - PRELOAD_ROUND_KEYS (rk, rounds, vl); + PRELOAD_ROUND_KEYS (rk, rounds); - iv = __riscv_vle32_v_u32m1((void *)iv_arg, vl); + iv = __riscv_vle32_v_u32m1((void *)iv_arg, blk_vl32); - for (; nblocks >= 4; nblocks -= 4) + iv_m4 = __riscv_vundefined_u32m4(); + while (nblocks >= max_blocks_m4) { - vuint32m4_t data4blks = unaligned_load_u32m4(inbuf, vl * 4); - vuint32m4_t iv_m4 = __riscv_vlmul_ext_v_u32m1_u32m4(iv); - vuint32m4_t iv4blks = __riscv_vslideup_vx_u32m4(iv_m4, data4blks, 4, 16); + size_t vl_m4 = max_blocks_m4 * 4; + size_t vl_m1 = max_blocks_m1 * 4; + vuint32m4_t data_blks = unaligned_load_u32m4(inbuf, vl_m4); - iv = __riscv_vlmul_trunc_v_u32m4_u32m1( - __riscv_vslidedown_vx_u32m4(data4blks, 12, 16)); + iv_m4 = __riscv_vset_v_u32m1_u32m4(iv_m4, 0, iv); + iv_m4 = __riscv_vslideup_vx_u32m4(iv_m4, data_blks, blk_vl32, vl_m4); - AES_CRYPT(d, m4, rounds, data4blks, vl * 4); + iv = __riscv_vslidedown_vx_u32m1( + __riscv_vget_v_u32m4_u32m1(data_blks, 3), vl_m1 - blk_vl32, vl_m1); - data4blks = __riscv_vxor_vv_u32m4(iv4blks, data4blks, vl * 4); - unaligned_store_u32m4(outbuf, data4blks, vl * 4); - - inbuf += 4 * BLOCKSIZE; - outbuf += 4 * BLOCKSIZE; - } - - for (; nblocks; nblocks--) - { - vuint32m1_t data = unaligned_load_u32m1(inbuf, vl); - vuint32m1_t new_iv = data; + AES_CRYPT(d, m4, LOAD_KEY, rk, rounds, data_blks, vl_m4); - AES_CRYPT(d, m1, rounds, data, vl); + data_blks = __riscv_vxor_vv_u32m4(iv_m4, data_blks, vl_m4); + unaligned_store_u32m4(outbuf, data_blks, vl_m4); - data = __riscv_vxor_vv_u32m1(iv, data, vl); - unaligned_store_u32m1(outbuf, data, vl); - iv = new_iv; - - inbuf += BLOCKSIZE; - outbuf += BLOCKSIZE; + inbuf += max_blocks_m4 * BLOCKSIZE; + outbuf += max_blocks_m4 * BLOCKSIZE; + nblocks -= max_blocks_m4; } - __riscv_vse32_v_u32m1((void *)iv_arg, iv, vl); - - clear_vec_regs(); -} - -static ASM_FUNC_ATTR_NOINLINE FUNC_ATTR_OPT_O2 size_t -aes_riscv_ocb_enc (gcry_cipher_hd_t c, void *outbuf_arg, - const void *inbuf_arg, size_t nblocks) -{ - RIJNDAEL_context *ctx = (void *)&c->context.c; - unsigned char *outbuf = outbuf_arg; - const unsigned char *inbuf = inbuf_arg; - u64 n = c->u_mode.ocb.data_nblocks; - const u32 *rk = ctx->keyschenc32[0]; - int rounds = ctx->rounds; - size_t vl = 4; - vuint32m1_t iv; - vuint32m1_t ctr; - ROUND_KEY_VARIABLES; - - PRELOAD_ROUND_KEYS (rk, rounds, vl); + if (nblocks) + PRELOAD_ROUND_KEYS_RK10_13 (rk, rounds); - /* Preload Offset and Checksum */ - iv = __riscv_vle32_v_u32m1((void *)c->u_iv.iv, vl); - ctr = __riscv_vle32_v_u32m1((void *)c->u_ctr.ctr, vl); - - if (nblocks >= 4) + iv_m2 = __riscv_vundefined_u32m2(); + while (nblocks && nblocks >= max_blocks_m2) { - vuint32m1_t zero = __riscv_vmv_v_x_u32m1(0, vl); - vuint32m4_t ctr4blks = merge_4x_u32m1_to_u32m4(ctr, zero, zero, zero); + size_t vl_m2 = max_blocks_m2 * 4; + size_t vl_m1 = max_blocks_m1 * 4; + vuint32m2_t data_blks = unaligned_load_u32m2(inbuf, vl_m2); - for (; nblocks >= 4; nblocks -= 4) - { - const void *l; - vuint32m1_t l_ntzi; - vuint32m4_t data4blks = unaligned_load_u32m4(inbuf, vl * 4); - vuint32m1_t offset0, offset1, offset2, offset3; - vuint32m4_t offsets; + iv_m2 = __riscv_vset_v_u32m1_u32m2(iv_m2, 0, iv); + iv_m2 = __riscv_vslideup_vx_u32m2(iv_m2, data_blks, blk_vl32, vl_m2); - /* Checksum_i = Checksum_{i-1} xor P_i */ - ctr4blks = __riscv_vxor_vv_u32m4(ctr4blks, data4blks, vl * 4); - - /* Offset_i = Offset_{i-1} xor L_{ntz(i)} */ - /* C_i = Offset_i xor ENCIPHER(K, P_i xor Offset_i) */ - l = ocb_get_l(c, ++n); - l_ntzi = __riscv_vle32_v_u32m1(l, vl); - iv = __riscv_vxor_vv_u32m1(iv, l_ntzi, vl); - offset0 = iv; - - l = ocb_get_l(c, ++n); - l_ntzi = __riscv_vle32_v_u32m1(l, vl); - iv = __riscv_vxor_vv_u32m1(iv, l_ntzi, vl); - offset1 = iv; - - l = ocb_get_l(c, ++n); - l_ntzi = __riscv_vle32_v_u32m1(l, vl); - iv = __riscv_vxor_vv_u32m1(iv, l_ntzi, vl); - offset2 = iv; + iv = __riscv_vslidedown_vx_u32m1( + __riscv_vget_v_u32m2_u32m1(data_blks, 1), vl_m1 - blk_vl32, vl_m1); - l = ocb_get_l(c, ++n); - l_ntzi = __riscv_vle32_v_u32m1(l, vl); - iv = __riscv_vxor_vv_u32m1(iv, l_ntzi, vl); - offset3 = iv; + AES_CRYPT(d, m2, GET_PRELOADED_KEY, rk, rounds, data_blks, vl_m2); - offsets = merge_4x_u32m1_to_u32m4(offset0, offset1, offset2, offset3); + data_blks = __riscv_vxor_vv_u32m2(iv_m2, data_blks, vl_m2); + unaligned_store_u32m2(outbuf, data_blks, vl_m2); - data4blks = __riscv_vxor_vv_u32m4(offsets, data4blks, vl * 4); - - AES_CRYPT(e, m4, rounds, data4blks, vl * 4); - - data4blks = __riscv_vxor_vv_u32m4(offsets, data4blks, vl * 4); - - unaligned_store_u32m4(outbuf, data4blks, vl * 4); - - inbuf += 4 * BLOCKSIZE; - outbuf += 4 * BLOCKSIZE; - } - - /* Checksum_i = Checksum_{i-1} xor P_i */ - { - vuint32m1x4_t ctr0123 = split_u32m4_to_4x_u32m1(ctr4blks); - ctr = __riscv_vxor_vv_u32m1(__riscv_vget_v_u32m1x4_u32m1(ctr0123, 0), - __riscv_vget_v_u32m1x4_u32m1(ctr0123, 1), vl); - ctr = __riscv_vxor_vv_u32m1(ctr, __riscv_vget_v_u32m1x4_u32m1(ctr0123, 2), vl); - ctr = __riscv_vxor_vv_u32m1(ctr, __riscv_vget_v_u32m1x4_u32m1(ctr0123, 3), vl); - } + inbuf += max_blocks_m2 * BLOCKSIZE; + outbuf += max_blocks_m2 * BLOCKSIZE; + nblocks -= max_blocks_m2; } - for (; nblocks; nblocks--) + while (nblocks) { - const void *l; - vuint32m1_t l_ntzi; - vuint32m1_t data; - - data = unaligned_load_u32m1(inbuf, vl); + size_t curr_nblks = nblocks_to_nblocks_per_m1(max_blocks_m1, nblocks); + size_t vl_m1 = curr_nblks * 4; + vuint32m1_t data_blks = unaligned_load_u32m1(inbuf, vl_m1); + vuint32m1_t iv_m1 = __riscv_vslideup_vx_u32m1(iv, data_blks, blk_vl32, + vl_m1); - /* Checksum_i = Checksum_{i-1} xor P_i */ - ctr = __riscv_vxor_vv_u32m1(ctr, data, vl); + iv = __riscv_vslidedown_vx_u32m1(data_blks, vl_m1 - blk_vl32, vl_m1); - /* Offset_i = Offset_{i-1} xor L_{ntz(i)} */ - /* C_i = Offset_i xor ENCIPHER(K, P_i xor Offset_i) */ - l = ocb_get_l(c, ++n); - l_ntzi = __riscv_vle32_v_u32m1(l, vl); - iv = __riscv_vxor_vv_u32m1(iv, l_ntzi, vl); + AES_CRYPT(d, m1, GET_PRELOADED_KEY, rk, rounds, data_blks, vl_m1); - data = __riscv_vxor_vv_u32m1(data, iv, vl); + data_blks = __riscv_vxor_vv_u32m1(iv_m1, data_blks, vl_m1); + unaligned_store_u32m1(outbuf, data_blks, vl_m1); - AES_CRYPT(e, m1, rounds, data, vl); + inbuf += curr_nblks * BLOCKSIZE; + outbuf += curr_nblks * BLOCKSIZE; + nblocks -= curr_nblks; + } - data = __riscv_vxor_vv_u32m1(iv, data, vl); - unaligned_store_u32m1(outbuf, data, vl); + __riscv_vse32_v_u32m1((void *)iv_arg, iv, blk_vl32); - inbuf += BLOCKSIZE; - outbuf += BLOCKSIZE; - } + clear_vec_regs(); +} - c->u_mode.ocb.data_nblocks = n; +static ASM_FUNC_ATTR_INLINE vuint32m1_t +ocb_offsets_m1 (gcry_cipher_hd_t c, u64 *np, vuint32m1_t *ivp, + vuint32m1_t offsets, size_t nblocks, size_t vl_m1) +{ + vuint32m1_t iv = *ivp; + size_t blk_vl32 = BLOCKSIZE / 4; + size_t k = 0; - __riscv_vse32_v_u32m1((void *)c->u_iv.iv, iv, vl); - __riscv_vse32_v_u32m1((void *)c->u_ctr.ctr, ctr, vl); + do + { + const void *l = ocb_get_l(c, ++(*np)); + vuint32m1_t l_ntzi = __riscv_vle32_v_u32m1(l, blk_vl32); - clear_vec_regs(); + iv = __riscv_vxor_vv_u32m1(iv, l_ntzi, blk_vl32); + offsets = __riscv_vslideup_vx_u32m1(offsets, iv, k * blk_vl32, vl_m1); + } + while (++k < nblocks); - return 0; + *ivp = iv; + return offsets; } -static ASM_FUNC_ATTR_NOINLINE FUNC_ATTR_OPT_O2 size_t -aes_riscv_ocb_dec (gcry_cipher_hd_t c, void *outbuf_arg, - const void *inbuf_arg, size_t nblocks) +static ASM_FUNC_ATTR_INLINE FUNC_ATTR_OPT_O2 size_t +aes_riscv_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg, + const void *inbuf_arg, size_t nblocks, int encrypt) { RIJNDAEL_context *ctx = (void *)&c->context.c; unsigned char *outbuf = outbuf_arg; const unsigned char *inbuf = inbuf_arg; - u64 n = c->u_mode.ocb.data_nblocks; - const u32 *rk = ctx->keyschdec32[0]; + const u32 *rk = encrypt ? ctx->keyschenc32[0] : ctx->keyschdec32[0]; + int auth = encrypt < 0; int rounds = ctx->rounds; - size_t vl = 4; + size_t blk_vl32 = BLOCKSIZE / 4; + size_t max_blocks_m1 = __riscv_vsetvlmax_e32m1() / 4; + size_t max_blocks_m2 = max_blocks_m1 * 2; + size_t max_blocks_m4 = max_blocks_m1 * 4; + size_t vl_m1 = max_blocks_m1 * 4; + size_t vl_m2 = max_blocks_m2 * 4; + size_t vl_m4 = max_blocks_m4 * 4; vuint32m1_t iv; vuint32m1_t ctr; + vuint32m1_t offs_m1 = __riscv_vundefined_u32m1(); + size_t h; + u64 n; ROUND_KEY_VARIABLES; + ROUND_KEY_VARIABLES_RK10_13; - if (!ctx->decryption_prepared) + if (!encrypt && UNLIKELY(!ctx->decryption_prepared)) { do_prepare_decryption(ctx); ctx->decryption_prepared = 1; } - PRELOAD_ROUND_KEYS (rk, rounds, vl); + PRELOAD_ROUND_KEYS (rk, rounds); /* Preload Offset and Checksum */ - iv = __riscv_vle32_v_u32m1((void *)c->u_iv.iv, vl); - ctr = __riscv_vle32_v_u32m1((void *)c->u_ctr.ctr, vl); + if (auth) + { + n = c->u_mode.ocb.aad_nblocks; + iv = __riscv_vle32_v_u32m1((void *)c->u_mode.ocb.aad_offset, blk_vl32); + ctr = __riscv_vle32_v_u32m1((void *)c->u_mode.ocb.aad_sum, blk_vl32); + } + else + { + n = c->u_mode.ocb.data_nblocks; + iv = __riscv_vle32_v_u32m1((void *)c->u_iv.iv, blk_vl32); + ctr = __riscv_vle32_v_u32m1((void *)c->u_ctr.ctr, blk_vl32); + } + ctr = __riscv_vmv_v_v_u32m1_tu(__riscv_vmv_v_x_u32m1(0, vl_m1), ctr, + blk_vl32); - if (nblocks >= 4) + if (nblocks >= max_blocks_m4) { - vuint32m1_t zero = __riscv_vmv_v_x_u32m1(0, vl); - vuint32m4_t ctr4blks = merge_4x_u32m1_to_u32m4(ctr, zero, zero, zero); + vuint32m4_t offs_m4 = __riscv_vundefined_u32m4(); + vuint32m4_t ctr_m4 = + __riscv_vset_v_u32m1_u32m4(__riscv_vmv_v_x_u32m4(0, vl_m4), 0, ctr); + vuint32m2_t ctr_m2; - for (; nblocks >= 4; nblocks -= 4) + do { - const void *l; - vuint32m1_t l_ntzi; - vuint32m4_t data4blks = unaligned_load_u32m4(inbuf, vl * 4); - vuint32m1_t offset0, offset1, offset2, offset3; - vuint32m4_t offsets; - - /* Offset_i = Offset_{i-1} xor L_{ntz(i)} */ - /* P_i = Offset_i xor ENCIPHER(K, C_i xor Offset_i) */ - l = ocb_get_l(c, ++n); - l_ntzi = __riscv_vle32_v_u32m1(l, vl); - iv = __riscv_vxor_vv_u32m1(iv, l_ntzi, vl); - offset0 = iv; - - l = ocb_get_l(c, ++n); - l_ntzi = __riscv_vle32_v_u32m1(l, vl); - iv = __riscv_vxor_vv_u32m1(iv, l_ntzi, vl); - offset1 = iv; - - l = ocb_get_l(c, ++n); - l_ntzi = __riscv_vle32_v_u32m1(l, vl); - iv = __riscv_vxor_vv_u32m1(iv, l_ntzi, vl); - offset2 = iv; - - l = ocb_get_l(c, ++n); - l_ntzi = __riscv_vle32_v_u32m1(l, vl); - iv = __riscv_vxor_vv_u32m1(iv, l_ntzi, vl); - offset3 = iv; + vuint32m4_t data_blks = unaligned_load_u32m4(inbuf, vl_m4); - offsets = merge_4x_u32m1_to_u32m4(offset0, offset1, offset2, offset3); - - data4blks = __riscv_vxor_vv_u32m4(offsets, data4blks, vl * 4); + if (encrypt > 0) + { + /* Checksum_i = Checksum_{i-1} xor P_i */ + ctr_m4 = __riscv_vxor_vv_u32m4(ctr_m4, data_blks, vl_m4); + } - AES_CRYPT(d, m4, rounds, data4blks, vl * 4); + /* Offset_i = Offset_{i-1} xor L_{ntz(i)} */ + offs_m1 = ocb_offsets_m1(c, &n, &iv, offs_m1, max_blocks_m1, vl_m1); + offs_m4 = __riscv_vset_v_u32m1_u32m4(offs_m4, 0, offs_m1); + offs_m1 = ocb_offsets_m1(c, &n, &iv, offs_m1, max_blocks_m1, vl_m1); + offs_m4 = __riscv_vset_v_u32m1_u32m4(offs_m4, 1, offs_m1); + offs_m1 = ocb_offsets_m1(c, &n, &iv, offs_m1, max_blocks_m1, vl_m1); + offs_m4 = __riscv_vset_v_u32m1_u32m4(offs_m4, 2, offs_m1); + offs_m1 = ocb_offsets_m1(c, &n, &iv, offs_m1, max_blocks_m1, vl_m1); + offs_m4 = __riscv_vset_v_u32m1_u32m4(offs_m4, 3, offs_m1); + + /* P_i = Offset_i xor CIPHER(K, C_i xor Offset_i) */ + data_blks = __riscv_vxor_vv_u32m4(offs_m4, data_blks, vl_m4); + + if (encrypt) + AES_CRYPT(e, m4, LOAD_KEY, rk, rounds, data_blks, vl_m4); + else + AES_CRYPT(d, m4, LOAD_KEY, rk, rounds, data_blks, vl_m4); - data4blks = __riscv_vxor_vv_u32m4(offsets, data4blks, vl * 4); + if (!auth) + { + data_blks = __riscv_vxor_vv_u32m4(offs_m4, data_blks, vl_m4); - unaligned_store_u32m4(outbuf, data4blks, vl * 4); + unaligned_store_u32m4(outbuf, data_blks, vl_m4); + outbuf += max_blocks_m4 * BLOCKSIZE; + } - /* Checksum_i = Checksum_{i-1} xor P_i */ - ctr4blks = __riscv_vxor_vv_u32m4(ctr4blks, data4blks, vl * 4); + if (encrypt <= 0) + { + /* Checksum_i = Checksum_{i-1} xor P_i */ + ctr_m4 = __riscv_vxor_vv_u32m4(ctr_m4, data_blks, vl_m4); + } - inbuf += 4 * BLOCKSIZE; - outbuf += 4 * BLOCKSIZE; + inbuf += max_blocks_m4 * BLOCKSIZE; + nblocks -= max_blocks_m4; } + while (nblocks >= max_blocks_m4); /* Checksum_i = Checksum_{i-1} xor P_i */ - { - vuint32m1x4_t ctr0123 = split_u32m4_to_4x_u32m1(ctr4blks); - ctr = __riscv_vxor_vv_u32m1(__riscv_vget_v_u32m1x4_u32m1(ctr0123, 0), - __riscv_vget_v_u32m1x4_u32m1(ctr0123, 1), vl); - ctr = __riscv_vxor_vv_u32m1(ctr, __riscv_vget_v_u32m1x4_u32m1(ctr0123, 2), vl); - ctr = __riscv_vxor_vv_u32m1(ctr, __riscv_vget_v_u32m1x4_u32m1(ctr0123, 3), vl); - } + ctr_m2 = __riscv_vxor_vv_u32m2(__riscv_vget_v_u32m4_u32m2(ctr_m4, 0), + __riscv_vget_v_u32m4_u32m2(ctr_m4, 1), + vl_m2); + ctr = __riscv_vxor_vv_u32m1(__riscv_vget_v_u32m2_u32m1(ctr_m2, 0), + __riscv_vget_v_u32m2_u32m1(ctr_m2, 1), + vl_m1); } - for (; nblocks; nblocks--) - { - const void *l; - vuint32m1_t l_ntzi; - vuint32m1_t data; - - /* Offset_i = Offset_{i-1} xor L_{ntz(i)} */ - /* P_i = Offset_i xor DECIPHER(K, C_i xor Offset_i) */ - l = ocb_get_l(c, ++n); - l_ntzi = __riscv_vle32_v_u32m1(l, vl); - data = unaligned_load_u32m1(inbuf, vl); - iv = __riscv_vxor_vv_u32m1(iv, l_ntzi, vl); - data = __riscv_vxor_vv_u32m1(data, iv, vl); + if (nblocks) + PRELOAD_ROUND_KEYS_RK10_13 (rk, rounds); - AES_CRYPT(d, m1, rounds, data, vl); - - data = __riscv_vxor_vv_u32m1(iv, data, vl); - unaligned_store_u32m1(outbuf, data, vl); + if (nblocks && nblocks >= max_blocks_m2) + { + vuint32m2_t offs_m2 = __riscv_vundefined_u32m2(); + vuint32m2_t ctr_m2 = + __riscv_vset_v_u32m1_u32m2(__riscv_vmv_v_x_u32m2(0, vl_m2), 0, ctr); - /* Checksum_i = Checksum_{i-1} xor P_i */ - ctr = __riscv_vxor_vv_u32m1(ctr, data, vl); + do + { + vuint32m2_t data_blks = unaligned_load_u32m2(inbuf, vl_m2); - inbuf += BLOCKSIZE; - outbuf += BLOCKSIZE; - } + if (encrypt > 0) + { + /* Checksum_i = Checksum_{i-1} xor P_i */ + ctr_m2 = __riscv_vxor_vv_u32m2(ctr_m2, data_blks, vl_m2); + } - c->u_mode.ocb.data_nblocks = n; + /* Offset_i = Offset_{i-1} xor L_{ntz(i)} */ + offs_m1 = ocb_offsets_m1(c, &n, &iv, offs_m1, max_blocks_m1, vl_m1); + offs_m2 = __riscv_vset_v_u32m1_u32m2(offs_m2, 0, offs_m1); + offs_m1 = ocb_offsets_m1(c, &n, &iv, offs_m1, max_blocks_m1, vl_m1); + offs_m2 = __riscv_vset_v_u32m1_u32m2(offs_m2, 1, offs_m1); - __riscv_vse32_v_u32m1((void *)c->u_iv.iv, iv, vl); - __riscv_vse32_v_u32m1((void *)c->u_ctr.ctr, ctr, vl); + /* P_i = Offset_i xor CIPHER(K, C_i xor Offset_i) */ + data_blks = __riscv_vxor_vv_u32m2(offs_m2, data_blks, vl_m2); - clear_vec_regs(); + if (encrypt) + AES_CRYPT(e, m2, GET_PRELOADED_KEY, rk, rounds, data_blks, vl_m2); + else + AES_CRYPT(d, m2, GET_PRELOADED_KEY, rk, rounds, data_blks, vl_m2); - return 0; -} + if (!auth) + { + data_blks = __riscv_vxor_vv_u32m2(offs_m2, data_blks, vl_m2); -size_t ASM_FUNC_ATTR_NOINLINE FUNC_ATTR_OPT_O2 -_gcry_aes_riscv_zvkned_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg, - const void *inbuf_arg, size_t nblocks, - int encrypt) -{ - if (encrypt) - return aes_riscv_ocb_enc(c, outbuf_arg, inbuf_arg, nblocks); - else - return aes_riscv_ocb_dec(c, outbuf_arg, inbuf_arg, nblocks); -} + unaligned_store_u32m2(outbuf, data_blks, vl_m2); + outbuf += max_blocks_m2 * BLOCKSIZE; + } -size_t ASM_FUNC_ATTR_NOINLINE FUNC_ATTR_OPT_O2 -_gcry_aes_riscv_zvkned_ocb_auth (gcry_cipher_hd_t c, const void *abuf_arg, - size_t nblocks) -{ - RIJNDAEL_context *ctx = (void *)&c->context.c; - const unsigned char *abuf = abuf_arg; - u64 n = c->u_mode.ocb.aad_nblocks; - const u32 *rk = ctx->keyschenc32[0]; - int rounds = ctx->rounds; - size_t vl = 4; - vuint32m1_t iv; - vuint32m1_t ctr; - ROUND_KEY_VARIABLES; + if (encrypt <= 0) + { + /* Checksum_i = Checksum_{i-1} xor P_i */ + ctr_m2 = __riscv_vxor_vv_u32m2(ctr_m2, data_blks, vl_m2); + } - PRELOAD_ROUND_KEYS (rk, rounds, vl); + inbuf += max_blocks_m2 * BLOCKSIZE; + nblocks -= max_blocks_m2; + } + while (nblocks >= max_blocks_m2); - /* Preload Offset and Sum */ - iv = __riscv_vle32_v_u32m1((void *)c->u_mode.ocb.aad_offset, vl); - ctr = __riscv_vle32_v_u32m1((void *)c->u_mode.ocb.aad_sum, vl); + /* Checksum_i = Checksum_{i-1} xor P_i */ + ctr = __riscv_vxor_vv_u32m1(__riscv_vget_v_u32m2_u32m1(ctr_m2, 0), + __riscv_vget_v_u32m2_u32m1(ctr_m2, 1), + vl_m1); + } - if (nblocks >= 4) + while (nblocks) { - vuint32m1_t zero = __riscv_vmv_v_x_u32m1(0, vl); - vuint32m4_t ctr4blks = merge_4x_u32m1_to_u32m4(ctr, zero, zero, zero); - - for (; nblocks >= 4; nblocks -= 4) - { - const void *l; - vuint32m1_t l_ntzi; - vuint32m4_t data4blks = unaligned_load_u32m4(abuf, vl * 4); - vuint32m1_t offset0, offset1, offset2, offset3; - vuint32m4_t offsets; + size_t curr_nblks = nblocks_to_nblocks_per_m1(max_blocks_m1, nblocks); + vuint32m1_t data_blks; - /* Offset_i = Offset_{i-1} xor L_{ntz(i)} */ - /* Sum_i = Sum_{i-1} xor ENCIPHER(K, A_i xor Offset_i) */ - l = ocb_get_l(c, ++n); - l_ntzi = __riscv_vle32_v_u32m1(l, vl); - iv = __riscv_vxor_vv_u32m1(iv, l_ntzi, vl); - offset0 = iv; + vl_m1 = curr_nblks * 4; - l = ocb_get_l(c, ++n); - l_ntzi = __riscv_vle32_v_u32m1(l, vl); - iv = __riscv_vxor_vv_u32m1(iv, l_ntzi, vl); - offset1 = iv; + data_blks = unaligned_load_u32m1(inbuf, vl_m1); - l = ocb_get_l(c, ++n); - l_ntzi = __riscv_vle32_v_u32m1(l, vl); - iv = __riscv_vxor_vv_u32m1(iv, l_ntzi, vl); - offset2 = iv; + if (encrypt > 0) + { + /* Checksum_i = Checksum_{i-1} xor P_i */ + ctr = __riscv_vxor_vv_u32m1_tu(ctr, ctr, data_blks, vl_m1); + } - l = ocb_get_l(c, ++n); - l_ntzi = __riscv_vle32_v_u32m1(l, vl); - iv = __riscv_vxor_vv_u32m1(iv, l_ntzi, vl); - offset3 = iv; + /* Offset_i = Offset_{i-1} xor L_{ntz(i)} */ + offs_m1 = ocb_offsets_m1(c, &n, &iv, offs_m1, curr_nblks, vl_m1); - offsets = merge_4x_u32m1_to_u32m4(offset0, offset1, offset2, offset3); + /* P_i = Offset_i xor CIPHER(K, C_i xor Offset_i) */ + data_blks = __riscv_vxor_vv_u32m1(offs_m1, data_blks, vl_m1); - data4blks = __riscv_vxor_vv_u32m4(offsets, data4blks, vl * 4); + if (encrypt) + AES_CRYPT(e, m1, GET_PRELOADED_KEY, rk, rounds, data_blks, vl_m1); + else + AES_CRYPT(d, m1, GET_PRELOADED_KEY, rk, rounds, data_blks, vl_m1); - AES_CRYPT(e, m4, rounds, data4blks, vl * 4); + if (!auth) + { + data_blks = __riscv_vxor_vv_u32m1(offs_m1, data_blks, vl_m1); - ctr4blks = __riscv_vxor_vv_u32m4(ctr4blks, data4blks, vl * 4); + unaligned_store_u32m1(outbuf, data_blks, vl_m1); + outbuf += curr_nblks * BLOCKSIZE; + } - abuf += 4 * BLOCKSIZE; + if (encrypt <= 0) + { + /* Checksum_i = Checksum_{i-1} xor P_i */ + ctr = __riscv_vxor_vv_u32m1_tu(ctr, ctr, data_blks, vl_m1); } - /* Checksum_i = Checksum_{i-1} xor P_i */ - { - vuint32m1x4_t ctr0123 = split_u32m4_to_4x_u32m1(ctr4blks); - ctr = __riscv_vxor_vv_u32m1(__riscv_vget_v_u32m1x4_u32m1(ctr0123, 0), - __riscv_vget_v_u32m1x4_u32m1(ctr0123, 1), vl); - ctr = __riscv_vxor_vv_u32m1(ctr, __riscv_vget_v_u32m1x4_u32m1(ctr0123, 2), vl); - ctr = __riscv_vxor_vv_u32m1(ctr, __riscv_vget_v_u32m1x4_u32m1(ctr0123, 3), vl); - } + inbuf += curr_nblks * BLOCKSIZE; + nblocks -= curr_nblks; } - for (; nblocks; nblocks--) + for (h = max_blocks_m1; h > 1; h /= 2) { - const void *l; - vuint32m1_t l_ntzi; - vuint32m1_t data; - - data = unaligned_load_u32m1(abuf, vl); - - /* Offset_i = Offset_{i-1} xor L_{ntz(i)} */ - /* Sum_i = Sum_{i-1} xor ENCIPHER(K, A_i xor Offset_i) */ - l = ocb_get_l(c, ++n); - l_ntzi = __riscv_vle32_v_u32m1(l, vl); - iv = __riscv_vxor_vv_u32m1(iv, l_ntzi, vl); - - data = __riscv_vxor_vv_u32m1(data, iv, vl); - - AES_CRYPT(e, m1, rounds, data, vl); - - ctr = __riscv_vxor_vv_u32m1(ctr, data, vl); - - abuf += BLOCKSIZE; + vuint32m1_t ctr_hi = __riscv_vslidedown_vx_u32m1(ctr, (h / 2) * blk_vl32, + h * blk_vl32); + ctr = __riscv_vxor_vv_u32m1(ctr, ctr_hi, (h / 2) * blk_vl32); } - c->u_mode.ocb.aad_nblocks = n; - - __riscv_vse32_v_u32m1((void *)c->u_mode.ocb.aad_offset, iv, vl); - __riscv_vse32_v_u32m1((void *)c->u_mode.ocb.aad_sum, ctr, vl); + if (auth) + { + c->u_mode.ocb.aad_nblocks = n; + __riscv_vse32_v_u32m1((void *)c->u_mode.ocb.aad_offset, iv, blk_vl32); + __riscv_vse32_v_u32m1((void *)c->u_mode.ocb.aad_sum, ctr, blk_vl32); + } + else + { + c->u_mode.ocb.data_nblocks = n; + __riscv_vse32_v_u32m1((void *)c->u_iv.iv, iv, blk_vl32); + __riscv_vse32_v_u32m1((void *)c->u_ctr.ctr, ctr, blk_vl32); + } clear_vec_regs(); return 0; } -static const u64 xts_gfmul_const[2] = { 0x87, 0x01 }; -static const u64 xts_swap64_const[2] = { 1, 0 }; +size_t ASM_FUNC_ATTR_NOINLINE FUNC_ATTR_OPT_O2 +_gcry_aes_riscv_zvkned_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg, + const void *inbuf_arg, size_t nblocks, + int encrypt) +{ + if (encrypt) + return aes_riscv_ocb_crypt(c, outbuf_arg, inbuf_arg, nblocks, 1); + else + return aes_riscv_ocb_crypt(c, outbuf_arg, inbuf_arg, nblocks, 0); +} -static ASM_FUNC_ATTR_INLINE vuint32m1_t -xts_gfmul_byA (vuint32m1_t vec_in, vuint64m1_t xts_gfmul, - vuint64m1_t xts_swap64, size_t vl) +size_t ASM_FUNC_ATTR_NOINLINE FUNC_ATTR_OPT_O2 +_gcry_aes_riscv_zvkned_ocb_auth (gcry_cipher_hd_t c, const void *abuf_arg, + size_t nblocks) { - vuint64m1_t in_u64 = cast_u32m1_u64m1(vec_in); - vuint64m1_t tmp1; + return aes_riscv_ocb_crypt(c, NULL, abuf_arg, nblocks, -1); +} - tmp1 = - __riscv_vrgather_vv_u64m1(cast_u32m1_u64m1(vec_in), xts_swap64, vl / 2); - tmp1 = cast_i64m1_u64m1( - __riscv_vsra_vx_i64m1(cast_u64m1_i64m1(tmp1), 63, vl / 2)); - in_u64 = __riscv_vadd_vv_u64m1(in_u64, in_u64, vl / 2); - tmp1 = __riscv_vand_vv_u64m1(tmp1, xts_gfmul, vl / 2); +/* 0x87 widens the S-bit overflow by seven bits, which must fit the lane. */ +#define XTS_GFMUL_POW_MAX_SHIFT (64 - 7) + +/* Multiply each 128-bit lane pair by alpha^S. */ +#define DEFINE_XTS_GFMUL_POW(MX, BOOL) \ + static ASM_FUNC_ATTR_INLINE vuint64##MX##_t \ + xts_gfmul_pow1_##MX (vuint64##MX##_t vec_in, vbool##BOOL##_t even, size_t s, \ + size_t vl) \ + { \ + vuint64##MX##_t hi = __riscv_vsrl_vx_u64##MX(vec_in, 64 - s, vl); \ + vuint64##MX##_t lo = __riscv_vsll_vx_u64##MX(vec_in, s, vl); \ + vuint64##MX##_t sw = __riscv_vslide1up_vx_u64##MX(hi, 0, vl); \ + vuint64##MX##_t acc; \ + \ + sw = __riscv_vslidedown_vx_u64##MX##_mu(even, sw, hi, 1, vl); \ + acc = __riscv_vsll_vx_u64##MX(sw, 5, vl); \ + acc = __riscv_vxor_vv_u64##MX(acc, sw, vl); \ + acc = __riscv_vsll_vx_u64##MX(acc, 1, vl); \ + acc = __riscv_vxor_vv_u64##MX(acc, sw, vl); \ + acc = __riscv_vsll_vx_u64##MX(acc, 1, vl); \ + sw = __riscv_vxor_vv_u64##MX##_mu(even, sw, sw, acc, vl); \ + return __riscv_vxor_vv_u64##MX(lo, sw, vl); \ + } \ + \ + static ASM_FUNC_ATTR_INLINE vuint32##MX##_t \ + xts_gfmul_pow_##MX (vuint32##MX##_t vec_in, vbool##BOOL##_t even, size_t s, \ + size_t vl_u32) \ + { \ + vuint64##MX##_t vec = cast_u32##MX##_u64##MX(vec_in); \ + size_t vl = vl_u32 / 2; \ + \ + while (UNLIKELY(s > XTS_GFMUL_POW_MAX_SHIFT)) \ + { \ + vec = xts_gfmul_pow1_##MX(vec, even, XTS_GFMUL_POW_MAX_SHIFT, vl); \ + s -= XTS_GFMUL_POW_MAX_SHIFT; \ + } \ + return cast_u64##MX##_u32##MX(xts_gfmul_pow1_##MX(vec, even, s, vl)); \ + } - return cast_u64m1_u32m1(__riscv_vxor_vv_u64m1(in_u64, tmp1, vl / 2)); -} +DEFINE_XTS_GFMUL_POW(m1, 64) +DEFINE_XTS_GFMUL_POW(m2, 32) +DEFINE_XTS_GFMUL_POW(m4, 16) -static ASM_FUNC_ATTR_NOINLINE FUNC_ATTR_OPT_O2 void -aes_riscv_xts_enc (void *context, unsigned char *tweak_arg, void *outbuf_arg, - const void *inbuf_arg, size_t nblocks) +static ASM_FUNC_ATTR_INLINE FUNC_ATTR_OPT_O2 void +aes_riscv_xts_crypt (void *context, unsigned char *tweak_arg, void *outbuf_arg, + const void *inbuf_arg, size_t nblocks, int encrypt) { RIJNDAEL_context *ctx = context; unsigned char *outbuf = outbuf_arg; const unsigned char *inbuf = inbuf_arg; - const u32 *rk = ctx->keyschenc32[0]; + const u32 *rk = encrypt ? ctx->keyschenc32[0] : ctx->keyschdec32[0]; int rounds = ctx->rounds; - size_t vl = 4; + size_t blk_vl32 = BLOCKSIZE / 4; + size_t max_blocks_m1 = __riscv_vsetvlmax_e32m1() / 4; + size_t max_blocks_m2 = max_blocks_m1 * 2; + size_t max_blocks_m4 = max_blocks_m1 * 4; + size_t vl64_m1 = max_blocks_m1 * 2; + size_t vl_m1 = max_blocks_m1 * 4; + vbool64_t even_m1; vuint32m1_t tweak; - vuint64m1_t xts_gfmul = __riscv_vle64_v_u64m1(xts_gfmul_const, vl / 2); - vuint64m1_t xts_swap64 = __riscv_vle64_v_u64m1(xts_swap64_const, vl / 2); + size_t k; ROUND_KEY_VARIABLES; + ROUND_KEY_VARIABLES_RK10_13; - PRELOAD_ROUND_KEYS (rk, rounds, vl); - - /* Preload tweak */ - tweak = __riscv_vle32_v_u32m1((void *)tweak_arg, vl); - - memory_barrier_with_vec(xts_gfmul); - memory_barrier_with_vec(xts_swap64); - - for (; nblocks >= 4; nblocks -= 4) + if (!encrypt && UNLIKELY(!ctx->decryption_prepared)) { - vuint32m4_t data4blks = unaligned_load_u32m4(inbuf, vl * 4); - vuint32m1_t tweak0, tweak1, tweak2, tweak3; - vuint32m4_t tweaks; - - tweak0 = tweak; - tweak = xts_gfmul_byA(tweak, xts_gfmul, xts_swap64, vl); - tweak1 = tweak; - tweak = xts_gfmul_byA(tweak, xts_gfmul, xts_swap64, vl); - tweak2 = tweak; - tweak = xts_gfmul_byA(tweak, xts_gfmul, xts_swap64, vl); - tweak3 = tweak; - tweak = xts_gfmul_byA(tweak, xts_gfmul, xts_swap64, vl); - - tweaks = merge_4x_u32m1_to_u32m4(tweak0, tweak1, tweak2, tweak3); + do_prepare_decryption(ctx); + ctx->decryption_prepared = 1; + } - data4blks = __riscv_vxor_vv_u32m4(tweaks, data4blks, vl * 4); + PRELOAD_ROUND_KEYS (rk, rounds); - AES_CRYPT(e, m4, rounds, data4blks, vl * 4); + /* Prepare tweak for full m1. */ + tweak = __riscv_vle32_v_u32m1((void *)tweak_arg, blk_vl32); - data4blks = __riscv_vxor_vv_u32m4(tweaks, data4blks, vl * 4); + even_m1 = __riscv_vmseq_vx_u64m1_b64( + __riscv_vand_vx_u64m1(__riscv_vid_v_u64m1(vl64_m1), 1, vl64_m1), 0, + vl64_m1); - unaligned_store_u32m4(outbuf, data4blks, vl * 4); + /* Double the tweak run until m1 is full. */ + for (k = 1; k < max_blocks_m1; k *= 2) + { + vuint32m1_t next = xts_gfmul_pow_m1(tweak, even_m1, k, k * blk_vl32); - inbuf += 4 * BLOCKSIZE; - outbuf += 4 * BLOCKSIZE; + tweak = __riscv_vslideup_vx_u32m1(tweak, next, k * blk_vl32, + 2 * k * blk_vl32); } - for (; nblocks; nblocks--) + if (nblocks >= max_blocks_m2) { - vuint32m1_t data = unaligned_load_u32m1(inbuf, vl); - vuint32m1_t tweak0 = tweak; + size_t vl64_m2 = max_blocks_m2 * 2; + size_t vl_m2 = max_blocks_m2 * 4; + vbool32_t even_m2 = __riscv_vmseq_vx_u64m2_b32( + __riscv_vand_vx_u64m2(__riscv_vid_v_u64m2(vl64_m2), 1, vl64_m2), 0, + vl64_m2); + vuint32m2_t tweaks_m2; + + /* Continue doubling from m1 to m2. */ + tweaks_m2 = __riscv_vset_v_u32m1_u32m2(__riscv_vundefined_u32m2(), 0, + tweak); + tweaks_m2 = __riscv_vset_v_u32m1_u32m2(tweaks_m2, 1, + xts_gfmul_pow_m1(tweak, even_m1, max_blocks_m1, vl_m1)); + + if (nblocks >= max_blocks_m4) + { + size_t vl64_m4 = max_blocks_m4 * 2; + size_t vl_m4 = max_blocks_m4 * 4; + vbool16_t even_m4 = __riscv_vmseq_vx_u64m4_b16( + __riscv_vand_vx_u64m4(__riscv_vid_v_u64m4(vl64_m4), 1, vl64_m4), + 0, vl64_m4); + vuint32m4_t tweaks_m4; + + /* Continue doubling from m2 to m4. */ + tweaks_m4 = __riscv_vset_v_u32m2_u32m4( + __riscv_vundefined_u32m4(), 0, tweaks_m2); + tweaks_m4 = __riscv_vset_v_u32m2_u32m4(tweaks_m4, 1, + xts_gfmul_pow_m2(tweaks_m2, even_m2, max_blocks_m2, vl_m2)); + + do + { + vuint32m4_t data_blks = unaligned_load_u32m4(inbuf, vl_m4); - data = __riscv_vxor_vv_u32m1(data, tweak0, vl); - tweak = xts_gfmul_byA(tweak, xts_gfmul, xts_swap64, vl); + data_blks = __riscv_vxor_vv_u32m4(tweaks_m4, data_blks, vl_m4); - AES_CRYPT(e, m1, rounds, data, vl); + if (encrypt) + AES_CRYPT(e, m4, LOAD_KEY, rk, rounds, data_blks, vl_m4); + else + AES_CRYPT(d, m4, LOAD_KEY, rk, rounds, data_blks, vl_m4); - data = __riscv_vxor_vv_u32m1(data, tweak0, vl); - unaligned_store_u32m1(outbuf, data, vl); + data_blks = __riscv_vxor_vv_u32m4(tweaks_m4, data_blks, vl_m4); - inbuf += BLOCKSIZE; - outbuf += BLOCKSIZE; - } + unaligned_store_u32m4(outbuf, data_blks, vl_m4); - __riscv_vse32_v_u32m1((void *)tweak_arg, tweak, vl); + tweaks_m4 = xts_gfmul_pow_m4(tweaks_m4, even_m4, max_blocks_m4, + vl_m4); - clear_vec_regs(); -} + inbuf += max_blocks_m4 * BLOCKSIZE; + outbuf += max_blocks_m4 * BLOCKSIZE; + nblocks -= max_blocks_m4; + } + while (nblocks >= max_blocks_m4); -static ASM_FUNC_ATTR_NOINLINE FUNC_ATTR_OPT_O2 void -aes_riscv_xts_dec (void *context, unsigned char *tweak_arg, void *outbuf_arg, - const void *inbuf_arg, size_t nblocks) -{ - RIJNDAEL_context *ctx = context; - unsigned char *outbuf = outbuf_arg; - const unsigned char *inbuf = inbuf_arg; - const u32 *rk = ctx->keyschdec32[0]; - int rounds = ctx->rounds; - size_t vl = 4; - vuint32m1_t tweak; - vuint64m1_t xts_gfmul = __riscv_vle64_v_u64m1(xts_gfmul_const, vl / 2); - vuint64m1_t xts_swap64 = __riscv_vle64_v_u64m1(xts_swap64_const, vl / 2); - ROUND_KEY_VARIABLES; + tweaks_m2 = __riscv_vget_v_u32m4_u32m2(tweaks_m4, 0); + } - if (!ctx->decryption_prepared) - { - do_prepare_decryption(ctx); - ctx->decryption_prepared = 1; - } + if (nblocks) + PRELOAD_ROUND_KEYS_RK10_13 (rk, rounds); - PRELOAD_ROUND_KEYS (rk, rounds, vl); + while (nblocks && nblocks >= max_blocks_m2) + { + vuint32m2_t data_blks = unaligned_load_u32m2(inbuf, vl_m2); - /* Preload tweak */ - tweak = __riscv_vle32_v_u32m1((void *)tweak_arg, vl); + data_blks = __riscv_vxor_vv_u32m2(tweaks_m2, data_blks, vl_m2); - memory_barrier_with_vec(xts_gfmul); - memory_barrier_with_vec(xts_swap64); + if (encrypt) + AES_CRYPT(e, m2, GET_PRELOADED_KEY, rk, rounds, data_blks, vl_m2); + else + AES_CRYPT(d, m2, GET_PRELOADED_KEY, rk, rounds, data_blks, vl_m2); - for (; nblocks >= 4; nblocks -= 4) - { - vuint32m4_t data4blks = unaligned_load_u32m4(inbuf, vl * 4); - vuint32m1_t tweak0, tweak1, tweak2, tweak3; - vuint32m4_t tweaks; + data_blks = __riscv_vxor_vv_u32m2(tweaks_m2, data_blks, vl_m2); - tweak0 = tweak; - tweak = xts_gfmul_byA(tweak, xts_gfmul, xts_swap64, vl); - tweak1 = tweak; - tweak = xts_gfmul_byA(tweak, xts_gfmul, xts_swap64, vl); - tweak2 = tweak; - tweak = xts_gfmul_byA(tweak, xts_gfmul, xts_swap64, vl); - tweak3 = tweak; - tweak = xts_gfmul_byA(tweak, xts_gfmul, xts_swap64, vl); + unaligned_store_u32m2(outbuf, data_blks, vl_m2); - tweaks = merge_4x_u32m1_to_u32m4(tweak0, tweak1, tweak2, tweak3); + tweaks_m2 = xts_gfmul_pow_m2(tweaks_m2, even_m2, max_blocks_m2, + vl_m2); - data4blks = __riscv_vxor_vv_u32m4(tweaks, data4blks, vl * 4); + inbuf += max_blocks_m2 * BLOCKSIZE; + outbuf += max_blocks_m2 * BLOCKSIZE; + nblocks -= max_blocks_m2; + } - AES_CRYPT(d, m4, rounds, data4blks, vl * 4); + tweak = __riscv_vlmul_trunc_v_u32m2_u32m1(tweaks_m2); + } + else if (nblocks) + PRELOAD_ROUND_KEYS_RK10_13 (rk, rounds); - data4blks = __riscv_vxor_vv_u32m4(tweaks, data4blks, vl * 4); + while (nblocks) + { + size_t curr_nblks = nblocks_to_nblocks_per_m1(max_blocks_m1, nblocks); + vuint32m1_t data_blks; - unaligned_store_u32m4(outbuf, data4blks, vl * 4); + vl_m1 = curr_nblks * 4; - inbuf += 4 * BLOCKSIZE; - outbuf += 4 * BLOCKSIZE; - } + data_blks = unaligned_load_u32m1(inbuf, vl_m1); - for (; nblocks; nblocks--) - { - vuint32m1_t data = unaligned_load_u32m1(inbuf, vl); - vuint32m1_t tweak0 = tweak; + data_blks = __riscv_vxor_vv_u32m1(tweak, data_blks, vl_m1); + + if (encrypt) + AES_CRYPT(e, m1, GET_PRELOADED_KEY, rk, rounds, data_blks, vl_m1); + else + AES_CRYPT(d, m1, GET_PRELOADED_KEY, rk, rounds, data_blks, vl_m1); - data = __riscv_vxor_vv_u32m1(data, tweak0, vl); - tweak = xts_gfmul_byA(tweak, xts_gfmul, xts_swap64, vl); + data_blks = __riscv_vxor_vv_u32m1(tweak, data_blks, vl_m1); - AES_CRYPT(d, m1, rounds, data, vl); + unaligned_store_u32m1(outbuf, data_blks, vl_m1); - data = __riscv_vxor_vv_u32m1(data, tweak0, vl); - unaligned_store_u32m1(outbuf, data, vl); + tweak = xts_gfmul_pow_m1(tweak, even_m1, curr_nblks, vl_m1); - inbuf += BLOCKSIZE; - outbuf += BLOCKSIZE; + inbuf += curr_nblks * BLOCKSIZE; + outbuf += curr_nblks * BLOCKSIZE; + nblocks -= curr_nblks; } - __riscv_vse32_v_u32m1((void *)tweak_arg, tweak, vl); + __riscv_vse32_v_u32m1((void *)tweak_arg, tweak, blk_vl32); clear_vec_regs(); } @@ -1628,9 +1824,9 @@ _gcry_aes_riscv_zvkned_xts_crypt (void *context, unsigned char *tweak_arg, size_t nblocks, int encrypt) { if (encrypt) - aes_riscv_xts_enc(context, tweak_arg, outbuf_arg, inbuf_arg, nblocks); + aes_riscv_xts_crypt(context, tweak_arg, outbuf_arg, inbuf_arg, nblocks, 1); else - aes_riscv_xts_dec(context, tweak_arg, outbuf_arg, inbuf_arg, nblocks); + aes_riscv_xts_crypt(context, tweak_arg, outbuf_arg, inbuf_arg, nblocks, 0); } #endif /* HAVE_COMPATIBLE_CC_RISCV_VECTOR_INTRINSICS */ diff --git a/cipher/rijndael.c b/cipher/rijndael.c index 647334c9..082b20ab 100644 --- a/cipher/rijndael.c +++ b/cipher/rijndael.c @@ -843,6 +843,7 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen, else if ((hwfeatures & HWF_RISCV_IMAFDC) && (hwfeatures & HWF_RISCV_B) /* Mandatory in RVA23U64 */ && (hwfeatures & HWF_RISCV_V) /* Mandatory in RVA23U64 */ + && (hwfeatures & HWF_RISCV_ZVKB) /* Mandatory in RVA23U64 (Zvbb) */ && (hwfeatures & HWF_RISCV_ZVKNED) /* Optional in RVA23U64 */ && _gcry_aes_riscv_zvkned_setup_acceleration(ctx)) { -- 2.53.0 From hl3 at proton.me Thu Oct 8 11:22:26 2026 From: hl3 at proton.me (HMBSbige) Date: Thu, 08 Oct 2026 09:22:26 +0000 Subject: x86: Incorrect CPUID leaf in Zen 5 detection Message-ID: Hi, The AVX512_BMM check in src/hwf-x86.c has two errors: 1. The leaf should be 0x80000021, not 0x21. 2. The availability check should use the maximum extended leaf    returned by CPUID(0x80000000).EAX, not the maximum basic leaf    from CPUID(0).EAX. The file currently never queries 0x80000000. The minimal fix would be:     unsigned int max_ext_level, amd_feat;     get_cpuid(0x80000000, &max_ext_level, NULL, NULL, NULL);     if (max_ext_level >= 0x80000021)       {         get_cpuid(0x80000021, &amd_feat, NULL, NULL, NULL);         if (amd_feat & (1 << 23))           has_avx512bmm = 1;       } Alternatively, I suggest identifying Zen 5 directly by vendor, family and model. Family 0x1A also includes Zen 6, so the absence of AVX512_BMM is not sufficient to identify Zen 5. Following Linux's classification in arch/x86/kernel/cpu/amd.c, the Zen 5 condition would be:     if (is_amd_cpu && family == 0x1a         && (model <= 0x2f             || (model >= 0x40 && model <= 0x4f)             || (model >= 0x60 && model <= 0x7f)             || (model >= 0xd0 && model <= 0xd7)))       {         cpu_details->int_vector_latency = 2;         cpu_details->prefer_gpr_over_scalar_int_vector = 1;       } This reuses the already decoded family/model and eliminates the AVX512_BMM probe. Unknown models would retain the default tuning until evaluated. Best regards, Bruce From jussi.kivilinna at iki.fi Thu Oct 8 21:13:55 2026 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Thu, 8 Oct 2026 22:13:55 +0300 Subject: x86: Incorrect CPUID leaf in Zen 5 detection In-Reply-To: References: Message-ID: Hello, On 08/10/2026 12:22, HMBSbige via Gcrypt-devel wrote: > Hi, > The AVX512_BMM check in src/hwf-x86.c has two errors: > > 1. The leaf should be 0x80000021, not 0x21. > 2. The availability check should use the maximum extended leaf >    returned by CPUID(0x80000000).EAX, not the maximum basic leaf >    from CPUID(0).EAX. The file currently never queries 0x80000000. Thanks for reporting this. > > The minimal fix would be: > >     unsigned int max_ext_level, amd_feat; > >     get_cpuid(0x80000000, &max_ext_level, NULL, NULL, NULL); >     if (max_ext_level >= 0x80000021) >       { >         get_cpuid(0x80000021, &amd_feat, NULL, NULL, NULL); >         if (amd_feat & (1 << 23)) >           has_avx512bmm = 1; >       } > > Alternatively, I suggest identifying Zen 5 directly by vendor, > family and model. Family 0x1A also includes Zen 6, so the absence > of AVX512_BMM is not sufficient to identify Zen 5. > > Following Linux's classification in arch/x86/kernel/cpu/amd.c, > the Zen 5 condition would be: > >     if (is_amd_cpu && family == 0x1a >         && (model <= 0x2f >             || (model >= 0x40 && model <= 0x4f) >             || (model >= 0x60 && model <= 0x7f) >             || (model >= 0xd0 && model <= 0xd7))) >       { >         cpu_details->int_vector_latency = 2; >         cpu_details->prefer_gpr_over_scalar_int_vector = 1; >       } > > This reuses the already decoded family/model and eliminates the > AVX512_BMM probe. Unknown models would retain the default tuning > until evaluated. I think this approach would make more sense. I'll switch to using it. -Jussi From jussi.kivilinna at iki.fi Sat Oct 10 10:40:23 2026 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sat, 10 Oct 2026 11:40:23 +0300 Subject: [PATCH] hwf-x86: detect Zen5 by family and model Message-ID: <20261010084023.4734-1-jussi.kivilinna@iki.fi> * src/hwf-x86.c (detect_x86_gnuc): Remove AVX512_BMM check; enable Zen5 tuning based on AMD family and model. -- AVX512_BMM check used basic CPUID leaf 0x21 instead of extended leaf 0x80000021, so Zen6 would also get Zen5 tuning. Identify Zen5 with same family and model ranges as Linux. Reported-by: HMBSbige Signed-off-by: Jussi Kivilinna --- src/hwf-x86.c | 20 +++++--------------- 1 file changed, 5 insertions(+), 15 deletions(-) diff --git a/src/hwf-x86.c b/src/hwf-x86.c index df3ba4f9..8035de6d 100644 --- a/src/hwf-x86.c +++ b/src/hwf-x86.c @@ -223,7 +223,6 @@ detect_x86_gnuc ( unsigned int fms, family, model; unsigned int result = 0; unsigned int is_amd_cpu = 0; - unsigned int has_avx512bmm = 0; unsigned int has_sse3 = 0; (void)os_supports_avx_avx2_registers; @@ -286,7 +285,6 @@ detect_x86_gnuc ( { /* This is an AMD CPU. */ is_amd_cpu = 1; - (void)is_amd_cpu; } /* Detect Intel features, that might also be supported by other @@ -436,18 +434,6 @@ detect_x86_gnuc ( result |= HWF_INTEL_SM4; } - /* Check additional feature flags. */ - if (max_cpuid_level >= 0x21 && has_sse3) - { - unsigned int amd_feat; - - get_cpuid(0x21, &amd_feat, NULL, NULL, NULL); - if (amd_feat & (1 << 23)) - { - has_avx512bmm = 1; - } - } - if ((result & HWF_INTEL_CPU) && family == 6) { /* These Intel Core processor models have SHLD/SHRD instruction that @@ -476,7 +462,11 @@ detect_x86_gnuc ( } } - if (is_amd_cpu && (family == 0x1a) && !has_avx512bmm) + if (is_amd_cpu && family == 0x1a + && (model <= 0x2f + || (model >= 0x40 && model <= 0x4f) + || (model >= 0x60 && model <= 0x7f) + || (model >= 0xd0 && model <= 0xd7))) { /* Zen5 has integer vector instruction latency of 2 and powerful * GPR integer performance. */ -- 2.53.0 From jussi.kivilinna at iki.fi Sat Oct 10 10:52:21 2026 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sat, 10 Oct 2026 11:52:21 +0300 Subject: [PATCH] rijndael-vaes: simplify OCB L-array setup Message-ID: <20261010085221.38390-1-jussi.kivilinna@iki.fi> * cipher/rijndael-vaes-avx2-amd64.S (_gcry_vaes_avx2_ocb_crypt_amd64): Build L-array table from broadcast L values with XOR and blend; avoid loading first round key twice. * cipher/rijndael-vaes-avx512-amd64.S (_gcry_vaes_avx512_ocb_aligned_crypt_amd64): Derive L-array vectors from earlier vectors by XOR with broadcast L pairs; avoid loading first round key twice. -- Previously each L combination was computed separately in 128-bit register and vectors were assembled with lane inserts. Now L values are loaded as broadcasts, first vector is assembled with blends and rest are derived by XORing earlier vector with broadcast L pair. AVX2 setup goes from 29 to 21 instructions and AVX512 setup from 54 to 21. Signed-off-by: Jussi Kivilinna --- cipher/rijndael-vaes-avx2-amd64.S | 46 ++++++++---------- cipher/rijndael-vaes-avx512-amd64.S | 75 ++++++++--------------------- 2 files changed, 40 insertions(+), 81 deletions(-) diff --git a/cipher/rijndael-vaes-avx2-amd64.S b/cipher/rijndael-vaes-avx2-amd64.S index 65fd79fe..f19d9557 100644 --- a/cipher/rijndael-vaes-avx2-amd64.S +++ b/cipher/rijndael-vaes-avx2-amd64.S @@ -1418,10 +1418,10 @@ _gcry_vaes_avx2_ocb_crypt_amd64: vmovdqu (%r14), %xmm15; /* Load offset. */ movq L_ARRAY_PTR_L, %r14; /* L-array ptr. */ vmovdqa (0 * 16)(%rdi), %xmm0; /* first key */ + vpxor %xmm0, %xmm15, %xmm15; /* offset ^ first key */ vpxor %xmm14, %xmm14, %xmm14; vpxor %xmm13, %xmm13, %xmm13; vpxor (%rdi, %rax, 4), %xmm0, %xmm0; /* first key ^ last key */ - vpxor (0 * 16)(%rdi), %xmm15, %xmm15; /* offset ^ first key */ vmovdqa %xmm0, (14 * 16)(%rsp); vmovdqa %xmm0, (15 * 16)(%rsp); @@ -2005,37 +2005,29 @@ _gcry_vaes_avx2_ocb_crypt_amd64: * - block15 = ntz{0} = offset ^ L[3] * - block16 = ntz{x} = offset ^ L[3] ^ L[ntz{x}] */ - vmovdqu (0 * 16)(%r14), %xmm0; - vmovdqu (1 * 16)(%r14), %xmm1; - vmovdqu (2 * 16)(%r14), %xmm2; - vmovdqu (3 * 16)(%r14), %xmm3; + vbroadcasti128 (0 * 16)(%r14), %ymm0; /* L[0] */ + vbroadcasti128 (1 * 16)(%r14), %ymm1; /* L[1] */ + vbroadcasti128 (2 * 16)(%r14), %ymm2; /* L[2] */ + vbroadcasti128 (3 * 16)(%r14), %ymm3; /* L[3] */ vpxor %ymm13, %ymm14, %ymm14; vmovdqa %ymm14, (20 * 16)(%rsp); - vpxor %xmm0, %xmm1, %xmm4; /* L[0] ^ L[1] */ - vpxor %xmm0, %xmm2, %xmm5; /* L[0] ^ L[2] */ - vpxor %xmm0, %xmm3, %xmm6; /* L[0] ^ L[3] */ - vpxor %xmm1, %xmm2, %xmm7; /* L[1] ^ L[2] */ - vpxor %xmm1, %xmm3, %xmm8; /* L[1] ^ L[3] */ - vpxor %xmm2, %xmm3, %xmm9; /* L[2] ^ L[3] */ - vpxor %xmm4, %xmm2, %xmm10; /* L[0] ^ L[1] ^ L[2] */ - vpxor %xmm5, %xmm3, %xmm11; /* L[0] ^ L[2] ^ L[3] */ - vpxor %xmm7, %xmm3, %xmm12; /* L[1] ^ L[2] ^ L[3] */ - vpxor %xmm0, %xmm8, %xmm13; /* L[0] ^ L[1] ^ L[3] */ - vpxor %xmm4, %xmm9, %xmm14; /* L[0] ^ L[1] ^ L[2] ^ L[3] */ - vinserti128 $1, %xmm4, %ymm0, %ymm0; - vinserti128 $1, %xmm7, %ymm1, %ymm1; - vinserti128 $1, %xmm5, %ymm10, %ymm10; - vinserti128 $1, %xmm9, %ymm2, %ymm2; - vinserti128 $1, %xmm14, %ymm11, %ymm11; - vinserti128 $1, %xmm8, %ymm12, %ymm12; - vinserti128 $1, %xmm6, %ymm13, %ymm13; + vpxor %ymm0, %ymm1, %ymm4; /* L[0] ^ L[1] */ + vpxor %ymm1, %ymm2, %ymm5; /* L[1] ^ L[2] */ + vpxor %ymm2, %ymm3, %ymm6; /* L[2] ^ L[3] */ + vpblendd $0xf0, %ymm4, %ymm0, %ymm0; /* blocks 1-2 */ + vpblendd $0xf0, %ymm5, %ymm1, %ymm1; /* blocks 3-4 */ + vpblendd $0xf0, %ymm6, %ymm2, %ymm2; /* blocks 7-8 */ + vpxor %ymm5, %ymm0, %ymm7; /* blocks 5-6 */ + vpxor %ymm6, %ymm0, %ymm8; /* blocks 9-10 */ + vpxor %ymm6, %ymm1, %ymm9; /* blocks 11-12 */ + vpxor %ymm6, %ymm7, %ymm10; /* blocks 13-14 */ vmovdqa %ymm0, (0 * 16)(%rsp); vmovdqa %ymm1, (2 * 16)(%rsp); - vmovdqa %ymm10, (4 * 16)(%rsp); + vmovdqa %ymm7, (4 * 16)(%rsp); vmovdqa %ymm2, (6 * 16)(%rsp); - vmovdqa %ymm11, (8 * 16)(%rsp); - vmovdqa %ymm12, (10 * 16)(%rsp); - vmovdqa %ymm13, (12 * 16)(%rsp); + vmovdqa %ymm8, (8 * 16)(%rsp); + vmovdqa %ymm9, (10 * 16)(%rsp); + vmovdqa %ymm10, (12 * 16)(%rsp); /* Aligned: Process 16 blocks per loop. */ .align 8 diff --git a/cipher/rijndael-vaes-avx512-amd64.S b/cipher/rijndael-vaes-avx512-amd64.S index 96cbdd0a..606e345b 100644 --- a/cipher/rijndael-vaes-avx512-amd64.S +++ b/cipher/rijndael-vaes-avx512-amd64.S @@ -1062,8 +1062,8 @@ _gcry_vaes_avx512_ocb_aligned_crypt_amd64: leal (, %r9d, 4), %eax; vmovdqu (%r13), %xmm15; /* Load offset. */ vmovdqa (0 * 16)(%rdi), %xmm0; /* first key */ + vpxor %xmm0, %xmm15, %xmm15; /* offset ^ first key */ vpxor (%rdi, %rax, 4), %xmm0, %xmm0; /* first key ^ last key */ - vpxor (0 * 16)(%rdi), %xmm15, %xmm15; /* offset ^ first key */ vshufi32x4 $0, %zmm0, %zmm0, %zmm30; vpxord %ymm29, %ymm29, %ymm29; @@ -1096,35 +1096,23 @@ _gcry_vaes_avx512_ocb_aligned_crypt_amd64: * - block16 = ntz{4} = offset ^ L[3] ^ L[4] * => zmm23 */ - vmovdqu (0 * 16)(%r14), %xmm0; /* L[0] */ - vmovdqu (1 * 16)(%r14), %xmm1; /* L[1] */ - vmovdqu (2 * 16)(%r14), %xmm2; /* L[2] */ - vmovdqu (3 * 16)(%r14), %xmm3; /* L[3] */ - vmovdqu32 (4 * 16)(%r14), %xmm16; /* L[4] */ - vpxor %xmm0, %xmm1, %xmm4; /* L[0] ^ L[1] */ - vpxor %xmm0, %xmm2, %xmm5; /* L[0] ^ L[2] */ - vpxor %xmm0, %xmm3, %xmm6; /* L[0] ^ L[3] */ - vpxor %xmm1, %xmm2, %xmm7; /* L[1] ^ L[2] */ - vpxor %xmm1, %xmm3, %xmm8; /* L[1] ^ L[3] */ - vpxor %xmm2, %xmm3, %xmm9; /* L[2] ^ L[3] */ - vpxord %xmm16, %xmm3, %xmm17; /* L[3] ^ L[4] */ - vpxor %xmm4, %xmm2, %xmm10; /* L[0] ^ L[1] ^ L[2] */ - vpxor %xmm5, %xmm3, %xmm11; /* L[0] ^ L[2] ^ L[3] */ - vpxor %xmm7, %xmm3, %xmm12; /* L[1] ^ L[2] ^ L[3] */ - vpxor %xmm0, %xmm8, %xmm13; /* L[0] ^ L[1] ^ L[3] */ - vpxor %xmm4, %xmm9, %xmm14; /* L[0] ^ L[1] ^ L[2] ^ L[3] */ - vinserti128 $1, %xmm4, %ymm0, %ymm0; - vinserti128 $1, %xmm7, %ymm1, %ymm1; + vbroadcasti32x4 (0 * 16)(%r14), %zmm0; /* L[0] */ + vbroadcasti32x4 (1 * 16)(%r14), %zmm1; /* L[1] */ + vbroadcasti32x4 (2 * 16)(%r14), %zmm2; /* L[2] */ + vbroadcasti32x4 (3 * 16)(%r14), %zmm3; /* L[3] */ + vbroadcasti32x4 (4 * 16)(%r14), %zmm7; /* L[4] */ + vpxord %zmm0, %zmm1, %zmm4; /* L[0] ^ L[1] */ + vpxord %zmm1, %zmm2, %zmm5; /* L[1] ^ L[2] */ + vpxord %zmm2, %zmm3, %zmm6; /* L[2] ^ L[3] */ + vpxord %zmm3, %zmm7, %zmm7; /* L[3] ^ L[4] */ + vpblendd $0xf0, %ymm4, %ymm0, %ymm0; /* blocks 1-2 */ + vpblendd $0xf0, %ymm5, %ymm1, %ymm1; /* blocks 3-4 */ vinserti32x8 $1, %ymm1, %zmm0, %zmm20; - vinserti128 $1, %xmm5, %ymm10, %ymm10; - vinserti128 $1, %xmm9, %ymm2, %ymm2; - vinserti32x8 $1, %ymm2, %zmm10, %zmm21; - vinserti128 $1, %xmm14, %ymm11, %ymm11; - vinserti128 $1, %xmm8, %ymm12, %ymm12; - vinserti32x8 $1, %ymm12, %zmm11, %zmm22; - vinserti128 $1, %xmm6, %ymm13, %ymm13; - vinserti32x4 $1, %xmm17, %ymm3, %ymm23; - vinserti32x8 $1, %ymm23, %zmm13, %zmm23; + vpxord %zmm5, %zmm20, %zmm21; /* blocks 5-8, lane 3 zero */ + vinserti32x4 $3, %xmm6, %zmm21, %zmm21; /* block 8 */ + vpxord %zmm6, %zmm20, %zmm22; /* blocks 9-12 */ + vpxord %zmm6, %zmm21, %zmm23; /* blocks 13-16, lane 3 zero */ + vinserti32x4 $3, %xmm7, %zmm23, %zmm23; /* block 16 */ /* * - block17 = ntz{0} = offset ^ L[0] ^ L[3] ^ L[4] @@ -1151,31 +1139,10 @@ _gcry_vaes_avx512_ocb_aligned_crypt_amd64: * - block32 = 0 (later filled with ntz{x} = offset ^ L[4] ^ L[ntz{x}]) * => zmm16 */ - vpxord %xmm16, %xmm0, %xmm0; /* L[0] ^ L[4] */ - vpxord %xmm16, %xmm1, %xmm1; /* L[1] ^ L[4] */ - vpxord %xmm16, %xmm2, %xmm2; /* L[2] ^ L[4] */ - vpxord %xmm16, %xmm4, %xmm4; /* L[0] ^ L[1] ^ L[4] */ - vpxord %xmm16, %xmm5, %xmm5; /* L[0] ^ L[2] ^ L[4] */ - vpxord %xmm16, %xmm6, %xmm6; /* L[0] ^ L[3] ^ L[4] */ - vpxord %xmm16, %xmm7, %xmm7; /* L[1] ^ L[2] ^ L[4] */ - vpxord %xmm16, %xmm8, %xmm8; /* L[1] ^ L[3] ^ L[4] */ - vpxord %xmm16, %xmm9, %xmm9; /* L[2] ^ L[3] ^ L[4] */ - vpxord %xmm16, %xmm10, %xmm10; /* L[0] ^ L[1] ^ L[2] ^ L[4] */ - vpxord %xmm16, %xmm11, %xmm11; /* L[0] ^ L[2] ^ L[3] ^ L[4] */ - vpxord %xmm16, %xmm12, %xmm12; /* L[1] ^ L[2] ^ L[3] ^ L[4] */ - vpxord %xmm16, %xmm13, %xmm13; /* L[0] ^ L[1] ^ L[3] ^ L[4] */ - vpxord %xmm16, %xmm14, %xmm14; /* L[0] ^ L[1] ^ L[2] ^ L[3] ^ L[4] */ - vinserti128 $1, %xmm13, %ymm6, %ymm6; - vinserti32x4 $1, %xmm12, %ymm8, %ymm24; - vinserti32x8 $1, %ymm24, %zmm6, %zmm24; - vinserti128 $1, %xmm11, %ymm14, %ymm14; - vinserti32x4 $1, %xmm2, %ymm9, %ymm25; - vinserti32x8 $1, %ymm25, %zmm14, %zmm25; - vinserti128 $1, %xmm10, %ymm5, %ymm5; - vinserti32x4 $1, %xmm1, %ymm7, %ymm26; - vinserti32x8 $1, %ymm26, %zmm5, %zmm26; - vinserti128 $1, %xmm0, %ymm4, %ymm4; - vinserti32x8 $1, %ymm16, %zmm4, %zmm16; + vpxord %zmm7, %zmm20, %zmm24; /* blocks 17-20 */ + vpxord %zmm7, %zmm21, %zmm25; /* blocks 21-24 */ + vpxord %zmm7, %zmm22, %zmm26; /* blocks 25-28 */ + vpxord %zmm7, %zmm23, %zmm16; /* blocks 29-32, lane 3 zero */ /* Aligned: Process 32 blocks per loop. */ .align 16 -- 2.53.0 From jussi.kivilinna at iki.fi Sat Oct 10 10:56:58 2026 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sat, 10 Oct 2026 11:56:58 +0300 Subject: [PATCH] cipher: add dependency tracking to custom compile rules Message-ID: <20261010085658.54572-1-jussi.kivilinna@iki.fi> * cipher/Makefile.am (custom_depflags, custom_depfile_o) (custom_depfile_lo): New. (tiger.o, tiger.lo): Use custom_depflags and custom_depfile_*. (rijndael-aesni.o, rijndael-aesni.lo, rijndael-ssse3-amd64.o) (rijndael-ssse3-amd64.lo, ...): Likewise. -- Custom compile rules for modules with special compiler flags did not generate dependency files, so changes to included headers did not rebuild these modules. Generate dependency files in same way as automake does for its own rules. Signed-off-by: Jussi Kivilinna --- cipher/Makefile.am | 192 +++++++++++++++++++++++++++++++-------------- 1 file changed, 132 insertions(+), 60 deletions(-) diff --git a/cipher/Makefile.am b/cipher/Makefile.am index 486bd99e..89c7a269 100644 --- a/cipher/Makefile.am +++ b/cipher/Makefile.am @@ -183,6 +183,18 @@ gost-s-box$(EXEEXT_FOR_BUILD): gost-s-box.c $(CPPFLAGS_FOR_BUILD) -o $@ $(srcdir)/gost-s-box.c +# Dependency tracking for the custom compile rules below, same as automake +# uses for its own rules. +if am__fastdepCC +custom_depflags = -MT $@ -MD -MP -MF $(DEPDIR)/$*.Tpo +custom_depfile_o = $(AM_V_at)$(am__mv) $(DEPDIR)/$*.Tpo $(DEPDIR)/$*.Po +custom_depfile_lo = $(AM_V_at)$(am__mv) $(DEPDIR)/$*.Tpo $(DEPDIR)/$*.Plo +else +custom_depflags = +custom_depfile_o = @: +custom_depfile_lo = @: +endif + if ENABLE_O_FLAG_MUNGING o_flag_munging = sed -e 's/[[:blank:]]-O\([2-9sgz][2-9sgz]*\)/ -O1 /g' -e 's/[[:blank:]]-Ofast/ -O1 /g' else @@ -192,10 +204,12 @@ endif # We need to lower the optimization for this module. tiger.o: $(srcdir)/tiger.c Makefile - `echo $(COMPILE) -c $< | $(o_flag_munging) ` + `echo $(COMPILE) $(custom_depflags) -c $< | $(o_flag_munging) ` + $(custom_depfile_o) tiger.lo: $(srcdir)/tiger.c Makefile - `echo $(LTCOMPILE) -c $< | $(o_flag_munging) ` + `echo $(LTCOMPILE) $(custom_depflags) -c $< | $(o_flag_munging) ` + $(custom_depfile_lo) # We need to disable instrumentation for these modules as they use cc as @@ -211,52 +225,68 @@ instrumentation_munging = cat endif rijndael-aesni.o: $(srcdir)/rijndael-aesni.c Makefile - `echo $(COMPILE) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) rijndael-aesni.lo: $(srcdir)/rijndael-aesni.c Makefile - `echo $(LTCOMPILE) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) rijndael-ssse3-amd64.o: $(srcdir)/rijndael-ssse3-amd64.c Makefile - `echo $(COMPILE) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) rijndael-ssse3-amd64.lo: $(srcdir)/rijndael-ssse3-amd64.c Makefile - `echo $(LTCOMPILE) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) cipher-gcm-intel-pclmul.o: $(srcdir)/cipher-gcm-intel-pclmul.c Makefile - `echo $(COMPILE) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) cipher-gcm-intel-pclmul.lo: $(srcdir)/cipher-gcm-intel-pclmul.c Makefile - `echo $(LTCOMPILE) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) sha1-intel-shaext.o: $(srcdir)/sha1-intel-shaext.c Makefile - `echo $(COMPILE) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) sha1-intel-shaext.lo: $(srcdir)/sha1-intel-shaext.c Makefile - `echo $(LTCOMPILE) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) sha256-intel-shaext.o: $(srcdir)/sha256-intel-shaext.c Makefile - `echo $(COMPILE) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) sha256-intel-shaext.lo: $(srcdir)/sha256-intel-shaext.c Makefile - `echo $(LTCOMPILE) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) sha256-ssse3-i386.o: $(srcdir)/sha256-ssse3-i386.c Makefile - `echo $(COMPILE) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) sha256-ssse3-i386.lo: $(srcdir)/sha256-ssse3-i386.c Makefile - `echo $(LTCOMPILE) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) sha512-intel-shaext.o: $(srcdir)/sha512-intel-shaext.c Makefile - `echo $(COMPILE) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) sha512-intel-shaext.lo: $(srcdir)/sha512-intel-shaext.c Makefile - `echo $(LTCOMPILE) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) crc-intel-pclmul.o: $(srcdir)/crc-intel-pclmul.c Makefile - `echo $(COMPILE) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) crc-intel-pclmul.lo: $(srcdir)/crc-intel-pclmul.c Makefile - `echo $(LTCOMPILE) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) if ENABLE_PPC_VCRYPTO_EXTRA_CFLAGS ppc_vcrypto_cflags = -O2 -maltivec -mvsx -mcrypto @@ -265,70 +295,92 @@ ppc_vcrypto_cflags = endif rijndael-ppc.o: $(srcdir)/rijndael-ppc.c Makefile - `echo $(COMPILE) $(ppc_vcrypto_cflags) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(ppc_vcrypto_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) rijndael-ppc.lo: $(srcdir)/rijndael-ppc.c Makefile - `echo $(LTCOMPILE) $(ppc_vcrypto_cflags) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(ppc_vcrypto_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) rijndael-ppc9le.o: $(srcdir)/rijndael-ppc9le.c Makefile - `echo $(COMPILE) $(ppc_vcrypto_cflags) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(ppc_vcrypto_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) rijndael-ppc9le.lo: $(srcdir)/rijndael-ppc9le.c Makefile - `echo $(LTCOMPILE) $(ppc_vcrypto_cflags) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(ppc_vcrypto_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) rijndael-p10le.o: $(srcdir)/rijndael-p10le.c Makefile - `echo $(COMPILE) $(ppc_vcrypto_cflags) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(ppc_vcrypto_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) rijndael-p10le.lo: $(srcdir)/rijndael-p10le.c Makefile - `echo $(LTCOMPILE) $(ppc_vcrypto_cflags) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(ppc_vcrypto_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) sha256-ppc.o: $(srcdir)/sha256-ppc.c Makefile - `echo $(COMPILE) $(ppc_vcrypto_cflags) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(ppc_vcrypto_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) sha256-ppc.lo: $(srcdir)/sha256-ppc.c Makefile - `echo $(LTCOMPILE) $(ppc_vcrypto_cflags) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(ppc_vcrypto_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) sha512-ppc.o: $(srcdir)/sha512-ppc.c Makefile - `echo $(COMPILE) $(ppc_vcrypto_cflags) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(ppc_vcrypto_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) sha512-ppc.lo: $(srcdir)/sha512-ppc.c Makefile - `echo $(LTCOMPILE) $(ppc_vcrypto_cflags) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(ppc_vcrypto_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) chacha20-ppc.o: $(srcdir)/chacha20-ppc.c Makefile - `echo $(COMPILE) $(ppc_vcrypto_cflags) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(ppc_vcrypto_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) chacha20-ppc.lo: $(srcdir)/chacha20-ppc.c Makefile - `echo $(LTCOMPILE) $(ppc_vcrypto_cflags) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(ppc_vcrypto_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) crc-ppc.o: $(srcdir)/crc-ppc.c Makefile - `echo $(COMPILE) $(ppc_vcrypto_cflags) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(ppc_vcrypto_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) crc-ppc.lo: $(srcdir)/crc-ppc.c Makefile - `echo $(LTCOMPILE) $(ppc_vcrypto_cflags) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(ppc_vcrypto_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) cipher-gcm-ppc.o: $(srcdir)/cipher-gcm-ppc.c Makefile - `echo $(COMPILE) $(ppc_vcrypto_cflags) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(ppc_vcrypto_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) cipher-gcm-ppc.lo: $(srcdir)/cipher-gcm-ppc.c Makefile - `echo $(LTCOMPILE) $(ppc_vcrypto_cflags) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(ppc_vcrypto_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) camellia-ppc8le.o: $(srcdir)/camellia-ppc8le.c Makefile - `echo $(COMPILE) $(ppc_vcrypto_cflags) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(ppc_vcrypto_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) camellia-ppc8le.lo: $(srcdir)/camellia-ppc8le.c Makefile - `echo $(LTCOMPILE) $(ppc_vcrypto_cflags) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(ppc_vcrypto_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) camellia-ppc9le.o: $(srcdir)/camellia-ppc9le.c Makefile - `echo $(COMPILE) $(ppc_vcrypto_cflags) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(ppc_vcrypto_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) camellia-ppc9le.lo: $(srcdir)/camellia-ppc9le.c Makefile - `echo $(LTCOMPILE) $(ppc_vcrypto_cflags) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(ppc_vcrypto_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) sm4-ppc.o: $(srcdir)/sm4-ppc.c Makefile - `echo $(COMPILE) $(ppc_vcrypto_cflags) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(ppc_vcrypto_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) sm4-ppc.lo: $(srcdir)/sm4-ppc.c Makefile - `echo $(LTCOMPILE) $(ppc_vcrypto_cflags) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(ppc_vcrypto_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) if ENABLE_AARCH64_NEON_INTRINSICS_EXTRA_CFLAGS aarch64_crypto_cflags = -O2 -march=armv8-a+simd+crypto @@ -339,22 +391,28 @@ aarch64_simd_cflags = endif camellia-aarch64-ce.o: $(srcdir)/camellia-aarch64-ce.c Makefile - `echo $(COMPILE) $(aarch64_crypto_cflags) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(aarch64_crypto_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) camellia-aarch64-ce.lo: $(srcdir)/camellia-aarch64-ce.c Makefile - `echo $(LTCOMPILE) $(aarch64_crypto_cflags) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(aarch64_crypto_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) cipher-gcm-aarch64-simd.o: $(srcdir)/cipher-gcm-aarch64-simd.c Makefile - `echo $(COMPILE) $(aarch64_simd_cflags) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(aarch64_simd_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) cipher-gcm-aarch64-simd.lo: $(srcdir)/cipher-gcm-aarch64-simd.c Makefile - `echo $(LTCOMPILE) $(aarch64_simd_cflags) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(aarch64_simd_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) rijndael-vp-aarch64.o: $(srcdir)/rijndael-vp-aarch64.c Makefile - `echo $(COMPILE) $(aarch64_simd_cflags) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(aarch64_simd_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) rijndael-vp-aarch64.lo: $(srcdir)/rijndael-vp-aarch64.c Makefile - `echo $(LTCOMPILE) $(aarch64_simd_cflags) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(aarch64_simd_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) if ENABLE_X86_AVX512_INTRINSICS_EXTRA_CFLAGS avx512f_cflags = -mavx512f -mavx512bw @@ -363,10 +421,12 @@ avx512f_cflags = endif serpent-avx512-x86.o: $(srcdir)/serpent-avx512-x86.c Makefile - `echo $(COMPILE) $(avx512f_cflags) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(avx512f_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) serpent-avx512-x86.lo: $(srcdir)/serpent-avx512-x86.c Makefile - `echo $(LTCOMPILE) $(avx512f_cflags) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(avx512f_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) if ENABLE_RISCV_VECTOR_INTRINSICS_EXTRA_CFLAGS # Note: -mstrict-align needed for SpacemiT K1 RVV bug (K1 does not support @@ -379,16 +439,20 @@ riscv_vector_cflags = endif chacha20-riscv-v.o: $(srcdir)/chacha20-riscv-v.c Makefile - `echo $(COMPILE) $(riscv_vector_cflags) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(riscv_vector_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) chacha20-riscv-v.lo: $(srcdir)/chacha20-riscv-v.c Makefile - `echo $(LTCOMPILE) $(riscv_vector_cflags) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(riscv_vector_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) rijndael-vp-riscv.o: $(srcdir)/rijndael-vp-riscv.c Makefile - `echo $(COMPILE) $(riscv_vector_cflags) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(riscv_vector_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) rijndael-vp-riscv.lo: $(srcdir)/rijndael-vp-riscv.c Makefile - `echo $(LTCOMPILE) $(riscv_vector_cflags) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(riscv_vector_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) if ENABLE_RISCV_VECTOR_CRYPTO_INTRINSICS_EXTRA_CFLAGS riscv_vector_crypto_aes_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvkned_zvkb @@ -401,25 +465,33 @@ riscv_vector_crypto_gcm_cflags = endif cipher-gcm-riscv-zvkg.o: $(srcdir)/cipher-gcm-riscv-zvkg.c Makefile - `echo $(COMPILE) $(riscv_vector_crypto_gcm_cflags) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(riscv_vector_crypto_gcm_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) cipher-gcm-riscv-zvkg.lo: $(srcdir)/cipher-gcm-riscv-zvkg.c Makefile - `echo $(LTCOMPILE) $(riscv_vector_crypto_gcm_cflags) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(riscv_vector_crypto_gcm_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) rijndael-riscv-zvkned.o: $(srcdir)/rijndael-riscv-zvkned.c Makefile - `echo $(COMPILE) $(riscv_vector_crypto_aes_cflags) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(riscv_vector_crypto_aes_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) rijndael-riscv-zvkned.lo: $(srcdir)/rijndael-riscv-zvkned.c Makefile - `echo $(LTCOMPILE) $(riscv_vector_crypto_aes_cflags) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(riscv_vector_crypto_aes_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) sha256-riscv-zvknha-zvkb.o: $(srcdir)/sha256-riscv-zvknha-zvkb.c Makefile - `echo $(COMPILE) $(riscv_vector_crypto_sha_cflags) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(riscv_vector_crypto_sha_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) sha256-riscv-zvknha-zvkb.lo: $(srcdir)/sha256-riscv-zvknha-zvkb.c Makefile - `echo $(LTCOMPILE) $(riscv_vector_crypto_sha_cflags) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(riscv_vector_crypto_sha_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) sha512-riscv-zvknhb-zvkb.o: $(srcdir)/sha512-riscv-zvknhb-zvkb.c Makefile - `echo $(COMPILE) $(riscv_vector_crypto_sha_cflags) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(riscv_vector_crypto_sha_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) sha512-riscv-zvknhb-zvkb.lo: $(srcdir)/sha512-riscv-zvknhb-zvkb.c Makefile - `echo $(LTCOMPILE) $(riscv_vector_crypto_sha_cflags) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(riscv_vector_crypto_sha_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) -- 2.53.0 From jussi.kivilinna at iki.fi Sun Oct 11 17:44:07 2026 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sun, 11 Oct 2026 18:44:07 +0300 Subject: [PATCH] poly1305-avx512: fix lost carry in conversion from base 2^44 Message-ID: <20261011154407.492622-1-jussi.kivilinna@iki.fi> * cipher/poly1305-amd64-avx512.S (POLY1305_BLOCKS): Use add with carry instead of OR for combining first and second limb. * tests/basic.c (check_mac): Add crafted Poly1305 test vector. -- After final partial reduction, first 44-bit limb of horizontally summed hash may exceed 44 bits. Combining limbs with OR then loses carry and gives wrong tag for some inputs. GnuPG-bug-id: 8514 Signed-off-by: Jussi Kivilinna --- cipher/poly1305-amd64-avx512.S | 6 ++- tests/basic.c | 71 ++++++++++++++++++++++++++++++++++ 2 files changed, 76 insertions(+), 1 deletion(-) diff --git a/cipher/poly1305-amd64-avx512.S b/cipher/poly1305-amd64-avx512.S index 250db072..1f7bd736 100644 --- a/cipher/poly1305-amd64-avx512.S +++ b/cipher/poly1305-amd64-avx512.S @@ -1401,12 +1401,12 @@ ELF(.size _gcry_poly1305_avx512_consts,.-_gcry_poly1305_avx512_consts) vpaddq xmm13, xmm13, xmm0; \ \ /* ; Put together A */ \ + /* ; First limb can exceed 44 bits after reduction, add with carry */ \ vmovq A0, xmm13; \ \ vmovq T0, xmm14; \ mov T1, T0; \ shl T1, 44; \ - or A0, T1; \ \ shr T0, 20; \ vmovq A2, xmm15; \ @@ -1415,6 +1415,10 @@ ELF(.size _gcry_poly1305_avx512_consts,.-_gcry_poly1305_avx512_consts) or A1, T0; \ shr A2, 40; \ \ + add A0, T1; \ + adc A1, 0; \ + adc A2, 0; \ + \ /* ; Clear powers of R */ \ vpxorq zmm0, zmm0, zmm0; \ vmovdqa64 [rsp + STACK_r_save], zmm0; \ diff --git a/tests/basic.c b/tests/basic.c index ab627786..d1793eea 100644 --- a/tests/basic.c +++ b/tests/basic.c @@ -17513,6 +17513,77 @@ check_mac (void) "\x12\x97\x6a\x08\xc4\x42\x6d\x0c\xe8\xa8\x24\x07\xc4\xf4\x82\x07" "\x80\xf8\xc2\x0a\xa7\x12\x02\xd1\xe2\x91\x79\xcb\xcb\x55\x5a\x57", "\x51\x54\xad\x0d\x2c\xb2\x6e\x01\x27\x4f\xc5\x11\x48\x49\x1f\x1b" }, + /* Crafted to stress final reduction of vector implementations */ + { GCRY_MAC_POLY1305, + "\xed\x2b\x47\x3b\x67\x72\x98\xd6\x9d\x70\xe7\x5b\x1e\x84\x00\x2d" + "\x3b\x63\x69\xcb\xaa\x33\x6c\xdf\xdf\x3f\xc5\x1b\x3b\x7f\x76\x59" + "\x78\xdd\x66\x4f\x81\x9f\xa1\x08\xa2\x1c\xe8\x00\x9b\x93\x7a\xbe" + "\x65\x9a\xce\x7f\x0d\x80\x52\x6d\x27\xe5\x68\x0f\x00\x07\x19\x6a" + "\xb7\x5c\x1e\x8b\x08\xfb\xcb\x35\x14\x74\x9a\x87\x25\x7a\x9a\x76" + "\x2c\xf1\x8f\x07\x36\x84\x0c\x05\x5d\x02\xae\xa8\x14\xa2\x08\x07" + "\x3a\xfc\x9b\xb0\x60\x0d\xff\x91\x16\xf4\x63\x9f\x72\x32\x56\x9a" + "\x3d\x09\x53\x5b\xa4\x77\x05\x1f\x74\x71\x37\x43\x8f\x56\xd6\xf2" + "\xea\x78\x12\x45\xf5\x82\x20\x46\x9d\x90\x7e\xb7\x43\x05\x46\xa4" + "\x5a\x59\xb8\xa2\x7c\x08\x8b\x6b\x17\x5b\x73\x1e\x57\x34\x7b\xa3" + "\x36\x43\xf4\xfb\x93\x48\x13\x52\xcb\x4c\x27\x50\xfa\x5d\x28\x97" + "\xb8\xd9\xbb\x98\x2b\x20\xbf\x02\x89\x6f\x27\x7a\x88\xd9\xa8\xce" + "\xe1\xe8\xf5\xb1\x9f\x63\x59\xa8\xd8\x62\xb5\xb9\x8a\x20\xe5\xb3" + "\x44\x30\x21\x7c\x40\x94\xce\x13\x9f\xc0\x20\x44\x81\x5f\x41\xf5" + "\xa5\xac\xeb\x4f\xc6\xa8\x80\xe9\xda\x37\xff\xaf\xb1\x15\x9f\xd1" + "\x08\x0f\xc3\xef\x84\x25\x99\x4d\x28\xdb\xe2\x1b\x03\x51\x52\xc5" + "\x8d\x83\x87\xb3\x47\xae\x30\x27\x06\x2b\x84\xde\xaf\xd8\x8c\x92" + "\xf1\xab\xa9\xb4\x19\x27\xc9\x24\x05\xff\x4b\x61\x4c\x8b\x1d\xde" + "\xbf\x4c\x2d\x25\x3a\x71\x63\xe2\x3c\xfc\x91\x94\xc8\x16\x78\xdc" + "\x6a\xa1\x99\x50\x26\x60\x8a\x74\x8b\x1f\x1e\x97\x96\x03\xd0\x79" + "\x99\x63\x9c\xd6\xe0\x9d\xc4\x3b\xb5\x3c\x72\x99\x74\x47\xdf\x1b" + "\xff\x07\x90\x16\xc5\x91\x0f\xcd\xb4\x66\x63\xc2\x13\x5e\x20\x70" + "\x01\x9e\x0e\x09\x6e\x48\x13\x7e\x93\xee\x6d\xb1\x35\x83\x61\x8b" + "\xb4\xfc\x2f\x4a\x14\x61\xd9\x53\x77\x3a\x82\xb8\xbc\x3b\x97\x3f" + "\x68\x61\x5d\xd4\xa0\x44\x8c\x0b\x0f\x60\xca\x5a\x9a\xbc\x29\xf1" + "\x68\x13\xa2\x27\xb0\x54\x40\x63\x62\x97\x75\xed\x18\x61\xf4\x8c" + "\x9a\xfe\xb6\xa6\x58\xe6\xd8\x60\xaa\x42\x75\x6d\x10\xb1\x77\xda" + "\x2f\x0b\xd2\x5d\xac\xde\x94\x50\x3c\xf1\x99\x71\x5e\xb2\x5b\x0c" + "\x1c\xb5\xd9\x59\x38\x7e\xaf\x4d\xc0\x24\x6e\x11\x44\x10\xb9\xe0" + "\x4d\x06\xc2\x6e\x72\x5e\x01\x33\x5a\x76\xe3\x0c\xec\x4c\x23\xf6" + "\xf9\x43\x93\xe9\xa5\x3c\x15\x10\xcb\x12\xcb\x1b\xc9\xe4\xce\x21" + "\x22\x9a\x94\x60\x27\x89\x0c\x05\x9e\xc9\x71\x1d\xce\x96\x06\x2b" + "\xbb\x9f\x68\xaf\xb3\xf2\x6d\xad\xab\xc4\x93\xda\x58\x7e\x45\xbd" + "\x18\x78\x17\x57\xee\x5b\x9a\xa0\xcf\x0e\x8f\x98\x24\x3e\xc1\xfb" + "\xed\x35\xd5\x6e\x26\x80\x0f\x73\x54\xfe\x2d\x72\x1e\x0b\x1b\x34" + "\x53\x77\x62\xf0\x1e\xe4\x03\x00\x86\xa9\xeb\x1a\xf4\xb6\x7e\xa2" + "\x88\x70\xe8\x4f\xf8\xa2\xc3\x7c\x9e\x01\xaa\x20\x7b\xb4\xc1\x8a" + "\x32\xa5\xab\xc4\x53\x5b\x59\x77\x8f\x38\x05\x42\x35\xc7\x0e\xd4" + "\x5c\xd8\x74\xe6\x6a\x52\xc8\xb9\x33\x34\x79\x17\xdd\xd7\x0c\xcb" + "\x65\xeb\x4a\xb8\x73\x24\xa4\x4d\xa4\x17\xf0\x4f\x8f\xe9\x3e\xf4" + "\x6c\x81\x1b\xd9\x49\x6f\xf8\x48\x40\x97\x92\xb6\x6a\x72\x5c\x1c" + "\x46\x97\xa6\xd8\x0b\x1c\x40\x0c\xca\x27\x1b\x5d\x7e\x48\xe9\xe5" + "\x72\xf1\xa2\xc1\x36\xe9\xdb\x9e\x51\x0b\xdf\xfd\xab\xd1\x20\xf9" + "\x27\xf7\x09\x60\xe8\xdd\xb9\xfe\x90\x35\xd1\xa0\xb5\x59\x99\x12" + "\xc3\x6f\x65\x58\xf4\xba\x30\xcf\x7a\x58\xec\x54\x57\x10\x38\xbc" + "\xf3\x7b\x3e\x1a\x51\x15\x9f\x29\x07\x43\x24\x1a\xfe\x74\xdb\x40" + "\x81\x1c\xfe\x2f\x7a\x79\xf0\x94\x35\xa0\xc3\x47\x1c\xc4\x5f\x0d" + "\x25\x2c\xd6\xd8\x6a\x45\xa2\xa2\x83\xde\xbe\xe8\x5b\x2a\x72\xdf" + "\xe9\x14\xb3\x99\x06\x22\xae\x56\x8c\x72\xb0\xa8\x62\x3d\xb2\x85" + "\x64\x87\xfe\x52\x85\x4e\x2a\xe6\x79\x5d\x0d\xef\x80\x77\x99\x36" + "\xfc\x00\xa6\x65\xb8\x66\x16\xa4\x0c\x0c\xf3\x9a\xa5\x4d\xa2\x51" + "\x4d\xda\x44\x5f\x73\x95\xca\x48\x34\x6b\x24\x49\xd2\xd9\x8f\x54" + "\x49\x27\x24\x3c\x9b\x36\x90\xfb\x29\x30\x36\xf4\xc7\x33\x81\xef" + "\x2f\x2b\xec\x18\x45\xa5\x6b\x3e\xf5\x53\x60\xce\x40\x38\x9c\xda" + "\xcb\x10\x47\x2e\xf9\xae\x19\x92\x4c\x5f\x92\xc7\x2b\x26\x16\x4e" + "\xeb\x9d\xab\x70\x0e\x48\xf0\x72\xb0\x03\xf9\x73\x8f\xff\xf3\xc4" + "\x18\xb1\x23\xf2\x35\x48\x61\x20\xe3\x9a\x85\xa3\xb5\x10\x5c\x9f" + "\x74\x45\xba\x36\x3c\x95\x4b\x26\x8f\xda\x4c\xb7\xa9\x35\xf7\x6e" + "\x55\x30\x3d\x80\xd2\x78\x5b\xcf\x56\xe9\x80\xe1\x97\x52\x62\x04" + "\xc0\x82\x0b\x2c\x37\x0c\xf2\xf1\xf3\x49\x43\x08\x14\xb6\xe6\x33" + "\x37\x22\x13\xf1\xa6\xe4\x64\xf7\x27\xd9\x8d\x5b\x79\x2c\x15\x30" + "\x27\x80\xbe\x5c\x2b\x5c\xb5\x2d\x72\x14\x56\x81\x68\x17\xd0\x41" + "\xa8\x42\x8d\xa2\xb2\x66\x2e\xb7\xeb\x84\x07\x08\x79\x7b\x68\x7a" + "\x53\x9b\xc5\xcd\x0a\x33\x09\x4a\x15\x42\x83\x14\x65\x72\x37\x4f", + "\x5b\x2c\x65\x07\x45\x87\xeb\xa1\x8e\x7d\x3f\x46\xe2\xf8\xf2\x18" + "\x6a\x65\x84\x82\x2b\xf7\x11\xdc\x39\x78\xb3\xf2\xfd\xdb\x0f\xc5", + "\x71\x65\x84\x82\x2b\xf7\x11\xdc\x39\x78\xb3\xf2\xfd\xdb\x0f\xc5", + NULL, + 1024, 32 }, { GCRY_MAC_POLY1305, "?", "????????????????????????????????", "\xc3\x88\xce\x8a\x52\xd6\xe7\x21\x86\xfa\xaa\x5d\x2d\x16\xf9\xa3" }, /* from http://cr.yp.to/mac/poly1305-20050329.pdf */ -- 2.53.0 From jussi.kivilinna at iki.fi Sun Oct 11 18:41:48 2026 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sun, 11 Oct 2026 19:41:48 +0300 Subject: [PATCH v2] poly1305-avx512: fix lost carry and bit 130 in base 2^44 conversions In-Reply-To: <20261011154407.492622-1-jussi.kivilinna@iki.fi> References: <20261011154407.492622-1-jussi.kivilinna@iki.fi> Message-ID: <20261011164148.676287-1-jussi.kivilinna@iki.fi> * cipher/poly1305-amd64-avx512.S (POLY1305_BLOCKS): Use add with carry instead of OR for combining first and second limb. Do not mask third limb of accumulator and R^4 to 42 bits. * tests/basic.c (check_one_mac): Add mode for writing data in two parts. (check_mac): Use two part writes; add crafted Poly1305 test vectors. -- After final partial reduction, first 44-bit limb of horizontally summed hash may exceed 44 bits. Combining limbs with OR then loses carry. Partially reduced accumulator and R^4 can be 2^130 or larger. Masking third limb to 42 bits then drops 2^130. Previous call leaves accumulator above 2^130 when hash is small modulo 2^130-5. R^4 is above 2^130 for keys where r^4 mod 2^130-5 is small. Both give wrong tag for some inputs. GnuPG-bug-id: 8514 Signed-off-by: Jussi Kivilinna --- cipher/poly1305-amd64-avx512.S | 12 +-- tests/basic.c | 137 ++++++++++++++++++++++++++++++++- 2 files changed, 143 insertions(+), 6 deletions(-) diff --git a/cipher/poly1305-amd64-avx512.S b/cipher/poly1305-amd64-avx512.S index 250db072..e51cc1b4 100644 --- a/cipher/poly1305-amd64-avx512.S +++ b/cipher/poly1305-amd64-avx512.S @@ -882,8 +882,7 @@ ELF(.size _gcry_poly1305_avx512_consts,.-_gcry_poly1305_avx512_consts) and A0, [.Lmask_44 ADD_RIP]; /* ;; Second limb (A[77:52]) */ \ vmovq xmm6, A0; \ \ - shrd A1, A2, 24; \ - and A1, [.Lmask_42 ADD_RIP]; /* ;; Third limb (A[129:88]) */ \ + shrd A1, A2, 24; /* ;; Third limb (A[151:88], A can be >= 2^130) */ \ vmovq xmm7, A1; \ \ /* ; Load first block of data (128 bytes) */ \ @@ -1008,8 +1007,7 @@ ELF(.size _gcry_poly1305_avx512_consts,.-_gcry_poly1305_avx512_consts) and A0, [.Lmask_44 ADD_RIP]; /* ;; Second limb (R^4[87:44]) */ \ vpbroadcastq zmm23, A0; \ \ - shrd A1, A2, 24; \ - and A1, [.Lmask_42 ADD_RIP]; /* ;; Third limb (R^4[129:88]) */ \ + shrd A1, A2, 24; /* ;; Third limb (R^4[151:88], R^4 can be >= 2^130) */ \ vpbroadcastq zmm24, A1; \ \ /* ; Generate 4*5*R^4 */ \ @@ -1401,12 +1399,12 @@ ELF(.size _gcry_poly1305_avx512_consts,.-_gcry_poly1305_avx512_consts) vpaddq xmm13, xmm13, xmm0; \ \ /* ; Put together A */ \ + /* ; First limb can exceed 44 bits after reduction, add with carry */ \ vmovq A0, xmm13; \ \ vmovq T0, xmm14; \ mov T1, T0; \ shl T1, 44; \ - or A0, T1; \ \ shr T0, 20; \ vmovq A2, xmm15; \ @@ -1415,6 +1413,10 @@ ELF(.size _gcry_poly1305_avx512_consts,.-_gcry_poly1305_avx512_consts) or A1, T0; \ shr A2, 40; \ \ + add A0, T1; \ + adc A1, 0; \ + adc A2, 0; \ + \ /* ; Clear powers of R */ \ vpxorq zmm0, zmm0, zmm0; \ vmovdqa64 [rsp + STACK_r_save], zmm0; \ diff --git a/tests/basic.c b/tests/basic.c index ab627786..c39f74de 100644 --- a/tests/basic.c +++ b/tests/basic.c @@ -16478,7 +16478,7 @@ check_one_mac (int algo, const char *data, int datalen, goto out; } - if (test_buffering) + if (test_buffering == 1) { for (i = 0; i < datalen; i++) { @@ -16544,6 +16544,17 @@ check_one_mac (int algo, const char *data, int datalen, piecelen = piecelen * 2 - ((piecelen != startlen) ? startlen : 0); } } + else if (test_buffering == 2) + { + clutter_vector_registers(); + err = gcry_mac_write (hd, data, datalen / 2); + if (!err) + { + clutter_vector_registers(); + err = gcry_mac_write (hd, data + datalen / 2, + datalen - datalen / 2); + } + } else { clutter_vector_registers(); @@ -17513,6 +17524,127 @@ check_mac (void) "\x12\x97\x6a\x08\xc4\x42\x6d\x0c\xe8\xa8\x24\x07\xc4\xf4\x82\x07" "\x80\xf8\xc2\x0a\xa7\x12\x02\xd1\xe2\x91\x79\xcb\xcb\x55\x5a\x57", "\x51\x54\xad\x0d\x2c\xb2\x6e\x01\x27\x4f\xc5\x11\x48\x49\x1f\x1b" }, + /* Crafted to stress final reduction of vector implementations */ + { GCRY_MAC_POLY1305, + "\xed\x2b\x47\x3b\x67\x72\x98\xd6\x9d\x70\xe7\x5b\x1e\x84\x00\x2d" + "\x3b\x63\x69\xcb\xaa\x33\x6c\xdf\xdf\x3f\xc5\x1b\x3b\x7f\x76\x59" + "\x78\xdd\x66\x4f\x81\x9f\xa1\x08\xa2\x1c\xe8\x00\x9b\x93\x7a\xbe" + "\x65\x9a\xce\x7f\x0d\x80\x52\x6d\x27\xe5\x68\x0f\x00\x07\x19\x6a" + "\xb7\x5c\x1e\x8b\x08\xfb\xcb\x35\x14\x74\x9a\x87\x25\x7a\x9a\x76" + "\x2c\xf1\x8f\x07\x36\x84\x0c\x05\x5d\x02\xae\xa8\x14\xa2\x08\x07" + "\x3a\xfc\x9b\xb0\x60\x0d\xff\x91\x16\xf4\x63\x9f\x72\x32\x56\x9a" + "\x3d\x09\x53\x5b\xa4\x77\x05\x1f\x74\x71\x37\x43\x8f\x56\xd6\xf2" + "\xea\x78\x12\x45\xf5\x82\x20\x46\x9d\x90\x7e\xb7\x43\x05\x46\xa4" + "\x5a\x59\xb8\xa2\x7c\x08\x8b\x6b\x17\x5b\x73\x1e\x57\x34\x7b\xa3" + "\x36\x43\xf4\xfb\x93\x48\x13\x52\xcb\x4c\x27\x50\xfa\x5d\x28\x97" + "\xb8\xd9\xbb\x98\x2b\x20\xbf\x02\x89\x6f\x27\x7a\x88\xd9\xa8\xce" + "\xe1\xe8\xf5\xb1\x9f\x63\x59\xa8\xd8\x62\xb5\xb9\x8a\x20\xe5\xb3" + "\x44\x30\x21\x7c\x40\x94\xce\x13\x9f\xc0\x20\x44\x81\x5f\x41\xf5" + "\xa5\xac\xeb\x4f\xc6\xa8\x80\xe9\xda\x37\xff\xaf\xb1\x15\x9f\xd1" + "\x08\x0f\xc3\xef\x84\x25\x99\x4d\x28\xdb\xe2\x1b\x03\x51\x52\xc5" + "\x8d\x83\x87\xb3\x47\xae\x30\x27\x06\x2b\x84\xde\xaf\xd8\x8c\x92" + "\xf1\xab\xa9\xb4\x19\x27\xc9\x24\x05\xff\x4b\x61\x4c\x8b\x1d\xde" + "\xbf\x4c\x2d\x25\x3a\x71\x63\xe2\x3c\xfc\x91\x94\xc8\x16\x78\xdc" + "\x6a\xa1\x99\x50\x26\x60\x8a\x74\x8b\x1f\x1e\x97\x96\x03\xd0\x79" + "\x99\x63\x9c\xd6\xe0\x9d\xc4\x3b\xb5\x3c\x72\x99\x74\x47\xdf\x1b" + "\xff\x07\x90\x16\xc5\x91\x0f\xcd\xb4\x66\x63\xc2\x13\x5e\x20\x70" + "\x01\x9e\x0e\x09\x6e\x48\x13\x7e\x93\xee\x6d\xb1\x35\x83\x61\x8b" + "\xb4\xfc\x2f\x4a\x14\x61\xd9\x53\x77\x3a\x82\xb8\xbc\x3b\x97\x3f" + "\x68\x61\x5d\xd4\xa0\x44\x8c\x0b\x0f\x60\xca\x5a\x9a\xbc\x29\xf1" + "\x68\x13\xa2\x27\xb0\x54\x40\x63\x62\x97\x75\xed\x18\x61\xf4\x8c" + "\x9a\xfe\xb6\xa6\x58\xe6\xd8\x60\xaa\x42\x75\x6d\x10\xb1\x77\xda" + "\x2f\x0b\xd2\x5d\xac\xde\x94\x50\x3c\xf1\x99\x71\x5e\xb2\x5b\x0c" + "\x1c\xb5\xd9\x59\x38\x7e\xaf\x4d\xc0\x24\x6e\x11\x44\x10\xb9\xe0" + "\x4d\x06\xc2\x6e\x72\x5e\x01\x33\x5a\x76\xe3\x0c\xec\x4c\x23\xf6" + "\xf9\x43\x93\xe9\xa5\x3c\x15\x10\xcb\x12\xcb\x1b\xc9\xe4\xce\x21" + "\x22\x9a\x94\x60\x27\x89\x0c\x05\x9e\xc9\x71\x1d\xce\x96\x06\x2b" + "\xbb\x9f\x68\xaf\xb3\xf2\x6d\xad\xab\xc4\x93\xda\x58\x7e\x45\xbd" + "\x18\x78\x17\x57\xee\x5b\x9a\xa0\xcf\x0e\x8f\x98\x24\x3e\xc1\xfb" + "\xed\x35\xd5\x6e\x26\x80\x0f\x73\x54\xfe\x2d\x72\x1e\x0b\x1b\x34" + "\x53\x77\x62\xf0\x1e\xe4\x03\x00\x86\xa9\xeb\x1a\xf4\xb6\x7e\xa2" + "\x88\x70\xe8\x4f\xf8\xa2\xc3\x7c\x9e\x01\xaa\x20\x7b\xb4\xc1\x8a" + "\x32\xa5\xab\xc4\x53\x5b\x59\x77\x8f\x38\x05\x42\x35\xc7\x0e\xd4" + "\x5c\xd8\x74\xe6\x6a\x52\xc8\xb9\x33\x34\x79\x17\xdd\xd7\x0c\xcb" + "\x65\xeb\x4a\xb8\x73\x24\xa4\x4d\xa4\x17\xf0\x4f\x8f\xe9\x3e\xf4" + "\x6c\x81\x1b\xd9\x49\x6f\xf8\x48\x40\x97\x92\xb6\x6a\x72\x5c\x1c" + "\x46\x97\xa6\xd8\x0b\x1c\x40\x0c\xca\x27\x1b\x5d\x7e\x48\xe9\xe5" + "\x72\xf1\xa2\xc1\x36\xe9\xdb\x9e\x51\x0b\xdf\xfd\xab\xd1\x20\xf9" + "\x27\xf7\x09\x60\xe8\xdd\xb9\xfe\x90\x35\xd1\xa0\xb5\x59\x99\x12" + "\xc3\x6f\x65\x58\xf4\xba\x30\xcf\x7a\x58\xec\x54\x57\x10\x38\xbc" + "\xf3\x7b\x3e\x1a\x51\x15\x9f\x29\x07\x43\x24\x1a\xfe\x74\xdb\x40" + "\x81\x1c\xfe\x2f\x7a\x79\xf0\x94\x35\xa0\xc3\x47\x1c\xc4\x5f\x0d" + "\x25\x2c\xd6\xd8\x6a\x45\xa2\xa2\x83\xde\xbe\xe8\x5b\x2a\x72\xdf" + "\xe9\x14\xb3\x99\x06\x22\xae\x56\x8c\x72\xb0\xa8\x62\x3d\xb2\x85" + "\x64\x87\xfe\x52\x85\x4e\x2a\xe6\x79\x5d\x0d\xef\x80\x77\x99\x36" + "\xfc\x00\xa6\x65\xb8\x66\x16\xa4\x0c\x0c\xf3\x9a\xa5\x4d\xa2\x51" + "\x4d\xda\x44\x5f\x73\x95\xca\x48\x34\x6b\x24\x49\xd2\xd9\x8f\x54" + "\x49\x27\x24\x3c\x9b\x36\x90\xfb\x29\x30\x36\xf4\xc7\x33\x81\xef" + "\x2f\x2b\xec\x18\x45\xa5\x6b\x3e\xf5\x53\x60\xce\x40\x38\x9c\xda" + "\xcb\x10\x47\x2e\xf9\xae\x19\x92\x4c\x5f\x92\xc7\x2b\x26\x16\x4e" + "\xeb\x9d\xab\x70\x0e\x48\xf0\x72\xb0\x03\xf9\x73\x8f\xff\xf3\xc4" + "\x18\xb1\x23\xf2\x35\x48\x61\x20\xe3\x9a\x85\xa3\xb5\x10\x5c\x9f" + "\x74\x45\xba\x36\x3c\x95\x4b\x26\x8f\xda\x4c\xb7\xa9\x35\xf7\x6e" + "\x55\x30\x3d\x80\xd2\x78\x5b\xcf\x56\xe9\x80\xe1\x97\x52\x62\x04" + "\xc0\x82\x0b\x2c\x37\x0c\xf2\xf1\xf3\x49\x43\x08\x14\xb6\xe6\x33" + "\x37\x22\x13\xf1\xa6\xe4\x64\xf7\x27\xd9\x8d\x5b\x79\x2c\x15\x30" + "\x27\x80\xbe\x5c\x2b\x5c\xb5\x2d\x72\x14\x56\x81\x68\x17\xd0\x41" + "\xa8\x42\x8d\xa2\xb2\x66\x2e\xb7\xeb\x84\x07\x08\x79\x7b\x68\x7a" + "\x53\x9b\xc5\xcd\x0a\x33\x09\x4a\x15\x42\x83\x14\x65\x72\x37\x4f", + "\x5b\x2c\x65\x07\x45\x87\xeb\xa1\x8e\x7d\x3f\x46\xe2\xf8\xf2\x18" + "\x6a\x65\x84\x82\x2b\xf7\x11\xdc\x39\x78\xb3\xf2\xfd\xdb\x0f\xc5", + "\x71\x65\x84\x82\x2b\xf7\x11\xdc\x39\x78\xb3\xf2\xfd\xdb\x0f\xc5", + NULL, + 1024, 32 }, + /* Weak key, r^4 mod 2^130-5 is small */ + { GCRY_MAC_POLY1305, + "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" + "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" + "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" + "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", + "\x53\xa6\xe6\x01\x5c\x9b\xc3\x00\x50\xf4\xf4\x0e\x90\x5c\x3c\x01" + "\x02\xfc\xe9\x7d\x82\xa0\xbb\x58\xc9\xb8\xe9\x7e\xec\xa1\x87\x56", + "\x09\x51\x86\x6b\x4e\xe4\xb4\xfb\xd5\xb6\x21\x99\x68\x23\x88\xe4", + NULL, + 256, 32 }, + /* Hash of first half is small modulo 2^130-5 */ + { GCRY_MAC_POLY1305, + "\x75\x88\xef\x9f\x88\x7f\xec\xcf\x91\x70\x9b\x5c\x5e\xa9\xab\x26" + "\x5a\xdd\x0b\xa8\x06\xac\xe4\x72\x58\x92\xb2\xfc\x46\x85\xc0\xe3" + "\xc5\x90\x0f\xe2\xb7\xa7\x58\xd7\xc2\x27\xdd\x8e\xa9\x7b\x79\x27" + "\xa0\x20\x6e\x93\xe1\x22\xf3\xb4\x66\x64\x07\xeb\x2a\xe2\x38\x40" + "\xe5\x1b\x04\xae\x5e\x0e\xe6\xc4\x8a\x9e\xbb\x8b\x5f\xdd\x3c\x13" + "\x78\x31\x4d\x7b\xe7\x2d\x05\xf9\xc9\x0a\x03\x81\x28\xf0\x80\xa4" + "\x88\x3f\xc0\xff\x33\xce\xe9\xc7\xbc\x01\x03\xf5\x29\x95\xee\x65" + "\x41\x31\xf9\x5d\x70\x54\xfe\x80\x01\x8c\xd9\xa4\xbb\xfd\x39\x7f" + "\x41\xab\xfa\x74\xdd\xa3\x3b\x55\xce\x7f\x2c\x50\xbb\xb8\x29\x29" + "\x8e\xd8\x9e\x15\xc8\x92\xd3\x73\x5d\x14\x93\x68\x71\xbe\x57\x22" + "\xd6\x70\xb0\xb9\x9a\x43\x5d\xf9\xda\x29\xad\xce\xf0\x4a\xc3\x8a" + "\x1f\xb7\x5b\xf6\xcc\xfd\xe9\x90\xb6\x2d\x86\x84\xfa\xb2\x6f\x9d" + "\xff\xae\x8f\xdf\x83\xa9\x0c\xee\x25\x26\xd5\x3c\xfe\xc5\xe1\xe1" + "\xb1\x84\x2c\xbf\x18\xf9\x4f\x87\x1d\xc2\x3f\xde\xcd\x26\x36\x9c" + "\xd3\x94\x72\xd1\x50\x86\xc1\x7b\xed\xb8\x4a\xd8\x44\x7f\x36\x02" + "\x2e\x58\x0d\x48\xa2\xdb\x90\x2d\x53\x90\x59\xab\x28\xd1\x02\xf3" + "\x97\xa9\xaf\x2b\x36\xe0\xc9\x17\xf9\x2e\x2c\x9f\x4f\x46\xb2\x99" + "\xae\x79\x34\x2c\x5e\x4c\xf6\xbb\x71\xd0\x54\x42\x37\x20\x1d\xd2" + "\x43\xba\x30\xd6\x23\x12\x4f\x8e\x58\xb3\x95\x8a\xcc\x6b\x4d\x15" + "\xfb\x10\x5e\x4c\x74\xd4\x0e\xfc\x90\xa7\xb6\xd3\xd5\x74\x90\x2c" + "\xf4\x97\xe2\x44\x41\x80\x23\xf4\xdb\x60\x02\x2e\x5c\xf3\x60\x93" + "\x47\xe4\xfd\x91\x7f\xa3\x73\x04\xbe\xee\x37\x80\xc7\xd0\xd2\x98" + "\x86\x9c\x08\x99\x3c\xcb\xc8\x1a\x46\xaf\xea\xda\x97\x66\x9e\x90" + "\x4f\x17\x11\x8b\xc7\x8a\x3a\xb7\x4d\x91\x47\x54\xea\x19\xad\xf0" + "\x0b\x8d\x37\x37\x2d\xba\x3d\x15\x27\x3f\x90\xf3\x75\xf8\x62\x09" + "\x11\x4b\x3d\x47\x75\x84\x71\x36\xad\x8a\xac\x30\xea\x52\xc6\xc5" + "\x56\xb7\x55\x54\xb8\xef\x5a\xda\x94\x4e\xcf\x7b\x88\x41\xa3\xc5" + "\x72\x23\x1a\x3b\x43\x8a\x1e\x04\x34\x52\x44\xef\x71\xbf\xdd\xdb" + "\x51\xe4\x02\xb0\xd4\xf5\x66\x4b\xfe\x2d\xe3\x54\x59\x52\x44\x72" + "\xb1\x31\x6b\xa2\xb9\xdd\x48\xb1\xff\x91\x3a\x77\xad\x19\xf4\x53" + "\xff\xe4\xb3\xd9\xde\xb7\xe5\x99\x87\xfd\xfd\x88\xa4\x5f\x7d\xb5" + "\xee\x40\xb1\xfd\x6a\x56\xac\xa8\x96\xcb\xc2\x65\x92\x14\xae\xd9", + "\xe3\x2b\x68\x47\xd9\x98\x0a\x0e\x63\xc5\x3d\x0a\xbc\x92\xaf\x5b" + "\x99\x21\xdd\x7a\x3c\xe2\x84\x09\xf4\x1a\x7e\xb0\x4a\x9e\xbf\x03", + "\x5d\xe1\xa5\x59\x97\xe7\x03\xff\x62\xb6\x1a\xcb\x61\x65\x1a\x25", + NULL, + 512, 32 }, { GCRY_MAC_POLY1305, "?", "????????????????????????????????", "\xc3\x88\xce\x8a\x52\xd6\xe7\x21\x86\xfa\xaa\x5d\x2d\x16\xf9\xa3" }, /* from http://cr.yp.to/mac/poly1305-20050329.pdf */ @@ -17608,6 +17740,9 @@ check_mac (void) check_one_mac (algos[i].algo, algos[i].data, dlen, algos[i].key, klen, algos[i].iv, algos[i].iv ? strlen(algos[i].iv) : 0, algos[i].expect, 1); + check_one_mac (algos[i].algo, algos[i].data, dlen, algos[i].key, klen, + algos[i].iv, algos[i].iv ? strlen(algos[i].iv) : 0, + algos[i].expect, 2); } #if USE_GOST28147 -- 2.53.0 From jussi.kivilinna at iki.fi Sun Oct 11 18:57:02 2026 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sun, 11 Oct 2026 19:57:02 +0300 Subject: [PATCH] poly1305-p10le: fix lost carry in conversion from base 2^26 In-Reply-To: <20261011154407.492622-1-jussi.kivilinna@iki.fi> References: <20261011154407.492622-1-jussi.kivilinna@iki.fi> Message-ID: <20261011165702.791514-1-jussi.kivilinna@iki.fi> * cipher/poly1305-p10le.s (gcry_poly1305_p10le_4blocks): Propagate carry through second, third and fourth limb before combining limbs with OR. -- After final carry reduction, second 26-bit limb may exceed 26 bits. Combining limbs with OR then loses carry and gives wrong tag for some inputs, for example when hash is small modulo 2^130-5. GnuPG-bug-id: 8515 Signed-off-by: Jussi Kivilinna --- cipher/poly1305-p10le.s | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/cipher/poly1305-p10le.s b/cipher/poly1305-p10le.s index 133b99a9..5437355a 100644 --- a/cipher/poly1305-p10le.s +++ b/cipher/poly1305-p10le.s @@ -758,6 +758,17 @@ loop_4blocks: vaddudm 5, 5, 10 vaddudm 8, 8, 11 + # carry v5..v7 so that only v8 can exceed 26 bits + vsrd 10, 5, 31 + vand 5, 5, 25 + vaddudm 6, 6, 10 + vsrd 10, 6, 31 + vand 6, 6, 25 + vaddudm 7, 7, 10 + vsrd 10, 7, 31 + vand 7, 7, 25 + vaddudm 8, 8, 10 + b do_final_update do_final_update: -- 2.53.0 From jussi.kivilinna at iki.fi Sun Oct 11 19:23:10 2026 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sun, 11 Oct 2026 20:23:10 +0300 Subject: [PATCH 01/12] tests/bench-slope: add --buf-start option Message-ID: <20261011172321.997183-1-jussi.kivilinna@iki.fi> * tests/bench-slope.c (BUF_END_SIZE): Remove. (BUF_RANGE_SIZE, buf_start_size): New. (bench_encrypt_init, bench_xts_encrypt_init, bench_hash_init) (bench_mac_init): Use 'buf_start_size' and 'BUF_RANGE_SIZE' for buffer size range. (print_help): Add '--buf-start'. (main): Parse '--buf-start'. -- Option moves measurement window to larger buffer sizes. This allows benchmarking implementations that switch to different code path at input size threshold. Signed-off-by: Jussi Kivilinna --- tests/bench-slope.c | 44 +++++++++++++++++++++++++++++++++++--------- 1 file changed, 35 insertions(+), 9 deletions(-) diff --git a/tests/bench-slope.c b/tests/bench-slope.c index 61eee018..50c303b7 100644 --- a/tests/bench-slope.c +++ b/tests/bench-slope.c @@ -87,11 +87,13 @@ static int in_fips_mode; /* From ~0 to ~4kbytes give comparable results with results from academia * (SUPERCOP). */ -#define BUF_END_SIZE (BUF_START_SIZE + 4096) +#define BUF_RANGE_SIZE 4096 /* With 128 byte steps, we get (4096)/64 = 64 data points. */ #define BUF_STEP_SIZE 64 +static unsigned int buf_start_size = BUF_START_SIZE; + /* Number of repeated measurements at each data point. The median of these * measurements is selected as data point further analysis. */ #define NUM_MEASUREMENT_REPETITIONS 64 @@ -1037,8 +1039,8 @@ bench_encrypt_init (struct bench_obj *obj) gcry_cipher_hd_t hd; int err, keylen; - obj->min_bufsize = BUF_START_SIZE; - obj->max_bufsize = BUF_END_SIZE; + obj->min_bufsize = buf_start_size; + obj->max_bufsize = buf_start_size + BUF_RANGE_SIZE; obj->step_size = BUF_STEP_SIZE; obj->num_measure_repetitions = num_measurement_repetitions; @@ -1137,8 +1139,8 @@ bench_xts_encrypt_init (struct bench_obj *obj) gcry_cipher_hd_t hd; int err, keylen; - obj->min_bufsize = BUF_START_SIZE; - obj->max_bufsize = BUF_END_SIZE; + obj->min_bufsize = buf_start_size; + obj->max_bufsize = buf_start_size + BUF_RANGE_SIZE; obj->step_size = BUF_STEP_SIZE; obj->num_measure_repetitions = num_measurement_repetitions; @@ -1911,8 +1913,8 @@ bench_hash_init (struct bench_obj *obj) gcry_md_hd_t hd; int err; - obj->min_bufsize = BUF_START_SIZE; - obj->max_bufsize = BUF_END_SIZE; + obj->min_bufsize = buf_start_size; + obj->max_bufsize = buf_start_size + BUF_RANGE_SIZE; obj->step_size = BUF_STEP_SIZE; obj->num_measure_repetitions = num_measurement_repetitions; @@ -2041,8 +2043,8 @@ bench_mac_init (struct bench_obj *obj) unsigned int keylen; void *key; - obj->min_bufsize = BUF_START_SIZE; - obj->max_bufsize = BUF_END_SIZE; + obj->min_bufsize = buf_start_size; + obj->max_bufsize = buf_start_size + BUF_RANGE_SIZE; obj->step_size = BUF_STEP_SIZE; obj->num_measure_repetitions = num_measurement_repetitions; @@ -4512,6 +4514,9 @@ print_help (void) " --repetitions Use N repetitions (default " STR2(NUM_MEASUREMENT_REPETITIONS) ")", " --unaligned Use unaligned input buffers.", + " --buf-start Start buffer size for measurements", + " (multiple of 16, default " + STR2(BUF_START_SIZE) ")", " --no-quick-rng Use default random number generation", " --include-slow Include slow benchmarks in default run", " --csv Use CSV output format", @@ -4624,6 +4629,27 @@ main (int argc, char **argv) argc--; argv++; } + else if (!strcmp (*argv, "--buf-start")) + { + argc--; + argv++; + if (argc) + { + int val = atoi (*argv); + + buf_start_size = val; + if (val < 16 || val > 1024 * 1024 || (val % 16) != 0) + { + fprintf (stderr, + PGM + ": invalid value for --buf-start - using %d\n", + BUF_START_SIZE); + buf_start_size = BUF_START_SIZE; + } + argc--; + argv++; + } + } else if (!strcmp (*argv, "--disable-hwf")) { argc--; -- 2.53.0 From jussi.kivilinna at iki.fi Sun Oct 11 19:23:12 2026 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sun, 11 Oct 2026 20:23:12 +0300 Subject: [PATCH 03/12] chacha20-riscv: use 4-field segment load/store in multi-block path In-Reply-To: <20261011172321.997183-1-jussi.kivilinna@iki.fi> References: <20261011172321.997183-1-jussi.kivilinna@iki.fi> Message-ID: <20261011172321.997183-3-jussi.kivilinna@iki.fi> * cipher/chacha20-riscv-rvv.h (vxor_v_u32m1_u32m1x8): Replace with... (vxor_v_u32m1_u32m1x4): ... this. (unaligned_vlsseg8e32_v_u32m1x8): Replace with... (unaligned_vlsseg4e32_v_u32m1x4): ... this. (unaligned_vssseg8e32_v_u32m1x8): Replace with... (unaligned_vssseg4e32_v_u32m1x4): ... this. (chacha20_rvv_blocks): Use 4-field strided segment loads and stores instead of 8-field. -- With 8-field segment load/store, 16 state vectors and input and output tuples did not fit to vector register file and GCC spilled tuple to stack in multi-block loop. With 4-field segments, there is no spilling with GCC-14, GCC-15 or clang. Benchmark on SpacemiT K3, X100 core (2000 Mhz, out-of-order, VLEN=256), RISC-V Zvkb implementation: Before: CHACHA20 | nanosecs/byte mebibytes/sec cycles/byte STREAM enc | 1.52 ns/B 627.3 MiB/s 3.04 c/B POLY1305 enc | 2.33 ns/B 409.9 MiB/s 4.65 c/B After (STREAM ~1.19x, POLY1305 ~1.11x faster): CHACHA20 | nanosecs/byte mebibytes/sec cycles/byte STREAM enc | 1.28 ns/B 743.8 MiB/s 2.56 c/B POLY1305 enc | 2.09 ns/B 456.2 MiB/s 4.18 c/B Benchmark on SpacemiT K3, A100 core (1800 Mhz, in-order, VLEN=1024), RISC-V Zvkb implementation: Before: CHACHA20 | nanosecs/byte mebibytes/sec cycles/byte STREAM enc | 3.78 ns/B 252.6 MiB/s 6.80 c/B POLY1305 enc | 7.64 ns/B 124.9 MiB/s 13.75 c/B After (STREAM ~3.0x, POLY1305 ~2.0x faster): CHACHA20 | nanosecs/byte mebibytes/sec cycles/byte STREAM enc | 1.25 ns/B 765.6 MiB/s 2.24 c/B POLY1305 enc | 3.75 ns/B 254.2 MiB/s 6.75 c/B Benchmark on SpacemiT K1, X60 core (1600 Mhz, in-order, VLEN=256), RISC-V vector implementation (GCC-14): Before: CHACHA20 | nanosecs/byte mebibytes/sec cycles/byte STREAM enc | 3.44 ns/B 277.6 MiB/s 5.50 c/B POLY1305 enc | 6.53 ns/B 146.0 MiB/s 10.45 c/B After (STREAM ~1.06x faster): CHACHA20 | nanosecs/byte mebibytes/sec cycles/byte STREAM enc | 3.23 ns/B 295.3 MiB/s 5.17 c/B POLY1305 enc | 6.48 ns/B 147.1 MiB/s 10.37 c/B Signed-off-by: Jussi Kivilinna --- cipher/chacha20-riscv-rvv.h | 112 ++++++++++++++++-------------------- 1 file changed, 51 insertions(+), 61 deletions(-) diff --git a/cipher/chacha20-riscv-rvv.h b/cipher/chacha20-riscv-rvv.h index abd53c99..267584a5 100644 --- a/cipher/chacha20-riscv-rvv.h +++ b/cipher/chacha20-riscv-rvv.h @@ -108,10 +108,10 @@ __riscv_vreinterpret_v_u32m1_u64m1(a), \ __riscv_vreinterpret_v_u32m1_u64m1(b), vl / 2)) -#define vxor_v_u32m1_u32m1x8(data, idx, vs, vl) \ - __riscv_vset_v_u32m1_u32m1x8((data), (idx), \ +#define vxor_v_u32m1_u32m1x4(data, idx, vs, vl) \ + __riscv_vset_v_u32m1_u32m1x4((data), (idx), \ __riscv_vxor_vv_u32m1( \ - __riscv_vget_v_u32m1x8_u32m1((data), (idx)), (vs), (vl))) + __riscv_vget_v_u32m1x4_u32m1((data), (idx)), (vs), (vl))) #ifdef LOAD_GATHER_ROTATE_CONSTANTS @@ -167,16 +167,16 @@ unaligned_store_u32m1(void *dst, vuint32m1_t vec, size_t vl) #endif } -static ASM_FUNC_ATTR_INLINE vuint32m1x8_t -unaligned_vlsseg8e32_v_u32m1x8(const void *src, size_t vl) +static ASM_FUNC_ATTR_INLINE vuint32m1x4_t +unaligned_vlsseg4e32_v_u32m1x4(const void *src, size_t vl) { #ifdef RVV_UNALIGNED_NOT_ALLOWED if (UNLIKELY(((uintptr_t)src & 3) != 0)) { const byte *bsrc = src; vuint16m2_t indexes; - vuint8m1_t b0, b1, b2, b3, b4, b5, b6, b7; - vuint32m1x8_t data; + vuint8m1_t b0, b1, b2, b3; + vuint32m1x4_t data; indexes = gen_indexes(4 * vl, 64); @@ -184,28 +184,16 @@ unaligned_vlsseg8e32_v_u32m1x8(const void *src, size_t vl) b1 = __riscv_vluxei16_v_u8m1(bsrc + 1 * 4, indexes, vl * 4); b2 = __riscv_vluxei16_v_u8m1(bsrc + 2 * 4, indexes, vl * 4); b3 = __riscv_vluxei16_v_u8m1(bsrc + 3 * 4, indexes, vl * 4); - b4 = __riscv_vluxei16_v_u8m1(bsrc + 4 * 4, indexes, vl * 4); - b5 = __riscv_vluxei16_v_u8m1(bsrc + 5 * 4, indexes, vl * 4); - b6 = __riscv_vluxei16_v_u8m1(bsrc + 6 * 4, indexes, vl * 4); - b7 = __riscv_vluxei16_v_u8m1(bsrc + 7 * 4, indexes, vl * 4); - data = __riscv_vundefined_u32m1x8(); - data = __riscv_vset_v_u32m1_u32m1x8( + data = __riscv_vundefined_u32m1x4(); + data = __riscv_vset_v_u32m1_u32m1x4( data, 0, __riscv_vreinterpret_v_u8m1_u32m1(b0)); - data = __riscv_vset_v_u32m1_u32m1x8( + data = __riscv_vset_v_u32m1_u32m1x4( data, 1, __riscv_vreinterpret_v_u8m1_u32m1(b1)); - data = __riscv_vset_v_u32m1_u32m1x8( + data = __riscv_vset_v_u32m1_u32m1x4( data, 2, __riscv_vreinterpret_v_u8m1_u32m1(b2)); - data = __riscv_vset_v_u32m1_u32m1x8( + data = __riscv_vset_v_u32m1_u32m1x4( data, 3, __riscv_vreinterpret_v_u8m1_u32m1(b3)); - data = __riscv_vset_v_u32m1_u32m1x8( - data, 4, __riscv_vreinterpret_v_u8m1_u32m1(b4)); - data = __riscv_vset_v_u32m1_u32m1x8( - data, 5, __riscv_vreinterpret_v_u8m1_u32m1(b5)); - data = __riscv_vset_v_u32m1_u32m1x8( - data, 6, __riscv_vreinterpret_v_u8m1_u32m1(b6)); - data = __riscv_vset_v_u32m1_u32m1x8( - data, 7, __riscv_vreinterpret_v_u8m1_u32m1(b7)); return data; } @@ -213,45 +201,37 @@ unaligned_vlsseg8e32_v_u32m1x8(const void *src, size_t vl) #endif { /* Fast path for 32-bit aligned loads. */ - return __riscv_vlsseg8e32_v_u32m1x8(src, 64, vl); + return __riscv_vlsseg4e32_v_u32m1x4(src, 64, vl); } } static ASM_FUNC_ATTR_INLINE void -unaligned_vssseg8e32_v_u32m1x8(void *dst, vuint32m1x8_t data, size_t vl) +unaligned_vssseg4e32_v_u32m1x4(void *dst, vuint32m1x4_t data, size_t vl) { #ifdef RVV_UNALIGNED_NOT_ALLOWED if (UNLIKELY(((uintptr_t)dst & 3) != 0)) { byte *bdst = dst; vuint16m2_t indexes; - vuint8m1_t b0, b1, b2, b3, b4, b5, b6, b7; + vuint8m1_t b0, b1, b2, b3; indexes = gen_indexes(4 * vl, 64); - b0 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x8_u32m1(data, 0)); - b1 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x8_u32m1(data, 1)); - b2 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x8_u32m1(data, 2)); - b3 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x8_u32m1(data, 3)); - b4 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x8_u32m1(data, 4)); - b5 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x8_u32m1(data, 5)); - b6 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x8_u32m1(data, 6)); - b7 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x8_u32m1(data, 7)); + b0 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x4_u32m1(data, 0)); + b1 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x4_u32m1(data, 1)); + b2 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x4_u32m1(data, 2)); + b3 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x4_u32m1(data, 3)); __riscv_vsuxei16_v_u8m1(bdst + 0 * 4, indexes, b0, vl * 4); __riscv_vsuxei16_v_u8m1(bdst + 1 * 4, indexes, b1, vl * 4); __riscv_vsuxei16_v_u8m1(bdst + 2 * 4, indexes, b2, vl * 4); __riscv_vsuxei16_v_u8m1(bdst + 3 * 4, indexes, b3, vl * 4); - __riscv_vsuxei16_v_u8m1(bdst + 4 * 4, indexes, b4, vl * 4); - __riscv_vsuxei16_v_u8m1(bdst + 5 * 4, indexes, b5, vl * 4); - __riscv_vsuxei16_v_u8m1(bdst + 6 * 4, indexes, b6, vl * 4); - __riscv_vsuxei16_v_u8m1(bdst + 7 * 4, indexes, b7, vl * 4); } else #endif { /* Fast path for 32-bit aligned stores. */ - __riscv_vssseg8e32_v_u32m1x8(dst, 64, data, vl); + __riscv_vssseg4e32_v_u32m1x4(dst, 64, data, vl); return; } } @@ -299,7 +279,7 @@ chacha20_rvv_blocks(u32 *input, byte *dst, const byte *src, size_t nblks) { vuint32m1_t ctr; vbool32_t carry; - vuint32m1x8_t data; + vuint32m1x4_t data; if (vl < 4) break; @@ -358,31 +338,41 @@ chacha20_rvv_blocks(u32 *input, byte *dst, const byte *src, size_t nblks) s12 += vl; s13 += s12 < vl; - data = unaligned_vlsseg8e32_v_u32m1x8((const void *)src, vl); + data = unaligned_vlsseg4e32_v_u32m1x4((const void *)src, vl); - data = vxor_v_u32m1_u32m1x8(data, 0, x0, vl); - data = vxor_v_u32m1_u32m1x8(data, 1, x1, vl); - data = vxor_v_u32m1_u32m1x8(data, 2, x2, vl); - data = vxor_v_u32m1_u32m1x8(data, 3, x3, vl); - data = vxor_v_u32m1_u32m1x8(data, 4, x4, vl); - data = vxor_v_u32m1_u32m1x8(data, 5, x5, vl); - data = vxor_v_u32m1_u32m1x8(data, 6, x6, vl); - data = vxor_v_u32m1_u32m1x8(data, 7, x7, vl); + data = vxor_v_u32m1_u32m1x4(data, 0, x0, vl); + data = vxor_v_u32m1_u32m1x4(data, 1, x1, vl); + data = vxor_v_u32m1_u32m1x4(data, 2, x2, vl); + data = vxor_v_u32m1_u32m1x4(data, 3, x3, vl); - unaligned_vssseg8e32_v_u32m1x8((void *)dst, data, vl); + unaligned_vssseg4e32_v_u32m1x4((void *)dst, data, vl); - data = unaligned_vlsseg8e32_v_u32m1x8((const void *)(src + 32), vl); + data = unaligned_vlsseg4e32_v_u32m1x4((const void *)(src + 16), vl); - data = vxor_v_u32m1_u32m1x8(data, 0, x8, vl); - data = vxor_v_u32m1_u32m1x8(data, 1, x9, vl); - data = vxor_v_u32m1_u32m1x8(data, 2, x10, vl); - data = vxor_v_u32m1_u32m1x8(data, 3, x11, vl); - data = vxor_v_u32m1_u32m1x8(data, 4, x12, vl); - data = vxor_v_u32m1_u32m1x8(data, 5, x13, vl); - data = vxor_v_u32m1_u32m1x8(data, 6, x14, vl); - data = vxor_v_u32m1_u32m1x8(data, 7, x15, vl); + data = vxor_v_u32m1_u32m1x4(data, 0, x4, vl); + data = vxor_v_u32m1_u32m1x4(data, 1, x5, vl); + data = vxor_v_u32m1_u32m1x4(data, 2, x6, vl); + data = vxor_v_u32m1_u32m1x4(data, 3, x7, vl); - unaligned_vssseg8e32_v_u32m1x8((void *)(dst + 32), data, vl); + unaligned_vssseg4e32_v_u32m1x4((void *)(dst + 16), data, vl); + + data = unaligned_vlsseg4e32_v_u32m1x4((const void *)(src + 32), vl); + + data = vxor_v_u32m1_u32m1x4(data, 0, x8, vl); + data = vxor_v_u32m1_u32m1x4(data, 1, x9, vl); + data = vxor_v_u32m1_u32m1x4(data, 2, x10, vl); + data = vxor_v_u32m1_u32m1x4(data, 3, x11, vl); + + unaligned_vssseg4e32_v_u32m1x4((void *)(dst + 32), data, vl); + + data = unaligned_vlsseg4e32_v_u32m1x4((const void *)(src + 48), vl); + + data = vxor_v_u32m1_u32m1x4(data, 0, x12, vl); + data = vxor_v_u32m1_u32m1x4(data, 1, x13, vl); + data = vxor_v_u32m1_u32m1x4(data, 2, x14, vl); + data = vxor_v_u32m1_u32m1x4(data, 3, x15, vl); + + unaligned_vssseg4e32_v_u32m1x4((void *)(dst + 48), data, vl); src += vl * 64; dst += vl * 64; -- 2.53.0 From jussi.kivilinna at iki.fi Sun Oct 11 19:23:13 2026 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sun, 11 Oct 2026 20:23:13 +0300 Subject: [PATCH 04/12] chacha20-riscv: use generic C for less than four remaining blocks In-Reply-To: <20261011172321.997183-1-jussi.kivilinna@iki.fi> References: <20261011172321.997183-1-jussi.kivilinna@iki.fi> Message-ID: <20261011172321.997183-4-jussi.kivilinna@iki.fi> * cipher/chacha20-riscv-rvv.h (WORD_ROL, QUARTERROUND4_2, QUARTERROUND4) (ADD_U64, unaligned_load_u32m1, unaligned_store_u32m1): Remove. (chacha20_rvv_blocks): Remove 1 to 3 block code-path; return number of processed blocks. * cipher/chacha20-riscv-v.c (_gcry_chacha20_riscv_v_blocks): Return number of processed blocks. * cipher/chacha20-riscv-zvkb.c (_gcry_chacha20_riscv_zvkb_blocks): Likewise. * cipher/chacha20.c (_gcry_chacha20_riscv_v_blocks) (_gcry_chacha20_riscv_zvkb_blocks): Change return type to size_t. (chacha20_blocks) [USE_RISCV_V, USE_RISCV_ZVKB]: Use vector implementation only for 4 or more blocks and process remaining blocks with generic C. -- Vector code-path for 1 to 3 blocks was slower than generic C on SpacemiT K3 X100 and A100 cores. Now remaining 1 to 3 blocks are processed with generic C. Cycles per call for N block message, before/after: blocks | K3 X100 | K3 A100 | K1 (no Zvkb) 1 | 1188 / 750 | 5250 / 1162 | 1667 / 1517 2 | 1354 / 1125 | 9374 / 1800 | 2156 / 2616 3 | 2229 / 1521 | 13949 / 2418 | 3422 / 3717 33 | 6458 / 6042 | 10349 / 6187 | 12311 / 12217 Signed-off-by: Jussi Kivilinna --- cipher/chacha20-riscv-rvv.h | 210 +---------------------------------- cipher/chacha20-riscv-v.c | 2 +- cipher/chacha20-riscv-zvkb.c | 2 +- cipher/chacha20.c | 32 ++++-- 4 files changed, 29 insertions(+), 217 deletions(-) diff --git a/cipher/chacha20-riscv-rvv.h b/cipher/chacha20-riscv-rvv.h index 267584a5..b0641418 100644 --- a/cipher/chacha20-riscv-rvv.h +++ b/cipher/chacha20-riscv-rvv.h @@ -46,7 +46,6 @@ #define XOR(v, w) __riscv_vxor_vv_u32m1((v), (w), vl) #define PLUS(v, w) __riscv_vadd_vv_u32m1((v), (w), vl) -#define WORD_ROL(v, c) __riscv_vrgather_vv_u32m1((v), (rol##c), vl) #define QUARTERROUND_4(a0, b0, c0, d0, a1, b1, c1, d1, \ a2, b2, c2, d2, a3, b3, c3, d3) \ @@ -75,39 +74,6 @@ b0 = ROTATE(b0, 7); b1 = ROTATE(b1, 7); \ b2 = ROTATE(b2, 7); b3 = ROTATE(b3, 7); -#define QUARTERROUND4_2(x0, x1, x2, x3, y0, y1, y2, y3, rol_x1, rol_x2, rol_x3) \ - x0 = PLUS(x0, x1); y0 = PLUS(y0, y1); \ - x3 = XOR(x3, x0); y3 = XOR(y3, y0); \ - x3 = ROTATE16(x3); y3 = ROTATE16(y3); \ - x2 = PLUS(x2, x3); y2 = PLUS(y2, y3); \ - x1 = XOR(x1, x2); y1 = XOR(y1, y2); \ - x1 = ROTATE(x1, 12); y1 = ROTATE(y1, 12); \ - x0 = PLUS(x0, x1); y0 = PLUS(y0, y1); \ - x3 = XOR(x3, x0); y3 = XOR(y3, y0); \ - x3 = ROTATE8(x3); y3 = ROTATE8(y3); \ - x2 = PLUS(x2, x3); y2 = PLUS(y2, y3); \ - x3 = WORD_ROL(x3, rol_x3); y3 = WORD_ROL(y3, rol_x3);\ - x1 = XOR(x1, x2); y1 = XOR(y1, y2); \ - x2 = WORD_ROL(x2, rol_x2); y2 = WORD_ROL(y2, rol_x2); \ - x1 = ROTATE(x1, 7); y1 = ROTATE(y1, 7); \ - x1 = WORD_ROL(x1, rol_x1); y1 = WORD_ROL(y1, rol_x1); - -#define QUARTERROUND4(x0, x1, x2, x3, rol_x1, rol_x2, rol_x3) \ - x0 = PLUS(x0, x1); x3 = XOR(x3, x0); x3 = ROTATE16(x3); \ - x2 = PLUS(x2, x3); x1 = XOR(x1, x2); x1 = ROTATE(x1, 12); \ - x0 = PLUS(x0, x1); x3 = XOR(x3, x0); x3 = ROTATE8(x3); \ - x2 = PLUS(x2, x3); \ - x3 = WORD_ROL(x3, rol_x3); \ - x1 = XOR(x1, x2); \ - x2 = WORD_ROL(x2, rol_x2); \ - x1= ROTATE(x1, 7); \ - x1 = WORD_ROL(x1, rol_x1); - -#define ADD_U64(a, b) __riscv_vreinterpret_v_u64m1_u32m1( \ - __riscv_vadd_vv_u64m1( \ - __riscv_vreinterpret_v_u32m1_u64m1(a), \ - __riscv_vreinterpret_v_u32m1_u64m1(b), vl / 2)) - #define vxor_v_u32m1_u32m1x4(data, idx, vs, vl) \ __riscv_vset_v_u32m1_u32m1x4((data), (idx), \ __riscv_vxor_vv_u32m1( \ @@ -147,26 +113,6 @@ gen_indexes(size_t vl, size_t stride) return __riscv_vadd_vv_u16m2(idx_hi, idx_lo, vl * 4); } -static ASM_FUNC_ATTR_INLINE vuint32m1_t -unaligned_load_u32m1(const void *src, size_t vl) -{ -#ifdef RVV_UNALIGNED_NOT_ALLOWED - return __riscv_vreinterpret_v_u8m1_u32m1(__riscv_vle8_v_u8m1(src, vl * 4)); -#else - return __riscv_vle32_v_u32m1(src, vl); -#endif -} - -static ASM_FUNC_ATTR_INLINE void -unaligned_store_u32m1(void *dst, vuint32m1_t vec, size_t vl) -{ -#ifdef RVV_UNALIGNED_NOT_ALLOWED - __riscv_vse8_v_u8m1(dst, __riscv_vreinterpret_v_u32m1_u8m1(vec), vl * 4); -#else - __riscv_vse32_v_u32m1(dst, vec, vl); -#endif -} - static ASM_FUNC_ATTR_INLINE vuint32m1x4_t unaligned_vlsseg4e32_v_u32m1x4(const void *src, size_t vl) { @@ -236,15 +182,14 @@ unaligned_vssseg4e32_v_u32m1x4(void *dst, vuint32m1x4_t data, size_t vl) } } -static ASM_FUNC_ATTR_INLINE unsigned int +/* Processes blocks while 4 or more remain. Returns number of blocks + * processed, remaining 0 to 3 blocks are left for the caller. */ +static ASM_FUNC_ATTR_INLINE size_t chacha20_rvv_blocks(u32 *input, byte *dst, const byte *src, size_t nblks) { + size_t nblks_orig = nblks; unsigned int i; - if (nblks == 0) - return 0; - - /* Try use vector implementation when there is 4 or more blocks. */ if (nblks >= 4) { size_t vl = __riscv_vsetvl_e32m1(nblks) < 4 @@ -385,154 +330,9 @@ chacha20_rvv_blocks(u32 *input, byte *dst, const byte *src, size_t nblks) input[13] = s13; } - /* Use SIMD implementation for remaining blocks. */ - if (nblks > 0) - { - static const u32 rol_const[3][4] = - { - { 1, 2, 3, 0 }, - { 2, 3, 0, 1 }, - { 3, 0, 1, 2 } - }; - static const u32 one_u64_const[4] = { 1, 0, 0, 0 }; - size_t vl = 4; - vuint32m1_t rol1, rol2, rol3; - vuint32m1_t one_u64; - vuint32m1_t v0, v1, v2, v3; - vuint32m1_t v4, v5, v6, v7; - vuint32m1_t state0, state1, state2, state3; - vuint32m1_t i0, i1, i2, i3; - vuint32m1_t i4, i5, i6, i7; -#ifdef LOAD_GATHER_ROTATE_CONSTANTS - vuint16m1_t rot16 = gen_rot16(vl); - vuint8m1_t rot8 = gen_rot8(vl); -#endif - - rol1 = __riscv_vle32_v_u32m1(rol_const[0], vl); - rol2 = __riscv_vle32_v_u32m1(rol_const[1], vl); - rol3 = __riscv_vle32_v_u32m1(rol_const[2], vl); - one_u64 = __riscv_vle32_v_u32m1(one_u64_const, vl); - - state0 = __riscv_vle32_v_u32m1(&input[0], vl); - state1 = __riscv_vle32_v_u32m1(&input[4], vl); - state2 = __riscv_vle32_v_u32m1(&input[8], vl); - state3 = __riscv_vle32_v_u32m1(&input[12], vl); - - input[12] += nblks; - input[13] += input[12] < nblks; - - /* SIMD 2x block implementation */ - while (nblks >= 2) - { - v0 = state0; - v1 = state1; - v2 = state2; - v3 = state3; - - v4 = state0; - v5 = state1; - v6 = state2; - v7 = state3; - v7 = ADD_U64(v7, one_u64); - - i0 = unaligned_load_u32m1(src + 0 * 16, vl); - i1 = unaligned_load_u32m1(src + 1 * 16, vl); - i2 = unaligned_load_u32m1(src + 2 * 16, vl); - i3 = unaligned_load_u32m1(src + 3 * 16, vl); - - for (i = 20; i > 0; i -= 2) - { - QUARTERROUND4_2(v0, v1, v2, v3, v4, v5, v6, v7, 1, 2, 3); - QUARTERROUND4_2(v0, v1, v2, v3, v4, v5, v6, v7, 3, 2, 1); - } - - v0 = __riscv_vadd_vv_u32m1(v0, state0, vl); - v1 = __riscv_vadd_vv_u32m1(v1, state1, vl); - v2 = __riscv_vadd_vv_u32m1(v2, state2, vl); - v3 = __riscv_vadd_vv_u32m1(v3, state3, vl); - state3 = ADD_U64(state3, one_u64); - - v0 = __riscv_vxor_vv_u32m1(i0, v0, vl); - v1 = __riscv_vxor_vv_u32m1(i1, v1, vl); - v2 = __riscv_vxor_vv_u32m1(i2, v2, vl); - v3 = __riscv_vxor_vv_u32m1(i3, v3, vl); - - v4 = __riscv_vadd_vv_u32m1(v4, state0, vl); - v5 = __riscv_vadd_vv_u32m1(v5, state1, vl); - v6 = __riscv_vadd_vv_u32m1(v6, state2, vl); - v7 = __riscv_vadd_vv_u32m1(v7, state3, vl); - state3 = ADD_U64(state3, one_u64); - - i4 = unaligned_load_u32m1(src + 4 * 16, vl); - i5 = unaligned_load_u32m1(src + 5 * 16, vl); - i6 = unaligned_load_u32m1(src + 6 * 16, vl); - i7 = unaligned_load_u32m1(src + 7 * 16, vl); - - unaligned_store_u32m1(dst + 0 * 16, v0, vl); - unaligned_store_u32m1(dst + 1 * 16, v1, vl); - unaligned_store_u32m1(dst + 2 * 16, v2, vl); - unaligned_store_u32m1(dst + 3 * 16, v3, vl); - - v4 = __riscv_vxor_vv_u32m1(i4, v4, vl); - v5 = __riscv_vxor_vv_u32m1(i5, v5, vl); - v6 = __riscv_vxor_vv_u32m1(i6, v6, vl); - v7 = __riscv_vxor_vv_u32m1(i7, v7, vl); - - unaligned_store_u32m1(dst + 4 * 16, v4, vl); - unaligned_store_u32m1(dst + 5 * 16, v5, vl); - unaligned_store_u32m1(dst + 6 * 16, v6, vl); - unaligned_store_u32m1(dst + 7 * 16, v7, vl); - - src += 2 * 64; - dst += 2 * 64; - - nblks -= 2; - } - - /* 1x block implementation */ - while (nblks) - { - v0 = state0; - v1 = state1; - v2 = state2; - v3 = state3; - - i0 = unaligned_load_u32m1(src + 0 * 16, vl); - i1 = unaligned_load_u32m1(src + 1 * 16, vl); - i2 = unaligned_load_u32m1(src + 2 * 16, vl); - i3 = unaligned_load_u32m1(src + 3 * 16, vl); - - for (i = 20; i > 0; i -= 2) - { - QUARTERROUND4(v0, v1, v2, v3, 1, 2, 3); - QUARTERROUND4(v0, v1, v2, v3, 3, 2, 1); - } - - v0 = __riscv_vadd_vv_u32m1(v0, state0, vl); - v1 = __riscv_vadd_vv_u32m1(v1, state1, vl); - v2 = __riscv_vadd_vv_u32m1(v2, state2, vl); - v3 = __riscv_vadd_vv_u32m1(v3, state3, vl); - - state3 = ADD_U64(state3, one_u64); - - v0 = __riscv_vxor_vv_u32m1(i0, v0, vl); - v1 = __riscv_vxor_vv_u32m1(i1, v1, vl); - v2 = __riscv_vxor_vv_u32m1(i2, v2, vl); - v3 = __riscv_vxor_vv_u32m1(i3, v3, vl); - unaligned_store_u32m1(dst + 0 * 16, v0, vl); - unaligned_store_u32m1(dst + 1 * 16, v1, vl); - unaligned_store_u32m1(dst + 2 * 16, v2, vl); - unaligned_store_u32m1(dst + 3 * 16, v3, vl); - src += 64; - dst += 64; - - nblks--; - } - } - clear_vec_regs(); - return 0; + return nblks_orig - nblks; } #endif /* GCRY_CHACHA20_RISCV_RVV_H */ diff --git a/cipher/chacha20-riscv-v.c b/cipher/chacha20-riscv-v.c index 32fc09cc..8cdd5c81 100644 --- a/cipher/chacha20-riscv-v.c +++ b/cipher/chacha20-riscv-v.c @@ -42,7 +42,7 @@ #include "chacha20-riscv-rvv.h" -unsigned int ASM_FUNC_ATTR FUNC_ATTR_OPT_O2 +size_t ASM_FUNC_ATTR FUNC_ATTR_OPT_O2 _gcry_chacha20_riscv_v_blocks(u32 *state, byte *dst, const byte *src, size_t nblks) { diff --git a/cipher/chacha20-riscv-zvkb.c b/cipher/chacha20-riscv-zvkb.c index abb34e4b..5b7aa829 100644 --- a/cipher/chacha20-riscv-zvkb.c +++ b/cipher/chacha20-riscv-zvkb.c @@ -33,7 +33,7 @@ #include "chacha20-riscv-rvv.h" -unsigned int ASM_FUNC_ATTR FUNC_ATTR_OPT_O2 +size_t ASM_FUNC_ATTR FUNC_ATTR_OPT_O2 _gcry_chacha20_riscv_zvkb_blocks(u32 *state, byte *dst, const byte *src, size_t nblks) { diff --git a/cipher/chacha20.c b/cipher/chacha20.c index 64946835..18d7e049 100644 --- a/cipher/chacha20.c +++ b/cipher/chacha20.c @@ -280,9 +280,9 @@ unsigned int _gcry_chacha20_poly1305_aarch64_blocks4( #ifdef USE_RISCV_V -unsigned int _gcry_chacha20_riscv_v_blocks(u32 *state, byte *dst, - const byte *src, - size_t nblks); +size_t _gcry_chacha20_riscv_v_blocks(u32 *state, byte *dst, + const byte *src, + size_t nblks); unsigned int _gcry_chacha20_riscv_v_check_hw(void); @@ -290,9 +290,9 @@ unsigned int _gcry_chacha20_riscv_v_check_hw(void); #ifdef USE_RISCV_ZVKB -unsigned int _gcry_chacha20_riscv_zvkb_blocks(u32 *state, byte *dst, - const byte *src, - size_t nblks); +size_t _gcry_chacha20_riscv_zvkb_blocks(u32 *state, byte *dst, + const byte *src, + size_t nblks); #endif /* USE_RISCV_ZVKB */ @@ -463,16 +463,28 @@ chacha20_blocks (CHACHA20_context_t *ctx, byte *dst, const byte *src, #endif #ifdef USE_RISCV_ZVKB - if (ctx->use_riscv_zvkb) + if (ctx->use_riscv_zvkb && nblks >= 4) { - return _gcry_chacha20_riscv_zvkb_blocks(ctx->input, dst, src, nblks); + size_t done = _gcry_chacha20_riscv_zvkb_blocks(ctx->input, dst, src, + nblks); + dst += CHACHA20_BLOCK_SIZE * done; + src += CHACHA20_BLOCK_SIZE * done; + nblks -= done; + if (nblks == 0) + return burn; } #endif #ifdef USE_RISCV_V - if (ctx->use_riscv_v) + if (ctx->use_riscv_v && nblks >= 4) { - return _gcry_chacha20_riscv_v_blocks(ctx->input, dst, src, nblks); + size_t done = _gcry_chacha20_riscv_v_blocks(ctx->input, dst, src, + nblks); + dst += CHACHA20_BLOCK_SIZE * done; + src += CHACHA20_BLOCK_SIZE * done; + nblks -= done; + if (nblks == 0) + return burn; } #endif -- 2.53.0 From jussi.kivilinna at iki.fi Sun Oct 11 19:23:15 2026 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sun, 11 Oct 2026 20:23:15 +0300 Subject: [PATCH 06/12] sha512-riscv-zvknhb-zvkb: use LMUL=1 with VLEN>=256 In-Reply-To: <20261011172321.997183-1-jussi.kivilinna@iki.fi> References: <20261011172321.997183-1-jussi.kivilinna@iki.fi> Message-ID: <20261011172321.997183-6-jussi.kivilinna@iki.fi> * cipher/Makefile.am (EXTRA_libcipher_la_SOURCES): Add 'sha512-riscv-zvknhb-zvkb.h' and 'sha512-riscv-zvknhb-zvkb-m1.c'. (riscv_vector_crypto_sha_cflags): Remove. (riscv_vector_crypto_sha256_cflags) (riscv_vector_crypto_sha512_cflags) (riscv_vector_crypto_sha512_m1_cflags): New. * cipher/sha512-riscv-zvknhb-zvkb.c (working_vsha2cl_vv_u64m2) (QUAD_ROUND_W_SCHED, QUAD_ROUND_NO_SCHED, load_and_swap) (sha512_transform_zvknhb_zvkb): Move to 'sha512-riscv-zvknhb-zvkb.h'. (_gcry_sha512_transform_riscv_zvknhb_zvkb): Use LMUL=1 variant when VLEN>=256. * cipher/sha512-riscv-zvknhb-zvkb-m1.c: New. * cipher/sha512-riscv-zvknhb-zvkb.h: New template for LMUL=1 and LMUL=2 transform functions; prepare K+W for next quad-round ahead of use. * configure.ac: Add 'sha512-riscv-zvknhb-zvkb-m1.lo'. (GCRY_RISCV_VECTOR_CRYPTO_INTRINSICS_TEST): Add LMUL=1 SHA-512 intrinsics to 'test_sha2'. (MARCH_RVA23U64_WITH_VEC_CRYPTO): Add 'zvl256b'. -- Previously LMUL=2 was used for all VLEN. With VLEN>=256, four 64-bit words fit in LMUL=1 register and on SpacemiT K3 some LMUL=2 operations cost more than LMUL=1 even with vl=4. clang requires zvl256b for LMUL=1 SHA-512 intrinsics, so LMUL=1 variant is placed in separate file compiled with zvl256b. LMUL=2 variant and code-path selection are compiled without zvl256b as they run on VLEN=128. Benchmark on SpacemiT K3, X100 core (out-of-order, VLEN=256): Before | cycles/byte SHA512 | 2.48 c/B After | cycles/byte SHA512 | 2.05 c/B ~1.21x faster Benchmark on SpacemiT K3, A100 core (in-order, VLEN=1024): Before | cycles/byte SHA512 | 9.58 c/B After | cycles/byte SHA512 | 7.34 c/B ~1.31x faster Signed-off-by: Jussi Kivilinna --- cipher/Makefile.am | 28 +++-- cipher/sha512-riscv-zvknhb-zvkb-m1.c | 60 +++++++++ cipher/sha512-riscv-zvknhb-zvkb.c | 153 +++------------------- cipher/sha512-riscv-zvknhb-zvkb.h | 181 +++++++++++++++++++++++++++ configure.ac | 12 +- 5 files changed, 293 insertions(+), 141 deletions(-) create mode 100644 cipher/sha512-riscv-zvknhb-zvkb-m1.c create mode 100644 cipher/sha512-riscv-zvknhb-zvkb.h diff --git a/cipher/Makefile.am b/cipher/Makefile.am index b685a5f7..6c51ee94 100644 --- a/cipher/Makefile.am +++ b/cipher/Makefile.am @@ -154,7 +154,9 @@ EXTRA_libcipher_la_SOURCES = \ sha512-avx2-bmi2-amd64.S sha512-avx512-amd64.S \ sha512-intel-shaext.c \ sha512-armv7-neon.S sha512-armv8-aarch64-ce.S sha512-arm.S \ - sha512-ppc.c sha512-riscv-zvknhb-zvkb.c sha512-ssse3-i386.c \ + sha512-ppc.c sha512-riscv-zvknhb-zvkb.c sha512-riscv-zvknhb-zvkb.h \ + sha512-riscv-zvknhb-zvkb-m1.c \ + sha512-ssse3-i386.c \ sm3.c sm3-avx-bmi2-amd64.S sm3-intel-avx2-amd64.S sm3-aarch64.S \ sm3-armv8-aarch64-ce.S \ keccak.c keccak_permute_32.h keccak_permute_64.h \ @@ -466,12 +468,16 @@ rijndael-vp-riscv.lo: $(srcdir)/rijndael-vp-riscv.c Makefile if ENABLE_RISCV_VECTOR_CRYPTO_INTRINSICS_EXTRA_CFLAGS riscv_vector_crypto_zvkb_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvkb riscv_vector_crypto_aes_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvkned_zvkb -riscv_vector_crypto_sha_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvknha_zvknhb_zvkb +riscv_vector_crypto_sha256_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvknha_zvkb +riscv_vector_crypto_sha512_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvknhb_zvkb +riscv_vector_crypto_sha512_m1_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvknhb_zvkb_zvl256b riscv_vector_crypto_gcm_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvkg_zvkb else riscv_vector_crypto_zvkb_cflags = riscv_vector_crypto_aes_cflags = -riscv_vector_crypto_sha_cflags = +riscv_vector_crypto_sha256_cflags = +riscv_vector_crypto_sha512_cflags = +riscv_vector_crypto_sha512_m1_cflags = riscv_vector_crypto_gcm_cflags = endif @@ -500,17 +506,25 @@ rijndael-riscv-zvkned.lo: $(srcdir)/rijndael-riscv-zvkned.c Makefile $(custom_depfile_lo) sha256-riscv-zvknha-zvkb.o: $(srcdir)/sha256-riscv-zvknha-zvkb.c Makefile - `echo $(COMPILE) $(riscv_vector_crypto_sha_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(riscv_vector_crypto_sha256_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` $(custom_depfile_o) sha256-riscv-zvknha-zvkb.lo: $(srcdir)/sha256-riscv-zvknha-zvkb.c Makefile - `echo $(LTCOMPILE) $(riscv_vector_crypto_sha_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(riscv_vector_crypto_sha256_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` $(custom_depfile_lo) sha512-riscv-zvknhb-zvkb.o: $(srcdir)/sha512-riscv-zvknhb-zvkb.c Makefile - `echo $(COMPILE) $(riscv_vector_crypto_sha_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + `echo $(COMPILE) $(riscv_vector_crypto_sha512_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` $(custom_depfile_o) sha512-riscv-zvknhb-zvkb.lo: $(srcdir)/sha512-riscv-zvknhb-zvkb.c Makefile - `echo $(LTCOMPILE) $(riscv_vector_crypto_sha_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + `echo $(LTCOMPILE) $(riscv_vector_crypto_sha512_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) + +sha512-riscv-zvknhb-zvkb-m1.o: $(srcdir)/sha512-riscv-zvknhb-zvkb-m1.c Makefile + `echo $(COMPILE) $(riscv_vector_crypto_sha512_m1_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) + +sha512-riscv-zvknhb-zvkb-m1.lo: $(srcdir)/sha512-riscv-zvknhb-zvkb-m1.c Makefile + `echo $(LTCOMPILE) $(riscv_vector_crypto_sha512_m1_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` $(custom_depfile_lo) diff --git a/cipher/sha512-riscv-zvknhb-zvkb-m1.c b/cipher/sha512-riscv-zvknhb-zvkb-m1.c new file mode 100644 index 00000000..be1e984a --- /dev/null +++ b/cipher/sha512-riscv-zvknhb-zvkb-m1.c @@ -0,0 +1,60 @@ +/* sha512-riscv-zvknhb-zvkb-m1.c - RISC-V vector SHA-512, VLEN>=256 variant + * Copyright (C) 2026 Jussi Kivilinna + * + * This file is part of Libgcrypt. + * + * Libgcrypt is free software; you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as + * published by the Free Software Foundation; either version 2.1 of + * the License, or (at your option) any later version. + * + * Libgcrypt is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this program; if not, see . + */ + +#include + +#if defined (__riscv) && \ + defined(HAVE_COMPATIBLE_CC_RISCV_VECTOR_INTRINSICS) && \ + defined(HAVE_COMPATIBLE_CC_RISCV_VECTOR_CRYPTO_INTRINSICS) && \ + defined(USE_SHA512) + +#include "g10lib.h" +#include "simd-common-riscv.h" +#include + + +#define ALWAYS_INLINE inline __attribute__((always_inline)) +#define NO_INLINE __attribute__((noinline)) +#define NO_INSTRUMENT_FUNCTION __attribute__((no_instrument_function)) + +#define ASM_FUNC_ATTR NO_INSTRUMENT_FUNCTION +#define ASM_FUNC_ATTR_INLINE ASM_FUNC_ATTR ALWAYS_INLINE +#define ASM_FUNC_ATTR_NOINLINE ASM_FUNC_ATTR NO_INLINE + +#ifdef HAVE_GCC_ATTRIBUTE_OPTIMIZE +# define FUNC_ATTR_OPT_O2 __attribute__((optimize("-O2"))) +#else +# define FUNC_ATTR_OPT_O2 +#endif + + +#define SHA512_LMUL m1 +#define SHA512_MASK_T vbool64_t +#include "sha512-riscv-zvknhb-zvkb.h" + +unsigned int +_gcry_sha512_transform_riscv_zvknhb_zvkb_m1(u64 state[8], + const unsigned char *input_data, + size_t num_blks, + const u64 k[80]) +{ + return sha512_transform_zvknhb_zvkb_m1(state, input_data, num_blks, k); +} + +#endif /* HAVE_COMPATIBLE_CC_RISCV_VECTOR_INTRINSICS */ diff --git a/cipher/sha512-riscv-zvknhb-zvkb.c b/cipher/sha512-riscv-zvknhb-zvkb.c index fea8f32a..dd316b42 100644 --- a/cipher/sha512-riscv-zvknhb-zvkb.c +++ b/cipher/sha512-riscv-zvknhb-zvkb.c @@ -1,5 +1,5 @@ /* sha512-riscv-zvknhb-zvkb.c - RISC-V vector crypto implementation of SHA-512 - * Copyright (C) 2025 Jussi Kivilinna + * Copyright (C) 2025-2026 Jussi Kivilinna * * This file is part of Libgcrypt. * @@ -37,149 +37,36 @@ #define ASM_FUNC_ATTR_INLINE ASM_FUNC_ATTR ALWAYS_INLINE #define ASM_FUNC_ATTR_NOINLINE ASM_FUNC_ATTR NO_INLINE - -static ASM_FUNC_ATTR_INLINE vuint64m2_t -working_vsha2cl_vv_u64m2(vuint64m2_t hgcd, vuint64m2_t feba, - vuint64m2_t kw, size_t vl) -{ -#ifdef HAVE_BROKEN_VSHA2CL_INTRINSIC - asm ( - "vsetvli zero,%3,e64,m2,ta,ma;\n\t" - "vsha2cl.vv %0,%1,%2;\n\t" - : "+vr" (hgcd) - : "vr" (feba), "vr" (kw), "r" (vl) - : "vl", "vtype" - ); - return hgcd; -#else - return __riscv_vsha2cl_vv_u64m2(hgcd, feba, kw, vl); -#endif -} - - -/* Quad-round with message expansion (rounds 0-63) */ -#define QUAD_ROUND_W_SCHED(w0, w1, w2, w3) \ - k_tmp = k; \ - asm ("" : "+r" (k_tmp) :: "memory"); \ - v_k = __riscv_vle64_v_u64m2(k_tmp, vl); \ - k += 4; \ - v_kw = __riscv_vadd_vv_u64m2(v_k, w0, vl); \ - v_hgcd_work = working_vsha2cl_vv_u64m2(v_hgcd_work, v_feba_work, v_kw, vl); \ - v_feba_work = __riscv_vsha2ch_vv_u64m2(v_feba_work, v_hgcd_work, v_kw, vl); \ - v_w_merged = __riscv_vmerge_vvm_u64m2(w2, w1, merge_mask, vl); \ - w0 = __riscv_vsha2ms_vv_u64m2(w0, v_w_merged, w3, vl); - -/* Quad-round without message expansion (rounds 64-79) */ -#define QUAD_ROUND_NO_SCHED(w0) \ - k_tmp = k; \ - asm ("" : "+r" (k_tmp) :: "memory"); \ - v_k = __riscv_vle64_v_u64m2(k_tmp, vl); \ - k += 4; \ - v_kw = __riscv_vadd_vv_u64m2(v_k, w0, vl); \ - v_hgcd_work = working_vsha2cl_vv_u64m2(v_hgcd_work, v_feba_work, v_kw, vl); \ - v_feba_work = __riscv_vsha2ch_vv_u64m2(v_feba_work, v_hgcd_work, v_kw, vl); - - -static ASM_FUNC_ATTR_INLINE vuint64m2_t -load_and_swap(const byte *p, size_t vl) -{ -#ifdef RVV_UNALIGNED_NOT_ALLOWED - vuint8m2_t temp = __riscv_vle8_v_u8m2(p, vl * 8); - return __riscv_vrev8_v_u64m2(__riscv_vreinterpret_v_u8m2_u64m2(temp), vl); -#else - return __riscv_vrev8_v_u64m2(__riscv_vle64_v_u64m2((const void *)p, vl), vl); -#endif -} - - -static ASM_FUNC_ATTR_INLINE void -sha512_transform_zvknhb_zvkb (u64 state[8], const byte *data, - size_t nblocks, const u64 k_const[80]) -{ - static const u64 feba_hgcd_indices[4] = { 40, 32, 8, 0 }; - static const int feba_offset = 0; - static const int hgcd_offset = 16 / sizeof(u64); - size_t vl; - vuint64m2_t idx; - vuint64m2_t v_feba_work, v_feba; - vuint64m2_t v_hgcd_work, v_hgcd; - vuint64m2_t w0, w1, w2, w3; - vuint64m2_t v_k, v_kw, v_w_merged; - vbool32_t merge_mask; - vuint64m2_t v_feba_hgcd_idx; - - vl = 4; - idx = __riscv_vid_v_u64m2(vl); - merge_mask = __riscv_vmseq_vx_u64m2_b32(idx, 0, vl); - - v_feba_hgcd_idx = __riscv_vle64_v_u64m2(feba_hgcd_indices, vl); - - v_feba = __riscv_vluxei64_v_u64m2(state + feba_offset, v_feba_hgcd_idx, vl); - v_hgcd = __riscv_vluxei64_v_u64m2(state + hgcd_offset, v_feba_hgcd_idx, vl); - - while (nblocks > 0) - { - const u64 *k = k_const; - const u64 *k_tmp; - - v_feba_work = v_feba; - v_hgcd_work = v_hgcd; - - w0 = load_and_swap(data + 0, vl); - w1 = load_and_swap(data + 32, vl); - w2 = load_and_swap(data + 64, vl); - w3 = load_and_swap(data + 96, vl); - - QUAD_ROUND_W_SCHED(w0, w1, w2, w3); - QUAD_ROUND_W_SCHED(w1, w2, w3, w0); - QUAD_ROUND_W_SCHED(w2, w3, w0, w1); - QUAD_ROUND_W_SCHED(w3, w0, w1, w2); - QUAD_ROUND_W_SCHED(w0, w1, w2, w3); - QUAD_ROUND_W_SCHED(w1, w2, w3, w0); - QUAD_ROUND_W_SCHED(w2, w3, w0, w1); - QUAD_ROUND_W_SCHED(w3, w0, w1, w2); - QUAD_ROUND_W_SCHED(w0, w1, w2, w3); - QUAD_ROUND_W_SCHED(w1, w2, w3, w0); - QUAD_ROUND_W_SCHED(w2, w3, w0, w1); - QUAD_ROUND_W_SCHED(w3, w0, w1, w2); - QUAD_ROUND_W_SCHED(w0, w1, w2, w3); - QUAD_ROUND_W_SCHED(w1, w2, w3, w0); - QUAD_ROUND_W_SCHED(w2, w3, w0, w1); - QUAD_ROUND_W_SCHED(w3, w0, w1, w2); - - QUAD_ROUND_NO_SCHED(w0); - QUAD_ROUND_NO_SCHED(w1); - QUAD_ROUND_NO_SCHED(w2); - QUAD_ROUND_NO_SCHED(w3); - - v_feba = __riscv_vadd_vv_u64m2(v_feba, v_feba_work, vl); - v_hgcd = __riscv_vadd_vv_u64m2(v_hgcd, v_hgcd_work, vl); - - data += 128; - nblocks--; - } - - __riscv_vsuxei64_v_u64m2(state + feba_offset, v_feba_hgcd_idx, v_feba, vl); - __riscv_vsuxei64_v_u64m2(state + hgcd_offset, v_feba_hgcd_idx, v_hgcd, vl); - - clear_vec_regs(); -} - - #ifdef HAVE_GCC_ATTRIBUTE_OPTIMIZE # define FUNC_ATTR_OPT_O2 __attribute__((optimize("-O2"))) #else # define FUNC_ATTR_OPT_O2 #endif -unsigned int ASM_FUNC_ATTR FUNC_ATTR_OPT_O2 + +/* m2 variant for VLEN == 128 */ +#define SHA512_LMUL m2 +#define SHA512_MASK_T vbool32_t +#include "sha512-riscv-zvknhb-zvkb.h" + +/* m1 variant for VLEN >= 256, compiled separately with zvl256b */ +extern unsigned int +_gcry_sha512_transform_riscv_zvknhb_zvkb_m1(u64 state[8], + const unsigned char *input_data, + size_t num_blks, + const u64 k[80]); + +unsigned int _gcry_sha512_transform_riscv_zvknhb_zvkb(u64 state[8], const unsigned char *input_data, size_t num_blks, const u64 k[80]) { - sha512_transform_zvknhb_zvkb(state, input_data, num_blks, k); - return 0; + if (__riscv_vsetvlmax_e8m1() >= 32) + return _gcry_sha512_transform_riscv_zvknhb_zvkb_m1(state, input_data, + num_blks, k); + else + return sha512_transform_zvknhb_zvkb_m2(state, input_data, num_blks, k); } unsigned int ASM_FUNC_ATTR FUNC_ATTR_OPT_O2 diff --git a/cipher/sha512-riscv-zvknhb-zvkb.h b/cipher/sha512-riscv-zvknhb-zvkb.h new file mode 100644 index 00000000..e5a0b3a2 --- /dev/null +++ b/cipher/sha512-riscv-zvknhb-zvkb.h @@ -0,0 +1,181 @@ +/* sha512-riscv-zvknhb-zvkb.h - RISC-V vector crypto SHA-512 LMUL template + * Copyright (C) 2025-2026 Jussi Kivilinna + * + * This file is part of Libgcrypt. + * + * Libgcrypt is free software; you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as + * published by the Free Software Foundation; either version 2.1 of + * the License, or (at your option) any later version. + * + * Libgcrypt is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this program; if not, see . + */ + +/* Included once for each LMUL with SHA512_LMUL and SHA512_MASK_T set. */ + +#define CAT_(a, b) a##b +#define CAT(a, b) CAT_(a, b) + +#define FN(name) CAT(name##_, SHA512_LMUL) +#define V(name) CAT(__riscv_##name##_u64, SHA512_LMUL) +#define V8(name) CAT(__riscv_##name##_u8, SHA512_LMUL) +#define VEC_T CAT(CAT(vuint64, SHA512_LMUL), _t) +#define MASK_T SHA512_MASK_T + + +#ifdef HAVE_BROKEN_VSHA2CL_INTRINSIC +static ASM_FUNC_ATTR_INLINE VEC_T +FN(asm_vsha2cl) (VEC_T vd, VEC_T vs2, VEC_T vs1, size_t vl) +{ + asm ("vsetvli zero,%3,e64," STR2(SHA512_LMUL) ",ta,ma;\n\t" + "vsha2cl.vv %0,%1,%2;\n\t" + : "+vr" (vd) + : "vr" (vs2), "vr" (vs1), "r" (vl) + : "vl", "vtype"); + return vd; +} +# define VSHA2CL FN(asm_vsha2cl) +#else +# define VSHA2CL __riscv_vsha2cl +#endif + + +#define LOAD_K() \ + k_tmp = k; \ + asm ("" : "+r" (k_tmp) :: "memory"); \ + v_k = V(vle64_v)(k_tmp, vl); \ + k += 4; + +/* K+W for next quad-round and K for the one after that are prepared + * between vsha2cl and vsha2ch, ahead of use. */ + +/* Quad-round with message expansion (rounds 0-63) */ +#define QUAD_ROUND_W_SCHED(kw, kw_next, w0, w1, w2, w3) \ + v_hgcd_work = VSHA2CL(v_hgcd_work, v_feba_work, kw, vl); \ + kw_next = __riscv_vadd(v_k, w1, vl); \ + LOAD_K(); \ + v_feba_work = __riscv_vsha2ch(v_feba_work, v_hgcd_work, kw, vl); \ + v_w_merged = __riscv_vmerge(w2, w1, merge_mask, vl); \ + w0 = __riscv_vsha2ms(w0, v_w_merged, w3, vl); + +/* Quad-round without message expansion (rounds 64-75) */ +#define QUAD_ROUND_NO_SCHED(kw, kw_next, w_next, load_k) \ + v_hgcd_work = VSHA2CL(v_hgcd_work, v_feba_work, kw, vl); \ + kw_next = __riscv_vadd(v_k, w_next, vl); \ + load_k; \ + v_feba_work = __riscv_vsha2ch(v_feba_work, v_hgcd_work, kw, vl); + +/* Last quad-round (rounds 76-79) */ +#define QUAD_ROUND_LAST(kw) \ + v_hgcd_work = VSHA2CL(v_hgcd_work, v_feba_work, kw, vl); \ + v_feba_work = __riscv_vsha2ch(v_feba_work, v_hgcd_work, kw, vl); + + +static ASM_FUNC_ATTR_INLINE VEC_T +FN(load_and_swap) (const byte *p, size_t vl) +{ +#ifdef RVV_UNALIGNED_NOT_ALLOWED + return __riscv_vrev8(V(vreinterpret)(V8(vle8_v)(p, vl * 8)), vl); +#else + return __riscv_vrev8(V(vle64_v)((const void *)p, vl), vl); +#endif +} + +static ASM_FUNC_ATTR_NOINLINE FUNC_ATTR_OPT_O2 unsigned int +FN(sha512_transform_zvknhb_zvkb) (u64 state[8], const byte *data, + size_t nblocks, const u64 k_const[80]) +{ + static const u64 feba_hgcd_indices[4] = { 40, 32, 8, 0 }; + static const int feba_offset = 0; + static const int hgcd_offset = 16 / sizeof(u64); + size_t vl; + VEC_T idx; + VEC_T v_feba_work, v_feba; + VEC_T v_hgcd_work, v_hgcd; + VEC_T w0, w1, w2, w3; + VEC_T v_k, v_kw0, v_kw1, v_w_merged; + MASK_T merge_mask; + VEC_T v_feba_hgcd_idx; + + vl = 4; + idx = V(vid_v)(vl); + merge_mask = __riscv_vmseq(idx, 0, vl); + + v_feba_hgcd_idx = V(vle64_v)(feba_hgcd_indices, vl); + + v_feba = V(vluxei64_v)(state + feba_offset, v_feba_hgcd_idx, vl); + v_hgcd = V(vluxei64_v)(state + hgcd_offset, v_feba_hgcd_idx, vl); + + while (nblocks > 0) + { + const u64 *k = k_const; + const u64 *k_tmp; + + v_feba_work = v_feba; + v_hgcd_work = v_hgcd; + + w0 = FN(load_and_swap)(data + 0, vl); + w1 = FN(load_and_swap)(data + 32, vl); + w2 = FN(load_and_swap)(data + 64, vl); + w3 = FN(load_and_swap)(data + 96, vl); + + LOAD_K(); + v_kw0 = __riscv_vadd(v_k, w0, vl); + LOAD_K(); + + QUAD_ROUND_W_SCHED(v_kw0, v_kw1, w0, w1, w2, w3); + QUAD_ROUND_W_SCHED(v_kw1, v_kw0, w1, w2, w3, w0); + QUAD_ROUND_W_SCHED(v_kw0, v_kw1, w2, w3, w0, w1); + QUAD_ROUND_W_SCHED(v_kw1, v_kw0, w3, w0, w1, w2); + QUAD_ROUND_W_SCHED(v_kw0, v_kw1, w0, w1, w2, w3); + QUAD_ROUND_W_SCHED(v_kw1, v_kw0, w1, w2, w3, w0); + QUAD_ROUND_W_SCHED(v_kw0, v_kw1, w2, w3, w0, w1); + QUAD_ROUND_W_SCHED(v_kw1, v_kw0, w3, w0, w1, w2); + QUAD_ROUND_W_SCHED(v_kw0, v_kw1, w0, w1, w2, w3); + QUAD_ROUND_W_SCHED(v_kw1, v_kw0, w1, w2, w3, w0); + QUAD_ROUND_W_SCHED(v_kw0, v_kw1, w2, w3, w0, w1); + QUAD_ROUND_W_SCHED(v_kw1, v_kw0, w3, w0, w1, w2); + QUAD_ROUND_W_SCHED(v_kw0, v_kw1, w0, w1, w2, w3); + QUAD_ROUND_W_SCHED(v_kw1, v_kw0, w1, w2, w3, w0); + QUAD_ROUND_W_SCHED(v_kw0, v_kw1, w2, w3, w0, w1); + QUAD_ROUND_W_SCHED(v_kw1, v_kw0, w3, w0, w1, w2); + + QUAD_ROUND_NO_SCHED(v_kw0, v_kw1, w1, LOAD_K()); + QUAD_ROUND_NO_SCHED(v_kw1, v_kw0, w2, LOAD_K()); + QUAD_ROUND_NO_SCHED(v_kw0, v_kw1, w3, (void)0); + QUAD_ROUND_LAST(v_kw1); + + v_feba = __riscv_vadd(v_feba, v_feba_work, vl); + v_hgcd = __riscv_vadd(v_hgcd, v_hgcd_work, vl); + + data += 128; + nblocks--; + } + + __riscv_vsuxei64(state + feba_offset, v_feba_hgcd_idx, v_feba, vl); + __riscv_vsuxei64(state + hgcd_offset, v_feba_hgcd_idx, v_hgcd, vl); + + clear_vec_regs(); + return 0; +} + +#undef QUAD_ROUND_LAST +#undef QUAD_ROUND_NO_SCHED +#undef QUAD_ROUND_W_SCHED +#undef LOAD_K +#undef VSHA2CL +#undef MASK_T +#undef VEC_T +#undef V8 +#undef V +#undef FN +#undef CAT +#undef CAT_ +#undef SHA512_MASK_T +#undef SHA512_LMUL diff --git a/configure.ac b/configure.ac index 2a126041..e32519c9 100644 --- a/configure.ac +++ b/configure.ac @@ -2983,6 +2983,7 @@ m4_define([GCRY_RISCV_VECTOR_CRYPTO_INTRINSICS_TEST], vuint32m1_t m; vuint32m1_t idx = __riscv_vid_v_u32m1 (vl); vbool32_t merge_mask = __riscv_vmseq_vx_u32m1_b32 (idx, 0, vl); + vuint64m1_t c, d, w4; a = __riscv_vsha2cl_vv_u32m1(a, b, w0, vl); b = __riscv_vsha2ch_vv_u32m1(a, b, w0, vl); m = __riscv_vmerge_vvm_u32m1(w2, w1, merge_mask, vl); @@ -2993,6 +2994,14 @@ m4_define([GCRY_RISCV_VECTOR_CRYPTO_INTRINSICS_TEST], __riscv_vse32_v_u32m1(ptr + 3 * vl, w1, vl); __riscv_vse32_v_u32m1(ptr + 4 * vl, w2, vl); __riscv_vse32_v_u32m1(ptr + 5 * vl, w3, vl); + vl = __riscv_vsetvl_e64m1 (4); + c = __riscv_vle64_v_u64m1((void *)(ptr + 0 * vl), vl); + d = __riscv_vle64_v_u64m1((void *)(ptr + 2 * vl), vl); + w4 = __riscv_vle64_v_u64m1((void *)(ptr + 4 * vl), vl); + c = __riscv_vsha2cl_vv_u64m1(c, d, w4, vl); + d = __riscv_vsha2ch_vv_u64m1(c, d, w4, vl); + __riscv_vse64_v_u64m1((void *)(ptr + 0 * vl), c, vl); + __riscv_vse64_v_u64m1((void *)(ptr + 2 * vl), d, vl); } void test_aes_key(unsigned int *ptr) { @@ -3100,7 +3109,7 @@ if test "$gcry_cv_cc_riscv_zicclsm_enabled" = "yes" ; then else MARCH_RVA23U64_BASE=${MARCH_RVA22U64_WITH_VEC} fi -MARCH_RVA23U64_WITH_VEC_CRYPTO=${MARCH_RVA23U64_BASE}_zvbc_zvkb_zvkg_zvkn_zvks +MARCH_RVA23U64_WITH_VEC_CRYPTO=${MARCH_RVA23U64_BASE}_zvbc_zvkb_zvkg_zvkn_zvks_zvl256b AC_SUBST([MARCH_RVA23U64_BASE]) CFLAGS="$CFLAGS -O2 -march=$MARCH_RVA23U64_WITH_VEC_CRYPTO" @@ -4178,6 +4187,7 @@ if test "$found" = "1" ; then riscv64-*-*) # Build with the RISC-V vector cryptography implementation GCRYPT_ASM_DIGESTS="$GCRYPT_ASM_DIGESTS sha512-riscv-zvknhb-zvkb.lo" + GCRYPT_ASM_DIGESTS="$GCRYPT_ASM_DIGESTS sha512-riscv-zvknhb-zvkb-m1.lo" ;; esac -- 2.53.0 From jussi.kivilinna at iki.fi Sun Oct 11 19:23:14 2026 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sun, 11 Oct 2026 20:23:14 +0300 Subject: [PATCH 05/12] poly1305: add RISC-V vector intrinsics implementation In-Reply-To: <20261011172321.997183-1-jussi.kivilinna@iki.fi> References: <20261011172321.997183-1-jussi.kivilinna@iki.fi> Message-ID: <20261011172321.997183-5-jussi.kivilinna@iki.fi> * LICENSES: Add 'cipher/poly1305-riscv-v.c'. * cipher/Makefile.am: Add 'poly1305-riscv-v.c' and handling for 'poly1305-riscv-v.o' and 'poly1305-riscv-v.lo'. * cipher/poly1305-internal.h (POLY1305_USE_RISCV_V): New. (poly1305_context_s) [POLY1305_USE_RISCV_V]: Add 'use_riscv_v', 'riscv_v_npowers' and 'riscv_v_powers'. * cipher/poly1305-riscv-v.c: New. * cipher/poly1305.c [POLY1305_USE_RISCV_V] (_gcry_poly1305_riscv_v_blocks) (_gcry_poly1305_riscv_v_check_hw): New. (poly1305_init) [POLY1305_USE_RISCV_V]: Add HW feature detection for RISC-V vector implementation. (poly1305_blocks) [POLY1305_USE_RISCV_V]: Add RISC-V vector code path. * configure.ac: Add 'poly1305-riscv-v.lo'. -- Patch adds RISC-V vector implementation of Poly1305, based on 'poly1305_blocks_vx' by Andy Polyakov from CRYPTOGAMS distribution 'riscv/poly1305-riscv.pl', converted to RVV intrinsics. Vector implementation is used for input of 1024 bytes or more. Benchmarks were run with '--buf-start 2048' to measure only vector code path. Benchmark on SpacemiT K3, X100 core (2000 Mhz, out-of-order, VLEN=256): Before: | nanosecs/byte mebibytes/sec cycles/byte POLY1305_AES | 0.750 ns/B 1271 MiB/s 1.50 c/B After (~1.29x faster): | nanosecs/byte mebibytes/sec cycles/byte POLY1305_AES | 0.579 ns/B 1646 MiB/s 1.16 c/B Benchmark on SpacemiT K3, A100 core (1800 Mhz, in-order, VLEN=1024): Before: | nanosecs/byte mebibytes/sec cycles/byte POLY1305_AES | 2.22 ns/B 428.9 MiB/s 4.00 c/B After (~1.75x faster): | nanosecs/byte mebibytes/sec cycles/byte POLY1305_AES | 1.27 ns/B 753.2 MiB/s 2.28 c/B Signed-off-by: Jussi Kivilinna --- LICENSES | 1 + cipher/Makefile.am | 10 +- cipher/poly1305-internal.h | 12 + cipher/poly1305-riscv-v.c | 565 +++++++++++++++++++++++++++++++++++++ cipher/poly1305.c | 36 +++ configure.ac | 3 + 6 files changed, 626 insertions(+), 1 deletion(-) create mode 100644 cipher/poly1305-riscv-v.c diff --git a/LICENSES b/LICENSES index e61dca9a..ec74dde8 100644 --- a/LICENSES +++ b/LICENSES @@ -170,6 +170,7 @@ with any binary distributions derived from the GNU C Library. For files: - cipher/cipher-gcm-ppc.c - cipher/keccak-amd64-avx512.S + - cipher/poly1305-riscv-v.c #+begin_quote Copyright (c) 2006, CRYPTOGAMS by diff --git a/cipher/Makefile.am b/cipher/Makefile.am index a3d012ba..b685a5f7 100644 --- a/cipher/Makefile.am +++ b/cipher/Makefile.am @@ -112,7 +112,7 @@ EXTRA_libcipher_la_SOURCES = \ md4.c \ md5.c \ poly1305-s390x.S poly1305-amd64-avx512.S \ - poly1305-p10le.s \ + poly1305-p10le.s poly1305-riscv-v.c \ rijndael.c rijndael-internal.h rijndael-tables.h \ rijndael-aesni.c rijndael-padlock.c \ rijndael-amd64.S rijndael-arm.S \ @@ -447,6 +447,14 @@ chacha20-riscv-v.lo: $(srcdir)/chacha20-riscv-v.c Makefile `echo $(LTCOMPILE) $(riscv_vector_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` $(custom_depfile_lo) +poly1305-riscv-v.o: $(srcdir)/poly1305-riscv-v.c Makefile + `echo $(COMPILE) $(riscv_vector_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) + +poly1305-riscv-v.lo: $(srcdir)/poly1305-riscv-v.c Makefile + `echo $(LTCOMPILE) $(riscv_vector_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) + rijndael-vp-riscv.o: $(srcdir)/rijndael-vp-riscv.c Makefile `echo $(COMPILE) $(riscv_vector_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` $(custom_depfile_o) diff --git a/cipher/poly1305-internal.h b/cipher/poly1305-internal.h index 809a2850..bbbecefe 100644 --- a/cipher/poly1305-internal.h +++ b/cipher/poly1305-internal.h @@ -55,6 +55,13 @@ # endif #endif +/* POLY1305_USE_RISCV_V indicates whether to enable RISC-V vector code. */ +#undef POLY1305_USE_RISCV_V +#if defined (__riscv) && defined(__riscv_xlen) && __riscv_xlen == 64 && \ + defined(HAVE_COMPATIBLE_CC_RISCV_VECTOR_INTRINSICS) +# define POLY1305_USE_RISCV_V 1 +#endif + typedef struct { @@ -74,6 +81,11 @@ typedef struct poly1305_context_s #ifdef POLY1305_USE_PPC_VEC unsigned int use_p10:1; #endif +#ifdef POLY1305_USE_RISCV_V + unsigned int use_riscv_v:1; + unsigned int riscv_v_npowers; + u32 riscv_v_powers[5][6]; +#endif } poly1305_context_t; diff --git a/cipher/poly1305-riscv-v.c b/cipher/poly1305-riscv-v.c new file mode 100644 index 00000000..08c5cb07 --- /dev/null +++ b/cipher/poly1305-riscv-v.c @@ -0,0 +1,565 @@ +/* poly1305-riscv-v.c - RISC-V vector implementation of Poly1305 + * + * Copyright (C) 2026 Jussi Kivilinna + * + * This file is part of Libgcrypt. + * + * Libgcrypt is free software; you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as + * published by the Free Software Foundation; either version 2.1 of + * the License, or (at your option) any later version. + * + * Libgcrypt is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this program; if not, see . + * + * --- + * + * Core function `_gcry_poly1305_riscv_v_blocks` based on RISC-V vector + * Poly1305 implementation `poly1305_blocks_vx` by Andy Polyakov from + * CRYPTOGAMS distribution `riscv/poly1305-riscv.pl`. `poly1305_blocks_vx` + * was converted to RVV intrinsics and fitted to libgcrypt Poly1305 state + * and key power table. Key power precomputation was rewritten in C using + * base 2^26 arithmetic. + * + * Original copyright license follows: + * + * Copyright (c) 2006, CRYPTOGAMS by + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain copyright notices, + * this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * * Neither the name of the CRYPTOGAMS nor the names of its + * copyright holder and contributors may be used to endorse or + * promote products derived from this software without specific + * prior written permission. + * + * ALTERNATIVELY, provided that this notice is retained in full, this + * product may be distributed under the terms of the GNU General Public + * License (GPL), in which case the provisions of the GPL apply INSTEAD OF + * those given above. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include + +#include "poly1305-internal.h" + +#ifdef POLY1305_USE_RISCV_V + +#include "simd-common-riscv.h" +#include + + +#define ALWAYS_INLINE inline __attribute__((always_inline)) +#define NO_INLINE __attribute__((noinline)) +#define NO_INSTRUMENT_FUNCTION __attribute__((no_instrument_function)) + +#define ASM_FUNC_ATTR NO_INSTRUMENT_FUNCTION +#define ASM_FUNC_ATTR_INLINE ASM_FUNC_ATTR ALWAYS_INLINE +#define ASM_FUNC_ATTR_NOINLINE ASM_FUNC_ATTR NO_INLINE + +#ifdef HAVE_GCC_ATTRIBUTE_OPTIMIZE +# define FUNC_ATTR_OPT_O2 __attribute__((optimize("-O2"))) +#else +# define FUNC_ATTR_OPT_O2 +#endif + + +/* Vector code uses up to 16 lanes (VLEN=1024, largest supported by QEMU), + * needing key powers r^1 ... r^32. */ +#define MAX_LANES 16 + +#define MASK26 ((1U << 26) - 1) + + +/********************************************************************** + Key power precomputation + **********************************************************************/ + +/* r = r^2 mod 2^130-5, base 2^26 */ +static ASM_FUNC_ATTR_INLINE void +sqr_2_26 (u64 r[5]) +{ + u64 r0 = r[0], r1 = r[1], r2 = r[2], r3 = r[3], r4 = r[4]; + u64 d0, d1, d2, d3, d4, c; + + d0 = r0 * r0 + 5 * 2 * (r1 * r4 + r2 * r3); + d1 = 2 * r0 * r1 + 5 * (2 * r2 * r4 + r3 * r3); + d2 = 2 * r0 * r2 + r1 * r1 + 5 * 2 * r3 * r4; + d3 = 2 * (r0 * r3 + r1 * r2) + 5 * r4 * r4; + d4 = 2 * (r0 * r4 + r1 * r3) + r2 * r2; + + c = d0 >> 26; d0 &= MASK26; d1 += c; + c = d1 >> 26; d1 &= MASK26; d2 += c; + c = d2 >> 26; d2 &= MASK26; d3 += c; + c = d3 >> 26; d3 &= MASK26; d4 += c; + c = d4 >> 26; d4 &= MASK26; d0 += c * 5; + c = d0 >> 26; d0 &= MASK26; d1 += c; + + r[0] = d0; + r[1] = d1; + r[2] = d2; + r[3] = d3; + r[4] = d4; +} + +/* Power table row for limb 'k' holds r^32, r^16, ..., r^2, r^1, so that + * r^(2^i) is at index 5 - i and two neighboring powers can be loaded as + * single pair. */ +static ASM_FUNC_ATTR_NOINLINE size_t +compute_powers (poly1305_context_t *ctx, unsigned int npowers) +{ + u32 (*pwrs)[6] = ctx->riscv_v_powers; + unsigned int i = ctx->riscv_v_npowers; + u64 r[5]; + + if (i == 0) + { + POLY1305_STATE *st = &ctx->state; + u64 r0 = st->r[0] + ((u64)st->r[1] << 32); + u64 r1 = st->r[2] + ((u64)st->r[3] << 32); + + r[0] = r0 & MASK26; + r[1] = (r0 >> 26) & MASK26; + r[2] = ((r0 >> 52) | (r1 << 12)) & MASK26; + r[3] = (r1 >> 14) & MASK26; + r[4] = r1 >> 40; + i = 1; + pwrs[0][5] = r[0]; + pwrs[1][5] = r[1]; + pwrs[2][5] = r[2]; + pwrs[3][5] = r[3]; + pwrs[4][5] = r[4]; + } + else + { + r[0] = pwrs[0][6 - i]; + r[1] = pwrs[1][6 - i]; + r[2] = pwrs[2][6 - i]; + r[3] = pwrs[3][6 - i]; + r[4] = pwrs[4][6 - i]; + } + + for (; i < npowers; i++) + { + sqr_2_26 (r); + pwrs[0][5 - i] = r[0]; + pwrs[1][5 - i] = r[1]; + pwrs[2][5 - i] = r[2]; + pwrs[3][5 - i] = r[3]; + pwrs[4][5 - i] = r[4]; + } + + ctx->riscv_v_npowers = npowers; + + return sizeof (r) + 128; +} + + +/********************************************************************** + RISC-V vector extension Poly1305 + **********************************************************************/ + +/* Load 'vl' blocks, de-interleaved to four 32-bit words. */ +static ASM_FUNC_ATTR_INLINE vuint32mf2x4_t +load_blocks (const byte *inp, u32 *alignbuf, size_t vl) +{ +#ifdef RVV_UNALIGNED_NOT_ALLOWED + if (UNLIKELY (((uintptr_t)inp & 3) != 0)) + { + __riscv_vse8_v_u8m2((byte *)alignbuf, + __riscv_vle8_v_u8m2(inp, vl * 16), vl * 16); + inp = (const byte *)alignbuf; + } +#else + (void)alignbuf; +#endif + return __riscv_vlseg4e32_v_u32mf2x4((const void *)inp, vl); +} + +/* Base 2^32 -> 2^26, with padbit. */ +#define CONVERT_2_26(m, t) \ + m##0 = __riscv_vget_v_u32mf2x4_u32mf2(t, 0); \ + m##1 = __riscv_vget_v_u32mf2x4_u32mf2(t, 1); \ + m##2 = __riscv_vget_v_u32mf2x4_u32mf2(t, 2); \ + m##3 = __riscv_vget_v_u32mf2x4_u32mf2(t, 3); \ + m##4 = __riscv_vor_vx_u32mf2( \ + __riscv_vsrl_vx_u32mf2(m##3, 8, vl), 1 << 24, vl); \ + m##3 = __riscv_vand_vx_u32mf2( \ + __riscv_vor_vv_u32mf2( \ + __riscv_vsll_vx_u32mf2(m##3, 18, vl), \ + __riscv_vsrl_vx_u32mf2(m##2, 14, vl), vl), MASK26, vl); \ + m##2 = __riscv_vand_vx_u32mf2( \ + __riscv_vor_vv_u32mf2( \ + __riscv_vsll_vx_u32mf2(m##2, 12, vl), \ + __riscv_vsrl_vx_u32mf2(m##1, 20, vl), vl), MASK26, vl); \ + m##1 = __riscv_vand_vx_u32mf2( \ + __riscv_vor_vv_u32mf2( \ + __riscv_vsll_vx_u32mf2(m##1, 6, vl), \ + __riscv_vsrl_vx_u32mf2(m##0, 26, vl), vl), MASK26, vl); \ + m##0 = __riscv_vand_vx_u32mf2(m##0, MASK26, vl) + +#define ADD_HASH(m, h) \ + m##0 = __riscv_vadd_vv_u32mf2(m##0, h##0, vl); \ + m##1 = __riscv_vadd_vv_u32mf2(m##1, h##1, vl); \ + m##2 = __riscv_vadd_vv_u32mf2(m##2, h##2, vl); \ + m##3 = __riscv_vadd_vv_u32mf2(m##3, h##3, vl); \ + m##4 = __riscv_vadd_vv_u32mf2(m##4, h##4, vl) + +/* + * d4 = h0*r4 + h1*r3 + h2*r2 + h3*r1 + h4*r0 + * d3 = h0*r3 + h1*r2 + h2*r1 + h3*r0 + h4*5*r4 + * d2 = h0*r2 + h1*r1 + h2*r0 + h3*5*r4 + h4*5*r3 + * d1 = h0*r1 + h1*r0 + h2*5*r4 + h3*5*r3 + h4*5*r2 + * d0 = h0*r0 + h1*5*r4 + h2*5*r3 + h3*5*r2 + h4*5*r1 + */ +#define MAC_ROWS_1_4(d, h, r, op) \ + d##4 = op (d##4, r##3, h##1, vl); \ + d##3 = op (d##3, r##2, h##1, vl); \ + d##2 = op (d##2, r##1, h##1, vl); \ + d##1 = op (d##1, r##0, h##1, vl); \ + d##0 = op (d##0, r##4x5, h##1, vl); \ + \ + d##4 = op (d##4, r##2, h##2, vl); \ + d##3 = op (d##3, r##1, h##2, vl); \ + d##2 = op (d##2, r##0, h##2, vl); \ + d##1 = op (d##1, r##4x5, h##2, vl); \ + d##0 = op (d##0, r##3x5, h##2, vl); \ + \ + d##4 = op (d##4, r##1, h##3, vl); \ + d##3 = op (d##3, r##0, h##3, vl); \ + d##2 = op (d##2, r##4x5, h##3, vl); \ + d##1 = op (d##1, r##3x5, h##3, vl); \ + d##0 = op (d##0, r##2x5, h##3, vl); \ + \ + d##4 = op (d##4, r##0, h##4, vl); \ + d##3 = op (d##3, r##4x5, h##4, vl); \ + d##2 = op (d##2, r##3x5, h##4, vl); \ + d##1 = op (d##1, r##2x5, h##4, vl); \ + d##0 = op (d##0, r##1x5, h##4, vl) + +/* d = h * r, with scalar r */ +#define MUL_VX(d, h, r) \ + d##4 = __riscv_vwmulu_vx_u64m1(h##0, r##4, vl); \ + d##3 = __riscv_vwmulu_vx_u64m1(h##0, r##3, vl); \ + d##2 = __riscv_vwmulu_vx_u64m1(h##0, r##2, vl); \ + d##1 = __riscv_vwmulu_vx_u64m1(h##0, r##1, vl); \ + d##0 = __riscv_vwmulu_vx_u64m1(h##0, r##0, vl); \ + MAC_ROWS_1_4 (d, h, r, __riscv_vwmaccu_vx_u64m1) + +/* d += h * r, with scalar r */ +#define MAC_VX(d, h, r) \ + d##4 = __riscv_vwmaccu_vx_u64m1(d##4, r##4, h##0, vl); \ + d##3 = __riscv_vwmaccu_vx_u64m1(d##3, r##3, h##0, vl); \ + d##2 = __riscv_vwmaccu_vx_u64m1(d##2, r##2, h##0, vl); \ + d##1 = __riscv_vwmaccu_vx_u64m1(d##1, r##1, h##0, vl); \ + d##0 = __riscv_vwmaccu_vx_u64m1(d##0, r##0, h##0, vl); \ + MAC_ROWS_1_4 (d, h, r, __riscv_vwmaccu_vx_u64m1) + +/* d = h * r, with vector r */ +#define MUL_VV(d, h, r) \ + d##4 = __riscv_vwmulu_vv_u64m1(h##0, r##4, vl); \ + d##3 = __riscv_vwmulu_vv_u64m1(h##0, r##3, vl); \ + d##2 = __riscv_vwmulu_vv_u64m1(h##0, r##2, vl); \ + d##1 = __riscv_vwmulu_vv_u64m1(h##0, r##1, vl); \ + d##0 = __riscv_vwmulu_vv_u64m1(h##0, r##0, vl); \ + MAC_ROWS_1_4 (d, h, r, __riscv_vwmaccu_vv_u64m1) + +/* + * Lazy reduction as discussed in "NEON crypto" by D.J. Bernstein + * and P. Schwabe + * + * H0>>+H1>>+H2>>+H3>>+H4 + * H3>>+H4>>*5+H0>>+H1 + * + * >>+ denotes Hnext += Hn>>26, Hn &= 0x3ffffff. + */ +#define LAZY_REDC(h, d) do { \ + vuint64m1_t w0, w1, w2, w3, w4; \ + vuint32mf2_t c3; \ + \ + w3 = __riscv_vand_vx_u64m1(d##3, MASK26, vl); \ + d##3 = __riscv_vsrl_vx_u64m1(d##3, 26, vl); \ + w0 = __riscv_vand_vx_u64m1(d##0, MASK26, vl); \ + d##0 = __riscv_vsrl_vx_u64m1(d##0, 26, vl); \ + d##4 = __riscv_vadd_vv_u64m1(d##4, d##3, vl); \ + d##1 = __riscv_vadd_vv_u64m1(d##1, d##0, vl); \ + \ + w4 = __riscv_vand_vx_u64m1(d##4, MASK26, vl); \ + d##4 = __riscv_vsrl_vx_u64m1(d##4, 26, vl); \ + d##0 = __riscv_vsll_vx_u64m1(d##4, 2, vl); \ + w0 = __riscv_vadd_vv_u64m1(w0, d##4, vl); \ + w1 = __riscv_vand_vx_u64m1(d##1, MASK26, vl); \ + d##1 = __riscv_vsrl_vx_u64m1(d##1, 26, vl); \ + w0 = __riscv_vadd_vv_u64m1(w0, d##0, vl); \ + d##2 = __riscv_vadd_vv_u64m1(d##2, d##1, vl); \ + \ + d##0 = __riscv_vsrl_vx_u64m1(w0, 26, vl); \ + w0 = __riscv_vand_vx_u64m1(w0, MASK26, vl); \ + w2 = __riscv_vand_vx_u64m1(d##2, MASK26, vl); \ + d##2 = __riscv_vsrl_vx_u64m1(d##2, 26, vl); \ + w1 = __riscv_vadd_vv_u64m1(w1, d##0, vl); \ + w3 = __riscv_vadd_vv_u64m1(w3, d##2, vl); \ + \ + h##0 = __riscv_vncvt_x_x_w_u32mf2(w0, vl); \ + h##1 = __riscv_vncvt_x_x_w_u32mf2(w1, vl); \ + h##3 = __riscv_vncvt_x_x_w_u32mf2(w3, vl); \ + h##4 = __riscv_vncvt_x_x_w_u32mf2(w4, vl); \ + h##2 = __riscv_vncvt_x_x_w_u32mf2(w2, vl); \ + \ + c3 = __riscv_vsrl_vx_u32mf2(h##3, 26, vl); \ + h##3 = __riscv_vand_vx_u32mf2(h##3, MASK26, vl); \ + h##4 = __riscv_vadd_vv_u32mf2(h##4, c3, vl); \ + } while (0) + +#define LOAD_SCALAR_POWER(r, idx) \ + r##0 = pwrs[0][idx]; \ + r##1 = pwrs[1][idx]; \ + r##2 = pwrs[2][idx]; \ + r##3 = pwrs[3][idx]; \ + r##4 = pwrs[4][idx]; \ + r##1x5 = r##1 * 5; \ + r##2x5 = r##2 * 5; \ + r##3x5 = r##3 * 5; \ + r##4x5 = r##4 * 5 + +/* Process input with 'nblks' blocks. Blocks are processed in multiples of + * vector lane count and 'nblks' is updated to number of unprocessed + * trailing blocks. Caller must provide at least 2 blocks. */ +unsigned int ASM_FUNC_ATTR FUNC_ATTR_OPT_O2 +_gcry_poly1305_riscv_v_blocks (poly1305_context_t *ctx, const byte *inp, + size_t *nblks) +{ + POLY1305_STATE *st = &ctx->state; + u32 (*pwrs)[6] = ctx->riscv_v_powers; +#ifdef RVV_UNALIGNED_NOT_ALLOWED + u32 alignbuf[MAX_LANES * 4]; + size_t burn_stack = sizeof (alignbuf) + 32 * sizeof (u64); +#else + u32 *alignbuf = NULL; + size_t burn_stack = 32 * sizeof (u64); +#endif + vuint32mf2_t h0, h1, h2, h3, h4; + vuint32mf2_t lo0, lo1, lo2, lo3, lo4; + vuint32mf2_t hi0, hi1, hi2, hi3, hi4; + vuint64m1_t acc0, acc1, acc2, acc3, acc4; + vuint32mf2x4_t data; + u32 r4_0, r4_1, r4_2, r4_3, r4_4, r4_1x5, r4_2x5, r4_3x5, r4_4x5; + u32 r8_0, r8_1, r8_2, r8_3, r8_4, r8_1x5, r8_2x5, r8_3x5, r8_4x5; + u64 s0, s1, s2, s3, s4; + u64 d0, d1, d2, t; + size_t vl, n; + unsigned int idx; + + vl = __riscv_vsetvlmax_e32mf2(); + if (vl > MAX_LANES) + vl = MAX_LANES; + + /* Downsize lane count if input is too short. */ + while (vl > 2 && *nblks < 3 * vl) + vl >>= 1; + + n = *nblks & ~(vl - 1); + *nblks -= n; + + /* Index of r^vl in the power table. */ + for (idx = 5, t = vl; t > 1; t >>= 1) + idx--; + + if (ctx->riscv_v_npowers < 7 - idx) + burn_stack += compute_powers (ctx, 7 - idx); + + /* Convert hash value to base 2^26. */ + d0 = st->h[0] + ((u64)st->h[1] << 32); + d1 = st->h[2] + ((u64)st->h[3] << 32); + d2 = st->h[4]; + + t = (d2 & ~(u64)3) + (d2 >> 2); + d2 &= 3; + + s0 = d0 & MASK26; + s1 = (d0 >> 26) & MASK26; + s2 = ((d0 >> 52) | (d1 << 12)) & MASK26; + s3 = (d1 >> 14) & MASK26; + s4 = (d1 >> 40) | (d2 << 24); + + s0 += t & MASK26; + t >>= 26; + s1 += t & MASK26; + t >>= 26; + s2 += t; + + h0 = __riscv_vmv_s_x_u32mf2_tu(__riscv_vmv_v_x_u32mf2(0, vl), s0, vl); + h1 = __riscv_vmv_s_x_u32mf2_tu(__riscv_vmv_v_x_u32mf2(0, vl), s1, vl); + h2 = __riscv_vmv_s_x_u32mf2_tu(__riscv_vmv_v_x_u32mf2(0, vl), s2, vl); + h3 = __riscv_vmv_s_x_u32mf2_tu(__riscv_vmv_v_x_u32mf2(0, vl), s3, vl); + h4 = __riscv_vmv_s_x_u32mf2_tu(__riscv_vmv_v_x_u32mf2(0, vl), s4, vl); + + /* Two top-most powers, r^vl and r^(2*vl). */ + LOAD_SCALAR_POWER (r4_, idx); + LOAD_SCALAR_POWER (r8_, idx - 1); + + /* + * (((inp[0]*r^8 + inp[4]*r^4 + inp[8]) *r^4 + inp[12])*r^2 + )*r^2 + * (((inp[1]*r^8 + inp[5]*r^4 + inp[9]) *r^4 + inp[13])*r^2 + )*r^1 + * (((inp[2]*r^8 + inp[6]*r^4 + inp[10])*r^4 + inp[14]) ----^ + * (((inp[3]*r^8 + inp[7]*r^4 + inp[11])*r^4 + inp[15]) ----^ + * \_____________________/ + * (((inp[0]*r^8 + inp[4]*r^4 + inp[8]) *r^8 + inp[12]*r^4 + inp[16])*r^2 + )*r^2 + * (((inp[1]*r^8 + inp[5]*r^4 + inp[9]) *r^8 + inp[13]*r^4 + inp[17])*r^2 + )*r + * (((inp[2]*r^8 + inp[6]*r^4 + inp[10])*r^8 + inp[14]*r^4 + inp[18]) ----^ + * (((inp[3]*r^8 + inp[7]*r^4 + inp[11])*r^8 + inp[15]*r^4 + inp[19]) ----^ + * \_____________________/ \__________________________/ + * + * Note that we start with inp[vl:2*vl]*r^4. This is because it doesn't + * depend on reduction in previous iteration. + */ + + data = load_blocks (inp, alignbuf, vl); + inp += vl * 16; + n -= vl; + CONVERT_2_26 (lo, data); + + while (n >= 2 * vl) + { + data = load_blocks (inp, alignbuf, vl); + inp += vl * 16; + CONVERT_2_26 (hi, data); + + MUL_VX (acc, hi, r4_); + + /* (hash+inp[0:vl])*r^(2*vl) and accumulate */ + ADD_HASH (lo, h); + MAC_VX (acc, lo, r8_); + + data = load_blocks (inp, alignbuf, vl); + inp += vl * 16; + CONVERT_2_26 (lo, data); + + LAZY_REDC (h, acc); + + n -= 2 * vl; + } + + if (n) + { + ADD_HASH (lo, h); + MUL_VX (acc, lo, r4_); + + data = load_blocks (inp, alignbuf, vl); + CONVERT_2_26 (lo, data); + + LAZY_REDC (h, acc); + } + + /* Reduce lanes by multiplying upper half with r^(vl/2) and adding + * to lower half. */ + ADD_HASH (lo, h); + while (vl > 2) + { + idx++; + LOAD_SCALAR_POWER (r4_, idx); + MUL_VX (acc, lo, r4_); + + LAZY_REDC (h, acc); + + vl >>= 1; + lo0 = __riscv_vslidedown_vx_u32mf2(lo0, vl, vl); + lo1 = __riscv_vslidedown_vx_u32mf2(lo1, vl, vl); + lo2 = __riscv_vslidedown_vx_u32mf2(lo2, vl, vl); + lo3 = __riscv_vslidedown_vx_u32mf2(lo3, vl, vl); + lo4 = __riscv_vslidedown_vx_u32mf2(lo4, vl, vl); + ADD_HASH (lo, h); + } + + /* Last two lanes, multiply with r^2 and r^1. */ + hi0 = __riscv_vle32_v_u32mf2(&pwrs[0][4], vl); + hi1 = __riscv_vle32_v_u32mf2(&pwrs[1][4], vl); + hi2 = __riscv_vle32_v_u32mf2(&pwrs[2][4], vl); + hi3 = __riscv_vle32_v_u32mf2(&pwrs[3][4], vl); + hi4 = __riscv_vle32_v_u32mf2(&pwrs[4][4], vl); + { + vuint32mf2_t hi1x5 = __riscv_vmul_vx_u32mf2(hi1, 5, vl); + vuint32mf2_t hi2x5 = __riscv_vmul_vx_u32mf2(hi2, 5, vl); + vuint32mf2_t hi3x5 = __riscv_vmul_vx_u32mf2(hi3, 5, vl); + vuint32mf2_t hi4x5 = __riscv_vmul_vx_u32mf2(hi4, 5, vl); + + MUL_VV (acc, lo, hi); + } + + LAZY_REDC (h, acc); + + h0 = __riscv_vadd_vv_u32mf2(h0, __riscv_vslidedown_vx_u32mf2(h0, 1, 1), 1); + h1 = __riscv_vadd_vv_u32mf2(h1, __riscv_vslidedown_vx_u32mf2(h1, 1, 1), 1); + h2 = __riscv_vadd_vv_u32mf2(h2, __riscv_vslidedown_vx_u32mf2(h2, 1, 1), 1); + h3 = __riscv_vadd_vv_u32mf2(h3, __riscv_vslidedown_vx_u32mf2(h3, 1, 1), 1); + h4 = __riscv_vadd_vv_u32mf2(h4, __riscv_vslidedown_vx_u32mf2(h4, 1, 1), 1); + + s0 = __riscv_vmv_x_s_u32mf2_u32(h0); + s1 = __riscv_vmv_x_s_u32mf2_u32(h1); + s2 = __riscv_vmv_x_s_u32mf2_u32(h2); + s3 = __riscv_vmv_x_s_u32mf2_u32(h3); + s4 = __riscv_vmv_x_s_u32mf2_u32(h4); + + /* Convert hash value to base 2^64. */ + d0 = s0 + (s1 << 26); + t = s2 << 52; + d0 += t; + d1 = (s2 >> 12) + (d0 < t); + d1 += s3 << 14; + t = s4 << 40; + d1 += t; + d2 = (s4 >> 24) + (d1 < t); + + /* Lane sum can exceed 2^131-5, fold bits above 2^130 for poly1305_final. */ + t = (d2 & ~(u64)3) + (d2 >> 2); + d2 &= 3; + d0 += t; + t = d0 < t; + d1 += t; + d2 += d1 < t; + + st->h[0] = d0; + st->h[1] = d0 >> 32; + st->h[2] = d1; + st->h[3] = d1 >> 32; + st->h[4] = d2; + + clear_vec_regs (); + + return burn_stack; +} + +unsigned int ASM_FUNC_ATTR FUNC_ATTR_OPT_O2 +_gcry_poly1305_riscv_v_check_hw (void) +{ + return (__riscv_vsetvl_e8m1(16) == 16); +} + +#endif /* POLY1305_USE_RISCV_V */ diff --git a/cipher/poly1305.c b/cipher/poly1305.c index 8739a4ef..73b63f78 100644 --- a/cipher/poly1305.c +++ b/cipher/poly1305.c @@ -116,6 +116,17 @@ gcry_poly1305_p10le_4blocks(unsigned char *key, const byte *m, size_t len); #endif /* POLY1305_USE_PPC_VEC */ +#ifdef POLY1305_USE_RISCV_V + +extern unsigned int +_gcry_poly1305_riscv_v_blocks (poly1305_context_t *ctx, const byte *buf, + size_t *nblks); + +extern unsigned int _gcry_poly1305_riscv_v_check_hw (void); + +#endif /* POLY1305_USE_RISCV_V */ + + static void poly1305_init (poly1305_context_t *ctx, const byte key[POLY1305_KEYLEN]) { @@ -135,6 +146,14 @@ static void poly1305_init (poly1305_context_t *ctx, # endif #endif +#ifdef POLY1305_USE_RISCV_V + ctx->use_riscv_v = (features & HWF_RISCV_IMAFDC) + && (features & HWF_RISCV_B) /* Mandatory in RVA22U64 */ + && (features & HWF_RISCV_V) /* Optional in RVA22U64 */ + && _gcry_poly1305_riscv_v_check_hw(); + ctx->riscv_v_npowers = 0; +#endif + (void)features; ctx->leftover = 0; @@ -298,6 +317,23 @@ poly1305_blocks (poly1305_context_t *ctx, const byte *buf, size_t len, return poly1305_amd64_avx512_blocks(ctx, buf, len); #endif +#ifdef POLY1305_USE_RISCV_V + if ((high_pad & ctx->use_riscv_v) != 0 && len >= 64 * POLY1305_BLOCKSIZE) + { + size_t nblks = len / POLY1305_BLOCKSIZE; + unsigned int burn, nburn; + + burn = _gcry_poly1305_riscv_v_blocks(ctx, buf, &nblks); + if (nblks == 0) + return burn; + + buf += len - nblks * POLY1305_BLOCKSIZE; + len = nblks * POLY1305_BLOCKSIZE; + nburn = poly1305_blocks_generic(ctx, buf, len, high_pad); + return nburn > burn ? nburn : burn; + } +#endif + return poly1305_blocks_generic(ctx, buf, len, high_pad); } diff --git a/configure.ac b/configure.ac index fb778e38..2a126041 100644 --- a/configure.ac +++ b/configure.ac @@ -4323,6 +4323,9 @@ case "${host}" in x86_64-*-*) GCRYPT_ASM_DIGESTS="$GCRYPT_ASM_DIGESTS poly1305-amd64-avx512.lo" ;; + riscv64-*-*) + GCRYPT_ASM_DIGESTS="$GCRYPT_ASM_DIGESTS poly1305-riscv-v.lo" + ;; powerpc64le-*-*) # Build with the assembly implementation if test "$gcry_cv_gcc_inline_asm_ppc_altivec" = "yes" && -- 2.53.0 From jussi.kivilinna at iki.fi Sun Oct 11 19:23:16 2026 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sun, 11 Oct 2026 20:23:16 +0300 Subject: [PATCH 07/12] sm3: add RISC-V Zvksh intrinsics implementation In-Reply-To: <20261011172321.997183-1-jussi.kivilinna@iki.fi> References: <20261011172321.997183-1-jussi.kivilinna@iki.fi> Message-ID: <20261011172321.997183-7-jussi.kivilinna@iki.fi> * cipher/Makefile.am: Add 'sm3-riscv-zvksh-zvkb.c', 'sm3-riscv-zvksh-zvkb-m1.c' and 'sm3-riscv-zvksh-zvkb.h'. (riscv_vector_crypto_sm3_cflags) (riscv_vector_crypto_sm3_m1_cflags): New. * cipher/sm3-riscv-zvksh-zvkb.c: New. * cipher/sm3-riscv-zvksh-zvkb-m1.c: New. * cipher/sm3-riscv-zvksh-zvkb.h: New. * cipher/sm3.c (USE_RISCV_V_CRYPTO): New. [USE_RISCV_V_CRYPTO] (_gcry_sm3_riscv_v_check_hw) (_gcry_sm3_transform_riscv_zvksh_zvkb) (do_sm3_transform_riscv_zvksh): New. (sm3_init) [USE_RISCV_V_CRYPTO]: Enable new implementation if supported by HW. * configure.ac: Add 'sm3-riscv-zvksh-zvkb.lo' and 'sm3-riscv-zvksh-zvkb-m1.lo'. (GCRY_RISCV_VECTOR_CRYPTO_INTRINSICS_TEST): Add SM3 intrinsics test. * doc/gcrypt.texi: Add "riscv-zvksh". * src/g10lib.h [HAVE_CPU_ARCH_RISCV] (HWF_RISCV_ZVKSH): New. * src/hwf-riscv.c (HWF_RISCV_HWPROBE_EXT_ZVKSH): New. (hwprobe_features): Add Zvksh. * src/hwfeatures.c (hwflist) [HAVE_CPU_ARCH_RISCV]: Add "riscv-zvksh". -- LMUL=1 is used with VLEN>=256 and LMUL=2 with VLEN=128. clang requires zvl256b for LMUL=1 SM3 intrinsics, so LMUL=1 variant is placed in separate file compiled with zvl256b. LMUL=2 variant and code-path selection are compiled without zvl256b as they run on VLEN=128. Benchmark on SpacemiT K3, X100 core (out-of-order, VLEN=256): Before: | nanosecs/byte mebibytes/sec cycles/byte auto Mhz SM3 | 6.29 ns/B 151.5 MiB/s 12.56 c/B 1996 After (~4.2x faster): | nanosecs/byte mebibytes/sec cycles/byte auto Mhz SM3 | 1.49 ns/B 639.1 MiB/s 2.98 c/B 1997 Benchmark on SpacemiT K3, A100 core (in-order, VLEN=1024): Before: | nanosecs/byte mebibytes/sec cycles/byte auto Mhz SM3 | 11.71 ns/B 81.42 MiB/s 21.08 c/B 1800 After (~2.8x faster): | nanosecs/byte mebibytes/sec cycles/byte auto Mhz SM3 | 4.16 ns/B 229.0 MiB/s 7.50 c/B 1800 Signed-off-by: Jussi Kivilinna --- cipher/Makefile.am | 23 +++++- cipher/sm3-riscv-zvksh-zvkb-m1.c | 58 ++++++++++++++ cipher/sm3-riscv-zvksh-zvkb.c | 75 ++++++++++++++++++ cipher/sm3-riscv-zvksh-zvkb.h | 126 +++++++++++++++++++++++++++++++ cipher/sm3.c | 35 +++++++++ configure.ac | 22 ++++++ doc/gcrypt.texi | 1 + src/g10lib.h | 1 + src/hwf-riscv.c | 2 + src/hwfeatures.c | 1 + 10 files changed, 343 insertions(+), 1 deletion(-) create mode 100644 cipher/sm3-riscv-zvksh-zvkb-m1.c create mode 100644 cipher/sm3-riscv-zvksh-zvkb.c create mode 100644 cipher/sm3-riscv-zvksh-zvkb.h diff --git a/cipher/Makefile.am b/cipher/Makefile.am index 6c51ee94..50509914 100644 --- a/cipher/Makefile.am +++ b/cipher/Makefile.am @@ -158,7 +158,8 @@ EXTRA_libcipher_la_SOURCES = \ sha512-riscv-zvknhb-zvkb-m1.c \ sha512-ssse3-i386.c \ sm3.c sm3-avx-bmi2-amd64.S sm3-intel-avx2-amd64.S sm3-aarch64.S \ - sm3-armv8-aarch64-ce.S \ + sm3-armv8-aarch64-ce.S sm3-riscv-zvksh-zvkb.c sm3-riscv-zvksh-zvkb.h \ + sm3-riscv-zvksh-zvkb-m1.c \ keccak.c keccak_permute_32.h keccak_permute_64.h \ keccak-armv7-neon.S keccak-amd64-avx512.S \ stribog.c \ @@ -471,6 +472,8 @@ riscv_vector_crypto_aes_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvkned_zvkb riscv_vector_crypto_sha256_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvknha_zvkb riscv_vector_crypto_sha512_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvknhb_zvkb riscv_vector_crypto_sha512_m1_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvknhb_zvkb_zvl256b +riscv_vector_crypto_sm3_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvksh_zvkb +riscv_vector_crypto_sm3_m1_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvksh_zvkb_zvl256b riscv_vector_crypto_gcm_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvkg_zvkb else riscv_vector_crypto_zvkb_cflags = @@ -478,6 +481,8 @@ riscv_vector_crypto_aes_cflags = riscv_vector_crypto_sha256_cflags = riscv_vector_crypto_sha512_cflags = riscv_vector_crypto_sha512_m1_cflags = +riscv_vector_crypto_sm3_cflags = +riscv_vector_crypto_sm3_m1_cflags = riscv_vector_crypto_gcm_cflags = endif @@ -528,3 +533,19 @@ sha512-riscv-zvknhb-zvkb-m1.o: $(srcdir)/sha512-riscv-zvknhb-zvkb-m1.c Makefile sha512-riscv-zvknhb-zvkb-m1.lo: $(srcdir)/sha512-riscv-zvknhb-zvkb-m1.c Makefile `echo $(LTCOMPILE) $(riscv_vector_crypto_sha512_m1_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` $(custom_depfile_lo) + +sm3-riscv-zvksh-zvkb.o: $(srcdir)/sm3-riscv-zvksh-zvkb.c Makefile + `echo $(COMPILE) $(riscv_vector_crypto_sm3_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) + +sm3-riscv-zvksh-zvkb.lo: $(srcdir)/sm3-riscv-zvksh-zvkb.c Makefile + `echo $(LTCOMPILE) $(riscv_vector_crypto_sm3_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) + +sm3-riscv-zvksh-zvkb-m1.o: $(srcdir)/sm3-riscv-zvksh-zvkb-m1.c Makefile + `echo $(COMPILE) $(riscv_vector_crypto_sm3_m1_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) + +sm3-riscv-zvksh-zvkb-m1.lo: $(srcdir)/sm3-riscv-zvksh-zvkb-m1.c Makefile + `echo $(LTCOMPILE) $(riscv_vector_crypto_sm3_m1_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) diff --git a/cipher/sm3-riscv-zvksh-zvkb-m1.c b/cipher/sm3-riscv-zvksh-zvkb-m1.c new file mode 100644 index 00000000..278037a5 --- /dev/null +++ b/cipher/sm3-riscv-zvksh-zvkb-m1.c @@ -0,0 +1,58 @@ +/* sm3-riscv-zvksh-zvkb-m1.c - RISC-V vector crypto SM3, VLEN>=256 variant + * Copyright (C) 2026 Jussi Kivilinna + * + * This file is part of Libgcrypt. + * + * Libgcrypt is free software; you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as + * published by the Free Software Foundation; either version 2.1 of + * the License, or (at your option) any later version. + * + * Libgcrypt is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this program; if not, see . + */ + +#include + +#if defined (__riscv) && \ + defined(HAVE_COMPATIBLE_CC_RISCV_VECTOR_INTRINSICS) && \ + defined(HAVE_COMPATIBLE_CC_RISCV_VECTOR_CRYPTO_INTRINSICS) && \ + defined(USE_SM3) + +#include "g10lib.h" +#include "simd-common-riscv.h" +#include + + +#define ALWAYS_INLINE inline __attribute__((always_inline)) +#define NO_INLINE __attribute__((noinline)) +#define NO_INSTRUMENT_FUNCTION __attribute__((no_instrument_function)) + +#define ASM_FUNC_ATTR NO_INSTRUMENT_FUNCTION +#define ASM_FUNC_ATTR_INLINE ASM_FUNC_ATTR ALWAYS_INLINE +#define ASM_FUNC_ATTR_NOINLINE ASM_FUNC_ATTR NO_INLINE + +#ifdef HAVE_GCC_ATTRIBUTE_OPTIMIZE +# define FUNC_ATTR_OPT_O2 __attribute__((optimize("-O2"))) +#else +# define FUNC_ATTR_OPT_O2 +#endif + + +#define SM3_LMUL m1 +#include "sm3-riscv-zvksh-zvkb.h" + +unsigned int +_gcry_sm3_transform_riscv_zvksh_zvkb_m1(u32 state[8], + const unsigned char *input_data, + size_t num_blks) +{ + return sm3_transform_zvksh_zvkb_m1(state, input_data, num_blks); +} + +#endif /* HAVE_COMPATIBLE_CC_RISCV_VECTOR_INTRINSICS */ diff --git a/cipher/sm3-riscv-zvksh-zvkb.c b/cipher/sm3-riscv-zvksh-zvkb.c new file mode 100644 index 00000000..fca9651e --- /dev/null +++ b/cipher/sm3-riscv-zvksh-zvkb.c @@ -0,0 +1,75 @@ +/* sm3-riscv-zvksh-zvkb.c - RISC-V vector crypto implementation of SM3 + * Copyright (C) 2026 Jussi Kivilinna + * + * This file is part of Libgcrypt. + * + * Libgcrypt is free software; you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as + * published by the Free Software Foundation; either version 2.1 of + * the License, or (at your option) any later version. + * + * Libgcrypt is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this program; if not, see . + */ + +#include + +#if defined (__riscv) && \ + defined(HAVE_COMPATIBLE_CC_RISCV_VECTOR_INTRINSICS) && \ + defined(HAVE_COMPATIBLE_CC_RISCV_VECTOR_CRYPTO_INTRINSICS) && \ + defined(USE_SM3) + +#include "g10lib.h" +#include "simd-common-riscv.h" +#include + + +#define ALWAYS_INLINE inline __attribute__((always_inline)) +#define NO_INLINE __attribute__((noinline)) +#define NO_INSTRUMENT_FUNCTION __attribute__((no_instrument_function)) + +#define ASM_FUNC_ATTR NO_INSTRUMENT_FUNCTION +#define ASM_FUNC_ATTR_INLINE ASM_FUNC_ATTR ALWAYS_INLINE +#define ASM_FUNC_ATTR_NOINLINE ASM_FUNC_ATTR NO_INLINE + +#ifdef HAVE_GCC_ATTRIBUTE_OPTIMIZE +# define FUNC_ATTR_OPT_O2 __attribute__((optimize("-O2"))) +#else +# define FUNC_ATTR_OPT_O2 +#endif + + +/* m2 variant for VLEN == 128 */ +#define SM3_LMUL m2 +#include "sm3-riscv-zvksh-zvkb.h" + +/* m1 variant for VLEN >= 256, compiled separately with zvl256b */ +extern unsigned int +_gcry_sm3_transform_riscv_zvksh_zvkb_m1(u32 state[8], + const unsigned char *input_data, + size_t num_blks); + +unsigned int +_gcry_sm3_transform_riscv_zvksh_zvkb(u32 state[8], + const unsigned char *input_data, + size_t num_blks) +{ + if (__riscv_vsetvlmax_e8m1() >= 32) + return _gcry_sm3_transform_riscv_zvksh_zvkb_m1(state, input_data, + num_blks); + else + return sm3_transform_zvksh_zvkb_m2(state, input_data, num_blks); +} + +unsigned int ASM_FUNC_ATTR FUNC_ATTR_OPT_O2 +_gcry_sm3_riscv_v_check_hw(void) +{ + return (__riscv_vsetvl_e32m2(8) == 8); +} + +#endif /* HAVE_COMPATIBLE_CC_RISCV_VECTOR_INTRINSICS */ diff --git a/cipher/sm3-riscv-zvksh-zvkb.h b/cipher/sm3-riscv-zvksh-zvkb.h new file mode 100644 index 00000000..ec22cda7 --- /dev/null +++ b/cipher/sm3-riscv-zvksh-zvkb.h @@ -0,0 +1,126 @@ +/* sm3-riscv-zvksh-zvkb.h - RISC-V vector crypto SM3 LMUL template + * Copyright (C) 2026 Jussi Kivilinna + * + * This file is part of Libgcrypt. + * + * Libgcrypt is free software; you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as + * published by the Free Software Foundation; either version 2.1 of + * the License, or (at your option) any later version. + * + * Libgcrypt is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this program; if not, see . + */ + +/* Included once for each LMUL with SM3_LMUL set. */ + +#define CAT_(a, b) a##b +#define CAT(a, b) CAT_(a, b) + +#define FN(name) CAT(name##_, SM3_LMUL) +#define V(name) CAT(__riscv_##name##_u32, SM3_LMUL) +#define V8(name) CAT(__riscv_##name##_u8, SM3_LMUL) +#define VEC_T CAT(CAT(vuint32, SM3_LMUL), _t) + + +/* Prepare vsm3c message inputs for eight rounds from W[j..j+7] in 'wa' + * and W[j+8..j+15] in 'wb'. vsm3c takes W[i], W[i+1] from elements 0-1 + * and W[i+4], W[i+5] from elements 4-5. */ +#define PREP_W(wa, wb, s1, s2, s3) \ + s1 = __riscv_vslidedown(wa, 2, vl); \ + s2 = __riscv_vslidedown(wa, 4, vl); \ + s2 = __riscv_vslideup(s2, wb, 4, vl); \ + s3 = __riscv_vslidedown(s2, 2, vl); + +/* Eight rounds with message inputs 'wa', 's1', 's2' and 's3'. */ +#define EIGHT_ROUNDS(wa, s1, s2, s3, rnds, prep_next, expand) \ + v_hash_work = __riscv_vsm3c(v_hash_work, wa, (rnds) + 0, vl); \ + prep_next; \ + v_hash_work = __riscv_vsm3c(v_hash_work, s1, (rnds) + 1, vl); \ + expand; \ + v_hash_work = __riscv_vsm3c(v_hash_work, s2, (rnds) + 2, vl); \ + v_hash_work = __riscv_vsm3c(v_hash_work, s3, (rnds) + 3, vl); + + +static ASM_FUNC_ATTR_INLINE VEC_T +FN(load_msg) (const byte *p, size_t vl) +{ +#ifdef RVV_UNALIGNED_NOT_ALLOWED + return V(vreinterpret)(V8(vle8_v)(p, vl * 4)); +#else + return V(vle32_v)((const void *)p, vl); +#endif +} + +static ASM_FUNC_ATTR_NOINLINE FUNC_ATTR_OPT_O2 unsigned int +FN(sm3_transform_zvksh_zvkb) (u32 state[8], const byte *data, size_t nblocks) +{ + size_t vl; + VEC_T v_hash, v_hash_work; + VEC_T w0, w1, w2; + VEC_T a1, a2, a3, b1, b2, b3; + + vl = 8; + + /* State is kept in big-endian byte order. */ + v_hash = __riscv_vrev8(V(vle32_v)(state, vl), vl); + + while (nblocks > 0) + { + v_hash_work = v_hash; + + w0 = FN(load_msg)(data + 0, vl); + w1 = FN(load_msg)(data + 32, vl); + + w2 = __riscv_vsm3me(w1, w0, vl); + PREP_W(w0, w1, a1, a2, a3); + + EIGHT_ROUNDS(w0, a1, a2, a3, 0, + PREP_W(w1, w2, b1, b2, b3), + w0 = __riscv_vsm3me(w2, w1, vl)); + EIGHT_ROUNDS(w1, b1, b2, b3, 4, + PREP_W(w2, w0, a1, a2, a3), + w1 = __riscv_vsm3me(w0, w2, vl)); + EIGHT_ROUNDS(w2, a1, a2, a3, 8, + PREP_W(w0, w1, b1, b2, b3), + w2 = __riscv_vsm3me(w1, w0, vl)); + EIGHT_ROUNDS(w0, b1, b2, b3, 12, + PREP_W(w1, w2, a1, a2, a3), + w0 = __riscv_vsm3me(w2, w1, vl)); + EIGHT_ROUNDS(w1, a1, a2, a3, 16, + PREP_W(w2, w0, b1, b2, b3), + w1 = __riscv_vsm3me(w0, w2, vl)); + EIGHT_ROUNDS(w2, b1, b2, b3, 20, + PREP_W(w0, w1, a1, a2, a3), + w2 = __riscv_vsm3me(w1, w0, vl)); + EIGHT_ROUNDS(w0, a1, a2, a3, 24, + PREP_W(w1, w2, b1, b2, b3), + (void)0); + EIGHT_ROUNDS(w1, b1, b2, b3, 28, (void)0, (void)0); + + v_hash = __riscv_vxor(v_hash, v_hash_work, vl); + + data += 64; + nblocks--; + } + + __riscv_vse32(state, __riscv_vrev8(v_hash, vl), vl); + + clear_vec_regs(); + return 0; +} + +#undef EIGHT_ROUNDS +#undef PREP_W +#undef VEC_T +#undef V8 +#undef V +#undef FN +#undef CAT +#undef CAT_ +#undef SM3_LMUL diff --git a/cipher/sm3.c b/cipher/sm3.c index 6001afff..8e1e71ed 100644 --- a/cipher/sm3.c +++ b/cipher/sm3.c @@ -87,6 +87,15 @@ # endif #endif +/* USE_RISCV_V_CRYPTO indicates whether to enable RISC-V vector cryptography + * extension code. */ +#undef USE_RISCV_V_CRYPTO +#if defined (__riscv) && \ + defined(HAVE_COMPATIBLE_CC_RISCV_VECTOR_INTRINSICS) && \ + defined(HAVE_COMPATIBLE_CC_RISCV_VECTOR_CRYPTO_INTRINSICS) +# define USE_RISCV_V_CRYPTO 1 +#endif + typedef struct { gcry_md_block_ctx_t bctx; @@ -167,6 +176,23 @@ do_sm3_transform_armv8_ce(void *context, const unsigned char *data, } #endif /* USE_ARM_CE */ +#ifdef USE_RISCV_V_CRYPTO +unsigned int _gcry_sm3_riscv_v_check_hw(void); + +unsigned int +_gcry_sm3_transform_riscv_zvksh_zvkb(u32 state[8], + const unsigned char *input_data, + size_t num_blks); + +static unsigned int +do_sm3_transform_riscv_zvksh(void *ctx, const unsigned char *data, + size_t nblks) +{ + SM3_CONTEXT *hd = ctx; + return _gcry_sm3_transform_riscv_zvksh_zvkb (hd->h, data, nblks); +} +#endif /* USE_RISCV_V_CRYPTO */ + static unsigned int transform (void *c, const unsigned char *data, size_t nblks); @@ -211,6 +237,15 @@ sm3_init (void *context, unsigned int flags) if (features & HWF_ARM_SM3) hd->bctx.bwrite = do_sm3_transform_armv8_ce; #endif +#ifdef USE_RISCV_V_CRYPTO + if ((features & HWF_RISCV_IMAFDC) + && (features & HWF_RISCV_B) /* Mandatory in RVA23U64 */ + && (features & HWF_RISCV_V) /* Mandatory in RVA23U64 */ + && (features & HWF_RISCV_ZVKB) /* Mandatory in RVA23U64 (Zvbb) */ + && (features & HWF_RISCV_ZVKSH) /* Optional in RVA23U64 (Zvks) */ + && _gcry_sm3_riscv_v_check_hw()) + hd->bctx.bwrite = do_sm3_transform_riscv_zvksh; +#endif (void)features; } diff --git a/configure.ac b/configure.ac index e32519c9..a6b1f325 100644 --- a/configure.ac +++ b/configure.ac @@ -3038,6 +3038,22 @@ m4_define([GCRY_RISCV_VECTOR_CRYPTO_INTRINSICS_TEST], vuint32m1_t d = __riscv_vghsh_vv_u32m1(a, b, c, vl); __riscv_vse32_v_u32m1(ptr + 0 * vl, d, vl); } + void test_sm3(unsigned int *ptr) + { + int vl = __riscv_vsetvl_e32m2 (8); + vuint32m2_t a = __riscv_vle32_v_u32m2(ptr + 0 * vl, vl); + vuint32m2_t b = __riscv_vle32_v_u32m2(ptr + 1 * vl, vl); + vuint32m2_t c = __riscv_vsm3me_vv_u32m2(a, b, vl); + vuint32m2_t d = __riscv_vsm3c_vi_u32m2(a, c, 31, vl); + vuint32m1_t e, f, g; + __riscv_vse32_v_u32m2(ptr + 0 * vl, d, vl); + vl = __riscv_vsetvl_e32m1 (8); + e = __riscv_vle32_v_u32m1(ptr + 0 * vl, vl); + f = __riscv_vle32_v_u32m1(ptr + 1 * vl, vl); + g = __riscv_vsm3me_vv_u32m1(e, f, vl); + e = __riscv_vsm3c_vi_u32m1(e, g, 31, vl); + __riscv_vse32_v_u32m1(ptr + 0 * vl, e, vl); + } void test_rev8(unsigned int *ptr) { int vl = __riscv_vsetvl_e32m1 (4); @@ -3076,6 +3092,7 @@ m4_define([GCRY_RISCV_VECTOR_CRYPTO_INTRINSICS_TEST], test_aes_key(buf); test_aes_crypt(buf); test_ghash(buf); + test_sm3(buf); test_rev8(buf); test_rotate(buf); test_inline_vec_asm(buf); @@ -4269,6 +4286,11 @@ if test "$found" = "1" ; then GCRYPT_ASM_DIGESTS="$GCRYPT_ASM_DIGESTS sm3-aarch64.lo" GCRYPT_ASM_DIGESTS="$GCRYPT_ASM_DIGESTS sm3-armv8-aarch64-ce.lo" ;; + riscv64-*-*) + # Build with the RISC-V vector cryptography implementation + GCRYPT_ASM_DIGESTS="$GCRYPT_ASM_DIGESTS sm3-riscv-zvksh-zvkb.lo" + GCRYPT_ASM_DIGESTS="$GCRYPT_ASM_DIGESTS sm3-riscv-zvksh-zvkb-m1.lo" + ;; esac fi diff --git a/doc/gcrypt.texi b/doc/gcrypt.texi index 8497bf4d..613920d8 100644 --- a/doc/gcrypt.texi +++ b/doc/gcrypt.texi @@ -623,6 +623,7 @@ are @item riscv-zvkned @item riscv-zvknha @item riscv-zvknhb + at item riscv-zvksh @end table To disable a feature for all processes using Libgcrypt 1.6 or newer, diff --git a/src/g10lib.h b/src/g10lib.h index 0b4ca0c7..8a2d706c 100644 --- a/src/g10lib.h +++ b/src/g10lib.h @@ -288,6 +288,7 @@ char **_gcry_strtokenize (const char *string, const char *delim); #define HWF_RISCV_ZVKNED (1 << 7) #define HWF_RISCV_ZVKNHA (1 << 8) #define HWF_RISCV_ZVKNHB (1 << 9) +#define HWF_RISCV_ZVKSH (1 << 10) #endif diff --git a/src/hwf-riscv.c b/src/hwf-riscv.c index 43a9cab4..adec2b02 100644 --- a/src/hwf-riscv.c +++ b/src/hwf-riscv.c @@ -197,6 +197,7 @@ detect_riscv_at_hwcap(void) #define HWF_RISCV_HWPROBE_EXT_ZVKNED (1U << 21) #define HWF_RISCV_HWPROBE_EXT_ZVKNHA (1U << 22) #define HWF_RISCV_HWPROBE_EXT_ZVKNHB (1U << 23) +#define HWF_RISCV_HWPROBE_EXT_ZVKSH (1U << 25) #define HWF_RISCV_HWPROBE_EXT_ZICOND (U64_C(1) << 35) #define HWF_RISCV_HWPROBE_IMA_FDC (HWF_RISCV_HWPROBE_IMA_FD \ @@ -227,6 +228,7 @@ static const struct hwprobe_feature_map_s hwprobe_features[] = { HWF_RISCV_HWPROBE_EXT_ZVKNED, HWF_RISCV_ZVKNED }, { HWF_RISCV_HWPROBE_EXT_ZVKNHA, HWF_RISCV_ZVKNHA }, { HWF_RISCV_HWPROBE_EXT_ZVKNHB, HWF_RISCV_ZVKNHB }, + { HWF_RISCV_HWPROBE_EXT_ZVKSH, HWF_RISCV_ZVKSH }, }; static int diff --git a/src/hwfeatures.c b/src/hwfeatures.c index 08281feb..2f836079 100644 --- a/src/hwfeatures.c +++ b/src/hwfeatures.c @@ -127,6 +127,7 @@ static struct { HWF_RISCV_ZVKNED, "riscv-zvkned" }, { HWF_RISCV_ZVKNHA, "riscv-zvknha" }, { HWF_RISCV_ZVKNHB, "riscv-zvknhb" }, + { HWF_RISCV_ZVKSH, "riscv-zvksh" }, #endif }; -- 2.53.0 From jussi.kivilinna at iki.fi Sun Oct 11 19:23:18 2026 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sun, 11 Oct 2026 20:23:18 +0300 Subject: [PATCH 09/12] rijndael-riscv-zvkned: optimize OCB offset generation In-Reply-To: <20261011172321.997183-1-jussi.kivilinna@iki.fi> References: <20261011172321.997183-1-jussi.kivilinna@iki.fi> Message-ID: <20261011172321.997183-9-jussi.kivilinna@iki.fi> * cipher/rijndael-riscv-zvkned.c (xor_broadcast128_u32m1) (xor_broadcast128_u32m2, xor_broadcast128_u32m4) (xor_fold_u32m4_u32m1_tu, xor_fold_u32m2_u32m1_tu) (xor_fold_u32m1_u32m1_tu, lo128_u32m2, lo128_u32m4, set_last_block_m1) (set_last_block_m2, set_last_block_m4, ocb_offsets_m2) (ocb_offset_table_m4, ocb_group_delta_top, ocb_table_offsets_m1) (ocb_table_offsets_m2, ocb_table_offsets_m4): New. (AES_CRYPT_NO_RK0): New. (AES_CRYPT): Use AES_CRYPT_NO_RK0. (ocb_offsets_m1): Take maximum number of blocks per LMUL=1 instead of vector length. (aes_riscv_ocb_crypt): Use offset table for LMUL=4 processing; process remaining blocks as single chunk with smallest LMUL that fits; merge first round key to offsets and last round key; accumulate checksum directly to LMUL=1. -- Previously OCB offsets for LMUL=4 processing were generated one block at a time with L lookup and vslideup. Now offsets for 4xVLEN block group aligned to block number are generated with single vaesz broadcast-xor of current offset over precomputed table and only last block of group needs L_{ntz} lookup. Unaligned head is handled with same table. Blocks remaining after LMUL=4 processing are handled as single chunk with smallest LMUL that fits. This chunk uses LMUL=4 offset table when it was generated and per-block offsets otherwise. First round key is merged to offsets and to last round key, which removes one AES instruction from processing loops. Benchmark on SpacemiT K3, X100 core (out-of-order, VLEN=256): Before | AES128 cycles/byte AES256 cycles/byte OCB enc | 1.10 c/B 1.35 c/B OCB dec | 1.10 c/B 1.35 c/B OCB auth | 1.05 c/B 1.28 c/B After | AES128 cycles/byte AES256 cycles/byte OCB enc | 0.835 c/B 1.09 c/B OCB dec | 0.860 c/B 1.12 c/B OCB auth | 0.835 c/B 1.09 c/B OCB: AES128 ~1.3x, AES256 ~1.2x faster Benchmark on SpacemiT K3, A100 core (in-order, VLEN=1024): Before | AES128 cycles/byte AES256 cycles/byte OCB enc | 3.70 c/B 3.91 c/B OCB dec | 3.66 c/B 3.38 c/B OCB auth | 3.63 c/B 3.90 c/B After | AES128 cycles/byte AES256 cycles/byte OCB enc | 0.971 c/B 1.20 c/B OCB dec | 0.877 c/B 1.12 c/B OCB auth | 0.836 c/B 1.10 c/B OCB: AES128 ~3.8x, AES256 ~3.3x faster Signed-off-by: Jussi Kivilinna --- cipher/rijndael-riscv-zvkned.c | 550 ++++++++++++++++++++++++++------- 1 file changed, 441 insertions(+), 109 deletions(-) diff --git a/cipher/rijndael-riscv-zvkned.c b/cipher/rijndael-riscv-zvkned.c index 45fa4d67..80338509 100644 --- a/cipher/rijndael-riscv-zvkned.c +++ b/cipher/rijndael-riscv-zvkned.c @@ -85,6 +85,51 @@ broadcast128_u32m1_u32m1(vuint32m1_t vec, size_t vl_u32) #endif } +static ASM_FUNC_ATTR_INLINE vuint32m1_t +xor_broadcast128_u32m1(vuint32m1_t vdst, vuint32m1_t vec, size_t vl_u32) +{ +#ifdef HAVE_BROKEN_VAES_VS_INTRINSIC + asm ( "vsetvli zero,%[vl],e32,m1,ta,ma;\n\t" + "vaesz.vs %[dst],%[src];\n\t" + : [dst] "+vr" (vdst) + : [vl] "r" (vl_u32), [src] "vr" (vec) + : "vl", "vtype"); + return vdst; +#else + return __riscv_vaesz_vs_u32m1_u32m1(vdst, vec, vl_u32); +#endif +} + +static ASM_FUNC_ATTR_INLINE vuint32m2_t +xor_broadcast128_u32m2(vuint32m2_t vdst, vuint32m1_t vec, size_t vl_u32) +{ +#ifdef HAVE_BROKEN_VAES_VS_INTRINSIC + asm ( "vsetvli zero,%[vl],e32,m2,ta,ma;\n\t" + "vaesz.vs %[dst],%[src];\n\t" + : [dst] "+vr" (vdst) + : [vl] "r" (vl_u32), [src] "vr" (vec) + : "vl", "vtype"); + return vdst; +#else + return __riscv_vaesz_vs_u32m1_u32m2(vdst, vec, vl_u32); +#endif +} + +static ASM_FUNC_ATTR_INLINE vuint32m4_t +xor_broadcast128_u32m4(vuint32m4_t vdst, vuint32m1_t vec, size_t vl_u32) +{ +#ifdef HAVE_BROKEN_VAES_VS_INTRINSIC + asm ( "vsetvli zero,%[vl],e32,m4,ta,ma;\n\t" + "vaesz.vs %[dst],%[src];\n\t" + : [dst] "+vr" (vdst) + : [vl] "r" (vl_u32), [src] "vr" (vec) + : "vl", "vtype"); + return vdst; +#else + return __riscv_vaesz_vs_u32m1_u32m4(vdst, vec, vl_u32); +#endif +} + static ASM_FUNC_ATTR_INLINE vuint32m1_t unaligned_load_u32m1(const void *ptr, size_t vl_u32) { @@ -145,6 +190,53 @@ unaligned_store_u32m4(void *ptr, vuint32m4_t vec, size_t vl_u32) #endif } +/* XOR first VL elements of VEC into ACC, folding LMUL=4 groups to LMUL=1. */ +static ASM_FUNC_ATTR_INLINE vuint32m1_t +xor_fold_u32m4_u32m1_tu(vuint32m1_t acc, vuint32m4_t vec, size_t vl, + size_t vl_m1) +{ + size_t vl0 = vl < vl_m1 ? vl : vl_m1; + size_t vl1, vl2, vl3; + + vl -= vl0; + vl1 = vl < vl_m1 ? vl : vl_m1; + vl -= vl1; + vl2 = vl < vl_m1 ? vl : vl_m1; + vl3 = vl - vl2; + + acc = __riscv_vxor_vv_u32m1_tu(acc, acc, __riscv_vget_v_u32m4_u32m1(vec, 0), + vl0); + acc = __riscv_vxor_vv_u32m1_tu(acc, acc, __riscv_vget_v_u32m4_u32m1(vec, 1), + vl1); + acc = __riscv_vxor_vv_u32m1_tu(acc, acc, __riscv_vget_v_u32m4_u32m1(vec, 2), + vl2); + acc = __riscv_vxor_vv_u32m1_tu(acc, acc, __riscv_vget_v_u32m4_u32m1(vec, 3), + vl3); + return acc; +} + +static ASM_FUNC_ATTR_INLINE vuint32m1_t +xor_fold_u32m2_u32m1_tu(vuint32m1_t acc, vuint32m2_t vec, size_t vl, + size_t vl_m1) +{ + size_t vl0 = vl < vl_m1 ? vl : vl_m1; + size_t vl1 = vl - vl0; + + acc = __riscv_vxor_vv_u32m1_tu(acc, acc, __riscv_vget_v_u32m2_u32m1(vec, 0), + vl0); + acc = __riscv_vxor_vv_u32m1_tu(acc, acc, __riscv_vget_v_u32m2_u32m1(vec, 1), + vl1); + return acc; +} + +static ASM_FUNC_ATTR_INLINE vuint32m1_t +xor_fold_u32m1_u32m1_tu(vuint32m1_t acc, vuint32m1_t vec, size_t vl, + size_t vl_m1) +{ + (void)vl_m1; + return __riscv_vxor_vv_u32m1_tu(acc, acc, vec, vl); +} + static ASM_FUNC_ATTR_INLINE size_t nblocks_to_nblocks_per_m1(size_t max_blocks_m1, size_t nblocks) { @@ -479,10 +571,9 @@ _gcry_aes_riscv_zvkned_prepare_decryption(RIJNDAEL_context *ctx) }) #ifdef HAVE_BROKEN_VAES_VS_INTRINSIC -#define AES_CRYPT(e_d, mx, load_key, rk, nrounds, blk, vlen) \ +#define AES_CRYPT_NO_RK0(e_d, mx, load_key, rk, nrounds, blk, vlen) \ ({ \ asm ( "vsetvli zero,%[vl],e32,"#mx",ta,ma;\n\t" \ - "vaesz.vs %[block],%[rk0];\n\t" \ "vaes"#e_d"m.vs %[block],%[rk1];\n\t" \ "vaes"#e_d"m.vs %[block],%[rk2];\n\t" \ "vaes"#e_d"m.vs %[block],%[rk3];\n\t" \ @@ -494,10 +585,9 @@ _gcry_aes_riscv_zvkned_prepare_decryption(RIJNDAEL_context *ctx) "vaes"#e_d"m.vs %[block],%[rk9];\n\t" \ : [block] "+vr" (blk) \ : [vl] "r" (vlen), \ - [rk0] "vr" (rk0), [rk1] "vr" (rk1), [rk2] "vr" (rk2), \ - [rk3] "vr" (rk3), [rk4] "vr" (rk4), [rk5] "vr" (rk5), \ - [rk6] "vr" (rk6), [rk7] "vr" (rk7), [rk8] "vr" (rk8), \ - [rk9] "vr" (rk9) \ + [rk1] "vr" (rk1), [rk2] "vr" (rk2), [rk3] "vr" (rk3), \ + [rk4] "vr" (rk4), [rk5] "vr" (rk5), [rk6] "vr" (rk6), \ + [rk7] "vr" (rk7), [rk8] "vr" (rk8), [rk9] "vr" (rk9) \ : "vl", "vtype"); \ if (LIKELY((nrounds) >= 12)) \ { \ @@ -533,10 +623,18 @@ _gcry_aes_riscv_zvkned_prepare_decryption(RIJNDAEL_context *ctx) : [vl] "r" (vlen), [rk_last] "vr" (rk_last) \ : "vl", "vtype"); \ }) +#define AES_CRYPT(e_d, mx, load_key, rk, nrounds, blk, vlen) \ + ({ \ + asm ( "vsetvli zero,%[vl],e32,"#mx",ta,ma;\n\t" \ + "vaesz.vs %[block],%[rk0];\n\t" \ + : [block] "+vr" (blk) \ + : [vl] "r" (vlen), [rk0] "vr" (rk0) \ + : "vl", "vtype"); \ + AES_CRYPT_NO_RK0(e_d, mx, load_key, rk, nrounds, blk, vlen); \ + }) #else -#define AES_CRYPT(e_d, mx, load_key, rk, nrounds, block, vl) \ +#define AES_CRYPT_NO_RK0(e_d, mx, load_key, rk, nrounds, block, vl) \ ({ \ - (block) = __riscv_vaesz_vs_u32m1_u32##mx((block), rk0, (vl)); \ (block) = __riscv_vaes##e_d##m_vs_u32m1_u32##mx((block), rk1, (vl)); \ (block) = __riscv_vaes##e_d##m_vs_u32m1_u32##mx((block), rk2, (vl)); \ (block) = __riscv_vaes##e_d##m_vs_u32m1_u32##mx((block), rk3, (vl)); \ @@ -566,6 +664,11 @@ _gcry_aes_riscv_zvkned_prepare_decryption(RIJNDAEL_context *ctx) } \ (block) = __riscv_vaes##e_d##f_vs_u32m1_u32##mx((block), rk_last, (vl)); \ }) +#define AES_CRYPT(e_d, mx, load_key, rk, nrounds, block, vl) \ + ({ \ + (block) = __riscv_vaesz_vs_u32m1_u32##mx((block), rk0, (vl)); \ + AES_CRYPT_NO_RK0(e_d, mx, load_key, rk, nrounds, block, vl); \ + }) #endif unsigned int ASM_FUNC_ATTR_NOINLINE FUNC_ATTR_OPT_O2 @@ -1331,12 +1434,45 @@ _gcry_aes_riscv_zvkned_cbc_dec (void *context, unsigned char *iv_arg, clear_vec_regs(); } +static ASM_FUNC_ATTR_INLINE vuint32m1_t +lo128_u32m2 (vuint32m2_t v) +{ + return __riscv_vlmul_trunc_v_u32m2_u32m1(v); +} + +static ASM_FUNC_ATTR_INLINE vuint32m1_t +lo128_u32m4 (vuint32m4_t v) +{ + return __riscv_vlmul_trunc_v_u32m4_u32m1(v); +} + +static ASM_FUNC_ATTR_INLINE vuint32m1_t +set_last_block_m1 (vuint32m1_t v, vuint32m1_t blk, size_t vl_m1) +{ + return __riscv_vslideup_vx_u32m1(v, blk, vl_m1 - BLOCKSIZE / 4, vl_m1); +} + +static ASM_FUNC_ATTR_INLINE vuint32m2_t +set_last_block_m2 (vuint32m2_t v, vuint32m1_t blk, size_t vl_m1) +{ + vuint32m1_t hi = __riscv_vget_v_u32m2_u32m1(v, 1); + return __riscv_vset_v_u32m1_u32m2(v, 1, set_last_block_m1(hi, blk, vl_m1)); +} + +static ASM_FUNC_ATTR_INLINE vuint32m4_t +set_last_block_m4 (vuint32m4_t v, vuint32m1_t blk, size_t vl_m1) +{ + vuint32m1_t hi = __riscv_vget_v_u32m4_u32m1(v, 3); + return __riscv_vset_v_u32m1_u32m4(v, 3, set_last_block_m1(hi, blk, vl_m1)); +} + static ASM_FUNC_ATTR_INLINE vuint32m1_t ocb_offsets_m1 (gcry_cipher_hd_t c, u64 *np, vuint32m1_t *ivp, - vuint32m1_t offsets, size_t nblocks, size_t vl_m1) + vuint32m1_t offsets, size_t nblocks, size_t max_blocks_m1) { vuint32m1_t iv = *ivp; size_t blk_vl32 = BLOCKSIZE / 4; + size_t vl_m1 = max_blocks_m1 * 4; size_t k = 0; do @@ -1353,6 +1489,184 @@ ocb_offsets_m1 (gcry_cipher_hd_t c, u64 *np, vuint32m1_t *ivp, return offsets; } +static ASM_FUNC_ATTR_INLINE vuint32m2_t +ocb_offsets_m2 (gcry_cipher_hd_t c, u64 *np, vuint32m1_t *ivp, + vuint32m2_t offsets, size_t nblocks, size_t max_blocks_m1) +{ + size_t nblks0 = nblocks < max_blocks_m1 ? nblocks : max_blocks_m1; + vuint32m1_t offs; + + offs = ocb_offsets_m1(c, np, ivp, __riscv_vget_v_u32m2_u32m1(offsets, 0), + nblks0, max_blocks_m1); + offsets = __riscv_vset_v_u32m1_u32m2(offsets, 0, offs); + if (nblocks > nblks0) + { + offs = ocb_offsets_m1(c, np, ivp, __riscv_vget_v_u32m2_u32m1(offsets, 1), + nblocks - nblks0, max_blocks_m1); + offsets = __riscv_vset_v_u32m1_u32m2(offsets, 1, offs); + } + + return offsets; +} + +/* Lane i of returned table is L_{ntz(1)} xor ... xor L_{ntz(i+1)}, that is, + * offset delta of block i+1 from start of max_blocks aligned group. Lanes + * 0 to max_blocks-2 are also valid for any smaller power-of-two group. Last + * lane of group depends on block number and is set separately. */ +static ASM_FUNC_ATTR_INLINE vuint32m4_t +ocb_offset_table_m4 (gcry_cipher_hd_t c, size_t max_blocks, size_t vl) +{ + vuint32m4_t zero = __riscv_vmv_v_x_u32m4(0, vl); + vuint32m4_t table = zero; + vuint32m4_t gray; + size_t blk_vl32 = BLOCKSIZE / 4; + size_t i; + + /* Delta is XOR of L_j for each bit j set in gray(i+1). */ + gray = __riscv_vsrl_vx_u32m4(__riscv_vid_v_u32m4(vl), 2, vl); + gray = __riscv_vadd_vx_u32m4(gray, 1, vl); + gray = __riscv_vxor_vv_u32m4(gray, __riscv_vsrl_vx_u32m4(gray, 1, vl), vl); + + for (i = 0; ((size_t)1 << i) < max_blocks; i++) + { + vbool8_t m = __riscv_vmsne_vx_u32m4_b8( + __riscv_vand_vx_u32m4(gray, 1U << i, vl), 0, vl); + vuint32m1_t l = + __riscv_vle32_v_u32m1((const void *)c->u_mode.ocb.L[i], blk_vl32); + + table = __riscv_vxor_vv_u32m4_mu(m, table, table, + xor_broadcast128_u32m4(zero, l, vl), vl); + } + + return table; +} + +/* Offset delta from start of max_blocks aligned group to last block of + * group, excluding L_{ntz(group end)}. */ +static ASM_FUNC_ATTR_INLINE vuint32m1_t +ocb_group_delta_top (gcry_cipher_hd_t c, size_t max_blocks) +{ + size_t blk_vl32 = BLOCKSIZE / 4; + + if (max_blocks < 2) + return __riscv_vmv_v_x_u32m1(0, blk_vl32); + + return __riscv_vle32_v_u32m1( + (const void *)c->u_mode.ocb.L[__builtin_ctzl(max_blocks) - 1], + blk_vl32); +} + +/* Offsets for CURR_NBLKS blocks after block number *NP. Blocks must not + * cross max_blocks aligned block number. */ +static ASM_FUNC_ATTR_INLINE vuint32m4_t +ocb_table_offsets_m4 (gcry_cipher_hd_t c, u64 *np, vuint32m1_t *ivp, + vuint32m4_t table, vuint32m1_t l_top, size_t max_blocks, + size_t curr_nblks, size_t vl_m1) +{ + size_t blk_vl32 = BLOCKSIZE / 4; + size_t pos = *np & (max_blocks - 1); + size_t vl = curr_nblks * 4; + vuint32m1_t base = *ivp; + vuint32m1_t iv; + vuint32m4_t offs; + + if (UNLIKELY(pos)) + { + /* Rewind base to offset of previous aligned block. */ + base = __riscv_vxor_vv_u32m1( + base, + lo128_u32m4(__riscv_vslidedown_vx_u32m4(table, (pos - 1) * 4, + blk_vl32)), + blk_vl32); + } + offs = xor_broadcast128_u32m4(table, base, (pos * 4) + vl); + *np += curr_nblks; + if (LIKELY((*np & (max_blocks - 1)) == 0)) + { + const void *l = ocb_get_l(c, *np); + + iv = __riscv_vxor_vv_u32m1(base, l_top, blk_vl32); + iv = __riscv_vxor_vv_u32m1(iv, __riscv_vle32_v_u32m1(l, blk_vl32), + blk_vl32); + offs = set_last_block_m4(offs, iv, vl_m1); + } + else + { + iv = lo128_u32m4(__riscv_vslidedown_vx_u32m4( + offs, (pos + curr_nblks - 1) * 4, blk_vl32)); + } + if (UNLIKELY(pos)) + offs = __riscv_vslidedown_vx_u32m4(offs, pos * 4, vl); + + *ivp = iv; + return offs; +} + +/* Offsets for first CURR_NBLKS blocks of max_blocks group. Block number *NP + * must be max_blocks aligned. */ +static ASM_FUNC_ATTR_INLINE vuint32m2_t +ocb_table_offsets_m2 (gcry_cipher_hd_t c, u64 *np, vuint32m1_t *ivp, + vuint32m2_t table, vuint32m1_t l_top, size_t max_blocks, + size_t curr_nblks, size_t vl_m1) +{ + size_t blk_vl32 = BLOCKSIZE / 4; + size_t vl = curr_nblks * 4; + vuint32m1_t iv = *ivp; + vuint32m2_t offs; + + offs = xor_broadcast128_u32m2(table, iv, vl); + *np += curr_nblks; + if (curr_nblks == max_blocks) + { + const void *l = ocb_get_l(c, *np); + + iv = __riscv_vxor_vv_u32m1(iv, l_top, blk_vl32); + iv = __riscv_vxor_vv_u32m1(iv, __riscv_vle32_v_u32m1(l, blk_vl32), + blk_vl32); + offs = set_last_block_m2(offs, iv, vl_m1); + } + else + { + iv = lo128_u32m2(__riscv_vslidedown_vx_u32m2(offs, (curr_nblks - 1) * 4, + blk_vl32)); + } + + *ivp = iv; + return offs; +} + +/* Offsets for first CURR_NBLKS blocks of max_blocks group. Block number *NP + * must be max_blocks aligned. */ +static ASM_FUNC_ATTR_INLINE vuint32m1_t +ocb_table_offsets_m1 (gcry_cipher_hd_t c, u64 *np, vuint32m1_t *ivp, + vuint32m1_t table, vuint32m1_t l_top, size_t max_blocks, + size_t curr_nblks, size_t vl_m1) +{ + size_t blk_vl32 = BLOCKSIZE / 4; + size_t vl = curr_nblks * 4; + vuint32m1_t iv = *ivp; + vuint32m1_t offs; + + offs = xor_broadcast128_u32m1(table, iv, vl); + *np += curr_nblks; + if (curr_nblks == max_blocks) + { + const void *l = ocb_get_l(c, *np); + + iv = __riscv_vxor_vv_u32m1(iv, l_top, blk_vl32); + iv = __riscv_vxor_vv_u32m1(iv, __riscv_vle32_v_u32m1(l, blk_vl32), + blk_vl32); + offs = set_last_block_m1(offs, iv, vl_m1); + } + else + { + iv = __riscv_vslidedown_vx_u32m1(offs, (curr_nblks - 1) * 4, blk_vl32); + } + + *ivp = iv; + return offs; +} + static ASM_FUNC_ATTR_INLINE FUNC_ATTR_OPT_O2 size_t aes_riscv_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg, const void *inbuf_arg, size_t nblocks, int encrypt) @@ -1368,11 +1682,11 @@ aes_riscv_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg, size_t max_blocks_m2 = max_blocks_m1 * 2; size_t max_blocks_m4 = max_blocks_m1 * 4; size_t vl_m1 = max_blocks_m1 * 4; - size_t vl_m2 = max_blocks_m2 * 4; size_t vl_m4 = max_blocks_m4 * 4; + vuint32m4_t table_m4 = __riscv_vundefined_u32m4(); + int have_table = 0; vuint32m1_t iv; vuint32m1_t ctr; - vuint32m1_t offs_m1 = __riscv_vundefined_u32m1(); size_t h; u64 n; ROUND_KEY_VARIABLES; @@ -1384,187 +1698,201 @@ aes_riscv_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg, ctx->decryption_prepared = 1; } + n = auth ? c->u_mode.ocb.aad_nblocks : c->u_mode.ocb.data_nblocks; + + if (nblocks > max_blocks_m2) + { + table_m4 = ocb_offset_table_m4(c, max_blocks_m4, vl_m4); + have_table = 1; + } + PRELOAD_ROUND_KEYS (rk, rounds); /* Preload Offset and Checksum */ if (auth) { - n = c->u_mode.ocb.aad_nblocks; iv = __riscv_vle32_v_u32m1((void *)c->u_mode.ocb.aad_offset, blk_vl32); ctr = __riscv_vle32_v_u32m1((void *)c->u_mode.ocb.aad_sum, blk_vl32); } else { - n = c->u_mode.ocb.data_nblocks; iv = __riscv_vle32_v_u32m1((void *)c->u_iv.iv, blk_vl32); ctr = __riscv_vle32_v_u32m1((void *)c->u_ctr.ctr, blk_vl32); } ctr = __riscv_vmv_v_v_u32m1_tu(__riscv_vmv_v_x_u32m1(0, vl_m1), ctr, blk_vl32); - if (nblocks >= max_blocks_m4) + /* First round key is merged to offsets. For encryption and decryption, + * it is cancelled out from output by merging it also to last round key. */ + iv = __riscv_vxor_vv_u32m1(iv, rk0, blk_vl32); + if (!auth) + rk_last = __riscv_vxor_vv_u32m1(rk_last, rk0, blk_vl32); + + if (have_table) { - vuint32m4_t offs_m4 = __riscv_vundefined_u32m4(); - vuint32m4_t ctr_m4 = - __riscv_vset_v_u32m1_u32m4(__riscv_vmv_v_x_u32m4(0, vl_m4), 0, ctr); - vuint32m2_t ctr_m2; + vuint32m1_t l_top = ocb_group_delta_top(c, max_blocks_m4); + /* Process blocks in chunks that end at max_blocks_m4 aligned block + * number or at end of input. */ do { - vuint32m4_t data_blks = unaligned_load_u32m4(inbuf, vl_m4); + size_t curr_nblks = max_blocks_m4 - (n & (max_blocks_m4 - 1)); + vuint32m4_t data_blks, offs; + size_t vl; + + curr_nblks = curr_nblks < nblocks ? curr_nblks : nblocks; + vl = curr_nblks * 4; + + data_blks = unaligned_load_u32m4(inbuf, vl); if (encrypt > 0) { /* Checksum_i = Checksum_{i-1} xor P_i */ - ctr_m4 = __riscv_vxor_vv_u32m4(ctr_m4, data_blks, vl_m4); + ctr = xor_fold_u32m4_u32m1_tu(ctr, data_blks, vl, vl_m1); } /* Offset_i = Offset_{i-1} xor L_{ntz(i)} */ - offs_m1 = ocb_offsets_m1(c, &n, &iv, offs_m1, max_blocks_m1, vl_m1); - offs_m4 = __riscv_vset_v_u32m1_u32m4(offs_m4, 0, offs_m1); - offs_m1 = ocb_offsets_m1(c, &n, &iv, offs_m1, max_blocks_m1, vl_m1); - offs_m4 = __riscv_vset_v_u32m1_u32m4(offs_m4, 1, offs_m1); - offs_m1 = ocb_offsets_m1(c, &n, &iv, offs_m1, max_blocks_m1, vl_m1); - offs_m4 = __riscv_vset_v_u32m1_u32m4(offs_m4, 2, offs_m1); - offs_m1 = ocb_offsets_m1(c, &n, &iv, offs_m1, max_blocks_m1, vl_m1); - offs_m4 = __riscv_vset_v_u32m1_u32m4(offs_m4, 3, offs_m1); + offs = ocb_table_offsets_m4(c, &n, &iv, table_m4, l_top, + max_blocks_m4, curr_nblks, vl_m1); /* P_i = Offset_i xor CIPHER(K, C_i xor Offset_i) */ - data_blks = __riscv_vxor_vv_u32m4(offs_m4, data_blks, vl_m4); + data_blks = __riscv_vxor_vv_u32m4(offs, data_blks, vl); if (encrypt) - AES_CRYPT(e, m4, LOAD_KEY, rk, rounds, data_blks, vl_m4); + AES_CRYPT_NO_RK0(e, m4, LOAD_KEY, rk, rounds, data_blks, vl); else - AES_CRYPT(d, m4, LOAD_KEY, rk, rounds, data_blks, vl_m4); + AES_CRYPT_NO_RK0(d, m4, LOAD_KEY, rk, rounds, data_blks, vl); if (!auth) { - data_blks = __riscv_vxor_vv_u32m4(offs_m4, data_blks, vl_m4); + data_blks = __riscv_vxor_vv_u32m4(offs, data_blks, vl); - unaligned_store_u32m4(outbuf, data_blks, vl_m4); - outbuf += max_blocks_m4 * BLOCKSIZE; + unaligned_store_u32m4(outbuf, data_blks, vl); + outbuf += curr_nblks * BLOCKSIZE; } if (encrypt <= 0) { /* Checksum_i = Checksum_{i-1} xor P_i */ - ctr_m4 = __riscv_vxor_vv_u32m4(ctr_m4, data_blks, vl_m4); + ctr = xor_fold_u32m4_u32m1_tu(ctr, data_blks, vl, vl_m1); } - inbuf += max_blocks_m4 * BLOCKSIZE; - nblocks -= max_blocks_m4; + inbuf += curr_nblks * BLOCKSIZE; + nblocks -= curr_nblks; } - while (nblocks >= max_blocks_m4); - - /* Checksum_i = Checksum_{i-1} xor P_i */ - ctr_m2 = __riscv_vxor_vv_u32m2(__riscv_vget_v_u32m4_u32m2(ctr_m4, 0), - __riscv_vget_v_u32m4_u32m2(ctr_m4, 1), - vl_m2); - ctr = __riscv_vxor_vv_u32m1(__riscv_vget_v_u32m2_u32m1(ctr_m2, 0), - __riscv_vget_v_u32m2_u32m1(ctr_m2, 1), - vl_m1); + while (nblocks > max_blocks_m2); } + /* Remaining blocks fit in single LMUL=2 or LMUL=1 chunk. With table, block + * number is now max_blocks_m4 aligned. */ if (nblocks) - PRELOAD_ROUND_KEYS_RK10_13 (rk, rounds); - - if (nblocks && nblocks >= max_blocks_m2) { - vuint32m2_t offs_m2 = __riscv_vundefined_u32m2(); - vuint32m2_t ctr_m2 = - __riscv_vset_v_u32m1_u32m2(__riscv_vmv_v_x_u32m2(0, vl_m2), 0, ctr); + PRELOAD_ROUND_KEYS_RK10_13 (rk, rounds); - do + if (nblocks > max_blocks_m1) { - vuint32m2_t data_blks = unaligned_load_u32m2(inbuf, vl_m2); + size_t vl = nblocks * 4; + vuint32m2_t data_blks, offs; + + data_blks = unaligned_load_u32m2(inbuf, vl); if (encrypt > 0) { /* Checksum_i = Checksum_{i-1} xor P_i */ - ctr_m2 = __riscv_vxor_vv_u32m2(ctr_m2, data_blks, vl_m2); + ctr = xor_fold_u32m2_u32m1_tu(ctr, data_blks, vl, vl_m1); } /* Offset_i = Offset_{i-1} xor L_{ntz(i)} */ - offs_m1 = ocb_offsets_m1(c, &n, &iv, offs_m1, max_blocks_m1, vl_m1); - offs_m2 = __riscv_vset_v_u32m1_u32m2(offs_m2, 0, offs_m1); - offs_m1 = ocb_offsets_m1(c, &n, &iv, offs_m1, max_blocks_m1, vl_m1); - offs_m2 = __riscv_vset_v_u32m1_u32m2(offs_m2, 1, offs_m1); + if (have_table) + { + vuint32m2_t table_m2 = + __riscv_vlmul_trunc_v_u32m4_u32m2(table_m4); + vuint32m1_t l_top = ocb_group_delta_top(c, max_blocks_m2); + + offs = ocb_table_offsets_m2(c, &n, &iv, table_m2, l_top, + max_blocks_m2, nblocks, vl_m1); + } + else + { + offs = ocb_offsets_m2(c, &n, &iv, __riscv_vundefined_u32m2(), + nblocks, max_blocks_m1); + } /* P_i = Offset_i xor CIPHER(K, C_i xor Offset_i) */ - data_blks = __riscv_vxor_vv_u32m2(offs_m2, data_blks, vl_m2); + data_blks = __riscv_vxor_vv_u32m2(offs, data_blks, vl); if (encrypt) - AES_CRYPT(e, m2, GET_PRELOADED_KEY, rk, rounds, data_blks, vl_m2); + AES_CRYPT_NO_RK0(e, m2, GET_PRELOADED_KEY, rk, rounds, data_blks, + vl); else - AES_CRYPT(d, m2, GET_PRELOADED_KEY, rk, rounds, data_blks, vl_m2); + AES_CRYPT_NO_RK0(d, m2, GET_PRELOADED_KEY, rk, rounds, data_blks, + vl); if (!auth) { - data_blks = __riscv_vxor_vv_u32m2(offs_m2, data_blks, vl_m2); + data_blks = __riscv_vxor_vv_u32m2(offs, data_blks, vl); - unaligned_store_u32m2(outbuf, data_blks, vl_m2); - outbuf += max_blocks_m2 * BLOCKSIZE; + unaligned_store_u32m2(outbuf, data_blks, vl); } if (encrypt <= 0) { /* Checksum_i = Checksum_{i-1} xor P_i */ - ctr_m2 = __riscv_vxor_vv_u32m2(ctr_m2, data_blks, vl_m2); + ctr = xor_fold_u32m2_u32m1_tu(ctr, data_blks, vl, vl_m1); } - - inbuf += max_blocks_m2 * BLOCKSIZE; - nblocks -= max_blocks_m2; } - while (nblocks >= max_blocks_m2); - - /* Checksum_i = Checksum_{i-1} xor P_i */ - ctr = __riscv_vxor_vv_u32m1(__riscv_vget_v_u32m2_u32m1(ctr_m2, 0), - __riscv_vget_v_u32m2_u32m1(ctr_m2, 1), - vl_m1); - } - - while (nblocks) - { - size_t curr_nblks = nblocks_to_nblocks_per_m1(max_blocks_m1, nblocks); - vuint32m1_t data_blks; + else + { + size_t vl = nblocks * 4; + vuint32m1_t data_blks, offs; - vl_m1 = curr_nblks * 4; + data_blks = unaligned_load_u32m1(inbuf, vl); - data_blks = unaligned_load_u32m1(inbuf, vl_m1); + if (encrypt > 0) + { + /* Checksum_i = Checksum_{i-1} xor P_i */ + ctr = xor_fold_u32m1_u32m1_tu(ctr, data_blks, vl, vl_m1); + } - if (encrypt > 0) - { - /* Checksum_i = Checksum_{i-1} xor P_i */ - ctr = __riscv_vxor_vv_u32m1_tu(ctr, ctr, data_blks, vl_m1); - } + /* Offset_i = Offset_{i-1} xor L_{ntz(i)} */ + if (have_table) + { + vuint32m1_t table_m1 = + __riscv_vlmul_trunc_v_u32m4_u32m1(table_m4); + vuint32m1_t l_top = ocb_group_delta_top(c, max_blocks_m1); - /* Offset_i = Offset_{i-1} xor L_{ntz(i)} */ - offs_m1 = ocb_offsets_m1(c, &n, &iv, offs_m1, curr_nblks, vl_m1); + offs = ocb_table_offsets_m1(c, &n, &iv, table_m1, l_top, + max_blocks_m1, nblocks, vl_m1); + } + else + { + offs = ocb_offsets_m1(c, &n, &iv, __riscv_vundefined_u32m1(), + nblocks, max_blocks_m1); + } - /* P_i = Offset_i xor CIPHER(K, C_i xor Offset_i) */ - data_blks = __riscv_vxor_vv_u32m1(offs_m1, data_blks, vl_m1); + /* P_i = Offset_i xor CIPHER(K, C_i xor Offset_i) */ + data_blks = __riscv_vxor_vv_u32m1(offs, data_blks, vl); - if (encrypt) - AES_CRYPT(e, m1, GET_PRELOADED_KEY, rk, rounds, data_blks, vl_m1); - else - AES_CRYPT(d, m1, GET_PRELOADED_KEY, rk, rounds, data_blks, vl_m1); + if (encrypt) + AES_CRYPT_NO_RK0(e, m1, GET_PRELOADED_KEY, rk, rounds, data_blks, + vl); + else + AES_CRYPT_NO_RK0(d, m1, GET_PRELOADED_KEY, rk, rounds, data_blks, + vl); - if (!auth) - { - data_blks = __riscv_vxor_vv_u32m1(offs_m1, data_blks, vl_m1); + if (!auth) + { + data_blks = __riscv_vxor_vv_u32m1(offs, data_blks, vl); - unaligned_store_u32m1(outbuf, data_blks, vl_m1); - outbuf += curr_nblks * BLOCKSIZE; - } + unaligned_store_u32m1(outbuf, data_blks, vl); + } - if (encrypt <= 0) - { - /* Checksum_i = Checksum_{i-1} xor P_i */ - ctr = __riscv_vxor_vv_u32m1_tu(ctr, ctr, data_blks, vl_m1); + if (encrypt <= 0) + { + /* Checksum_i = Checksum_{i-1} xor P_i */ + ctr = xor_fold_u32m1_u32m1_tu(ctr, data_blks, vl, vl_m1); + } } - - inbuf += curr_nblks * BLOCKSIZE; - nblocks -= curr_nblks; } for (h = max_blocks_m1; h > 1; h /= 2) @@ -1574,6 +1902,10 @@ aes_riscv_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg, ctr = __riscv_vxor_vv_u32m1(ctr, ctr_hi, (h / 2) * blk_vl32); } + /* Remove first round key from offset. */ + LOAD_KEY(rk0, 0); + iv = __riscv_vxor_vv_u32m1(iv, rk0, blk_vl32); + if (auth) { c->u_mode.ocb.aad_nblocks = n; -- 2.53.0 From jussi.kivilinna at iki.fi Sun Oct 11 19:23:17 2026 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sun, 11 Oct 2026 20:23:17 +0300 Subject: [PATCH 08/12] sm4: add RISC-V Zvksed intrinsics implementation In-Reply-To: <20261011172321.997183-1-jussi.kivilinna@iki.fi> References: <20261011172321.997183-1-jussi.kivilinna@iki.fi> Message-ID: <20261011172321.997183-8-jussi.kivilinna@iki.fi> * cipher/Makefile.am: Add 'sm4-riscv-zvksed-zvkb.c' and 'sm4-riscv-zvksed-zvkb.h'. (riscv_vector_crypto_sm4_cflags): New. * cipher/sm4-riscv-zvksed-zvkb.c: New. * cipher/sm4-riscv-zvksed-zvkb.h: New. * cipher/sm4.c (USE_RISCV_V_CRYPTO): New. (SM4_context) [USE_RISCV_V_CRYPTO]: Add 'use_riscv_zvksed'. [USE_RISCV_V_CRYPTO] (_gcry_sm4_riscv_v_check_hw) (_gcry_sm4_riscv_zvksed_expand_key, _gcry_sm4_riscv_zvksed_crypt) (_gcry_sm4_riscv_zvksed_ctr_enc, _gcry_sm4_riscv_zvksed_cbc_dec) (_gcry_sm4_riscv_zvksed_cfb_dec, sm4_riscv_zvksed_crypt_blk1_16): New. (sm4_expand_key, sm4_encrypt, sm4_decrypt, sm4_get_crypt_blk1_16_fn) (_gcry_sm4_ctr_enc, _gcry_sm4_cbc_dec, _gcry_sm4_cfb_dec) (sm4_crypt_blk1_32) [USE_RISCV_V_CRYPTO]: Add RISC-V Zvksed code-path. (sm4_setkey) [USE_RISCV_V_CRYPTO]: Enable new implementation if supported by HW. * configure.ac: Add 'sm4-riscv-zvksed-zvkb.lo'. (GCRY_RISCV_VECTOR_CRYPTO_INTRINSICS_TEST): Add SM4 intrinsics test. (HAVE_BROKEN_VSM4R_VS_INTRINSIC): New check. * doc/gcrypt.texi: Add "riscv-zvksed". * src/g10lib.h [HAVE_CPU_ARCH_RISCV] (HWF_RISCV_ZVKSED): New. (HWF_RISCV_ZVKSH): Move to bit 11. * src/hwf-riscv.c (HWF_RISCV_HWPROBE_EXT_ZVKSED): New. (hwprobe_features): Add Zvksed. * src/hwfeatures.c (hwflist) [HAVE_CPU_ARCH_RISCV]: Add "riscv-zvksed". -- Blocks are processed with LMUL=4 and tail with smallest LMUL that fits. CTR, CBC decryption and CFB decryption have vector implementations, other modes use generic bulk helpers. Benchmark on SpacemiT K3, X100 core (out-of-order, VLEN=256): Before: SM4 | nanosecs/byte mebibytes/sec cycles/byte auto Mhz ECB enc | 16.95 ns/B 56.27 MiB/s 33.84 c/B 1997 CBC enc | 19.86 ns/B 48.02 MiB/s 39.67 c/B 1997 CBC dec | 17.13 ns/B 55.66 MiB/s 34.23 c/B 1998 CFB dec | 17.16 ns/B 55.59 MiB/s 34.29 c/B 1999 CTR enc | 17.21 ns/B 55.43 MiB/s 34.36 c/B 1997 XTS enc | 17.23 ns/B 55.35 MiB/s 34.41 c/B 1997 GCM enc | 17.33 ns/B 55.02 MiB/s 34.62 c/B 1997 OCB enc | 17.47 ns/B 54.60 MiB/s 34.88 c/B 1997 After (CTR ~47x faster, CBC enc ~7.2x faster): SM4 | nanosecs/byte mebibytes/sec cycles/byte auto Mhz ECB enc | 0.407 ns/B 2345 MiB/s 0.812 c/B 1997 CBC enc | 2.75 ns/B 346.8 MiB/s 5.49 c/B 1997 CBC dec | 0.435 ns/B 2192 MiB/s 0.869 c/B 1997 CFB dec | 0.418 ns/B 2281 MiB/s 0.835 c/B 1997 CTR enc | 0.364 ns/B 2617 MiB/s 0.728 c/B 1997 XTS enc | 0.680 ns/B 1402 MiB/s 1.36 c/B 1997 GCM enc | 0.497 ns/B 1921 MiB/s 0.991 c/B 1997 OCB enc | 0.904 ns/B 1055 MiB/s 1.80 c/B 1997 Benchmark on SpacemiT K3, A100 core (in-order, VLEN=1024): Before: SM4 | nanosecs/byte mebibytes/sec cycles/byte auto Mhz ECB enc | 23.12 ns/B 41.24 MiB/s 41.62 c/B 1800 CBC enc | 25.35 ns/B 37.62 MiB/s 45.63 c/B 1800 CBC dec | 23.47 ns/B 40.63 MiB/s 42.25 c/B 1800 CFB dec | 23.66 ns/B 40.31 MiB/s 42.58 c/B 1800 CTR enc | 23.73 ns/B 40.18 MiB/s 42.72 c/B 1800 XTS enc | 23.79 ns/B 40.08 MiB/s 42.82 c/B 1800 GCM enc | 25.22 ns/B 37.82 MiB/s 45.39 c/B 1800 OCB enc | 24.58 ns/B 38.80 MiB/s 44.24 c/B 1800 After (CTR ~56x faster, CBC enc ~2.2x faster): SM4 | nanosecs/byte mebibytes/sec cycles/byte auto Mhz ECB enc | 0.675 ns/B 1413 MiB/s 1.21 c/B 1800 CBC enc | 11.65 ns/B 81.83 MiB/s 20.98 c/B 1800 CBC dec | 0.507 ns/B 1882 MiB/s 0.912 c/B 1800 CFB dec | 0.498 ns/B 1913 MiB/s 0.897 c/B 1800 CTR enc | 0.423 ns/B 2254 MiB/s 0.762 c/B 1800 XTS enc | 1.35 ns/B 708.6 MiB/s 2.42 c/B 1800 GCM enc | 1.59 ns/B 598.5 MiB/s 2.87 c/B 1800 OCB enc | 2.32 ns/B 410.3 MiB/s 4.18 c/B 1800 Signed-off-by: Jussi Kivilinna --- cipher/Makefile.am | 12 +- cipher/sm4-riscv-zvksed-zvkb.c | 322 +++++++++++++++++++++++++++++++++ cipher/sm4-riscv-zvksed-zvkb.h | 210 +++++++++++++++++++++ cipher/sm4.c | 108 +++++++++++ configure.ac | 64 +++++++ doc/gcrypt.texi | 1 + src/g10lib.h | 3 +- src/hwf-riscv.c | 2 + src/hwfeatures.c | 1 + 9 files changed, 721 insertions(+), 2 deletions(-) create mode 100644 cipher/sm4-riscv-zvksed-zvkb.c create mode 100644 cipher/sm4-riscv-zvksed-zvkb.h diff --git a/cipher/Makefile.am b/cipher/Makefile.am index 50509914..23c4655c 100644 --- a/cipher/Makefile.am +++ b/cipher/Makefile.am @@ -142,7 +142,7 @@ EXTRA_libcipher_la_SOURCES = \ sm4-intel-avx2-amd64.S sm4-intel-avx512-amd64.S \ sm4-gfni-avx2-amd64.S sm4-gfni-avx512-amd64.S \ sm4-aarch64.S sm4-armv8-aarch64-ce.S sm4-armv9-aarch64-sve-ce.S \ - sm4-ppc.c \ + sm4-ppc.c sm4-riscv-zvksed-zvkb.c sm4-riscv-zvksed-zvkb.h \ sha1.c sha1-ssse3-amd64.S sha1-avx-amd64.S sha1-avx-bmi2-amd64.S \ sha1-avx2-bmi2-amd64.S sha1-armv7-neon.S sha1-armv8-aarch32-ce.S \ sha1-armv8-aarch64-ce.S sha1-intel-shaext.c \ @@ -474,6 +474,7 @@ riscv_vector_crypto_sha512_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvknhb_zvkb riscv_vector_crypto_sha512_m1_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvknhb_zvkb_zvl256b riscv_vector_crypto_sm3_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvksh_zvkb riscv_vector_crypto_sm3_m1_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvksh_zvkb_zvl256b +riscv_vector_crypto_sm4_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvksed_zvkb riscv_vector_crypto_gcm_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvkg_zvkb else riscv_vector_crypto_zvkb_cflags = @@ -483,6 +484,7 @@ riscv_vector_crypto_sha512_cflags = riscv_vector_crypto_sha512_m1_cflags = riscv_vector_crypto_sm3_cflags = riscv_vector_crypto_sm3_m1_cflags = +riscv_vector_crypto_sm4_cflags = riscv_vector_crypto_gcm_cflags = endif @@ -549,3 +551,11 @@ sm3-riscv-zvksh-zvkb-m1.o: $(srcdir)/sm3-riscv-zvksh-zvkb-m1.c Makefile sm3-riscv-zvksh-zvkb-m1.lo: $(srcdir)/sm3-riscv-zvksh-zvkb-m1.c Makefile `echo $(LTCOMPILE) $(riscv_vector_crypto_sm3_m1_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` $(custom_depfile_lo) + +sm4-riscv-zvksed-zvkb.o: $(srcdir)/sm4-riscv-zvksed-zvkb.c Makefile + `echo $(COMPILE) $(riscv_vector_crypto_sm4_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) + +sm4-riscv-zvksed-zvkb.lo: $(srcdir)/sm4-riscv-zvksed-zvkb.c Makefile + `echo $(LTCOMPILE) $(riscv_vector_crypto_sm4_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) diff --git a/cipher/sm4-riscv-zvksed-zvkb.c b/cipher/sm4-riscv-zvksed-zvkb.c new file mode 100644 index 00000000..9bb525a0 --- /dev/null +++ b/cipher/sm4-riscv-zvksed-zvkb.c @@ -0,0 +1,322 @@ +/* sm4-riscv-zvksed-zvkb.c - RISC-V vector crypto implementation of SM4 + * Copyright (C) 2026 Jussi Kivilinna + * + * This file is part of Libgcrypt. + * + * Libgcrypt is free software; you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as + * published by the Free Software Foundation; either version 2.1 of + * the License, or (at your option) any later version. + * + * Libgcrypt is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this program; if not, see . + */ + +#include + +#if defined (__riscv) && \ + defined(HAVE_COMPATIBLE_CC_RISCV_VECTOR_INTRINSICS) && \ + defined(HAVE_COMPATIBLE_CC_RISCV_VECTOR_CRYPTO_INTRINSICS) && \ + defined(USE_SM4) + +#include "g10lib.h" +#include "bufhelp.h" +#include "simd-common-riscv.h" +#include + + +#define ALWAYS_INLINE inline __attribute__((always_inline)) +#define NO_INLINE __attribute__((noinline)) +#define NO_INSTRUMENT_FUNCTION __attribute__((no_instrument_function)) + +#define ASM_FUNC_ATTR NO_INSTRUMENT_FUNCTION +#define ASM_FUNC_ATTR_INLINE ASM_FUNC_ATTR ALWAYS_INLINE +#define ASM_FUNC_ATTR_NOINLINE ASM_FUNC_ATTR NO_INLINE + +#ifdef HAVE_GCC_ATTRIBUTE_OPTIMIZE +# define FUNC_ATTR_OPT_O2 __attribute__((optimize("-O2"))) +#else +# define FUNC_ATTR_OPT_O2 +#endif + + +#define SM4_ROUND_KEY_VARIABLES \ + vuint32m1_t rk0, rk1, rk2, rk3, rk4, rk5, rk6, rk7 + +#define SM4_ROUND_KEY_PARAMS \ + vuint32m1_t rk0, vuint32m1_t rk1, vuint32m1_t rk2, vuint32m1_t rk3, \ + vuint32m1_t rk4, vuint32m1_t rk5, vuint32m1_t rk6, vuint32m1_t rk7 + +#define SM4_ROUND_KEYS rk0, rk1, rk2, rk3, rk4, rk5, rk6, rk7 + +#define SM4_LOAD_ROUND_KEYS(rk) \ + rk0 = __riscv_vle32_v_u32m1(rk + 0 * 4, 4); \ + rk1 = __riscv_vle32_v_u32m1(rk + 1 * 4, 4); \ + rk2 = __riscv_vle32_v_u32m1(rk + 2 * 4, 4); \ + rk3 = __riscv_vle32_v_u32m1(rk + 3 * 4, 4); \ + rk4 = __riscv_vle32_v_u32m1(rk + 4 * 4, 4); \ + rk5 = __riscv_vle32_v_u32m1(rk + 5 * 4, 4); \ + rk6 = __riscv_vle32_v_u32m1(rk + 6 * 4, 4); \ + rk7 = __riscv_vle32_v_u32m1(rk + 7 * 4, 4) + +#define SM4_NEXT_BLOCKS(n) \ + in += (n) * 16; \ + out += (n) * 16; \ + nblocks -= (n) + +#define SM4_LMUL m1 +#define SM4_MASK64_T vbool64_t +#include "sm4-riscv-zvksed-zvkb.h" + +#define SM4_LMUL m2 +#define SM4_MASK64_T vbool32_t +#include "sm4-riscv-zvksed-zvkb.h" + +#define SM4_LMUL m4 +#define SM4_MASK64_T vbool16_t +#include "sm4-riscv-zvksed-zvkb.h" + + +unsigned int ASM_FUNC_ATTR_NOINLINE FUNC_ATTR_OPT_O2 +_gcry_sm4_riscv_zvksed_crypt (const u32 *rk, byte *out, const byte *in, + size_t nblocks) +{ + size_t max_blocks_m1 = __riscv_vsetvlmax_e32m1() / 4; + size_t max_blocks_m2 = max_blocks_m1 * 2; + size_t max_blocks_m4 = max_blocks_m1 * 4; + SM4_ROUND_KEY_VARIABLES; + + SM4_LOAD_ROUND_KEYS(rk); + + if (nblocks > max_blocks_m2) + { + vbool16_t even_m4 = even_mask_m4(); + + while (nblocks >= max_blocks_m4) + { + sm4_ecb_blocks_m4(SM4_ROUND_KEYS, even_m4, out, in, max_blocks_m4); + SM4_NEXT_BLOCKS(max_blocks_m4); + } + if (nblocks > max_blocks_m2) + { + sm4_ecb_blocks_m4(SM4_ROUND_KEYS, even_m4, out, in, nblocks); + SM4_NEXT_BLOCKS(nblocks); + } + } + if (nblocks > max_blocks_m1) + sm4_ecb_blocks_m2(SM4_ROUND_KEYS, even_mask_m2(), out, in, nblocks); + else if (nblocks) + sm4_ecb_blocks_m1(SM4_ROUND_KEYS, even_mask_m1(), out, in, nblocks); + + clear_vec_regs(); + + return 0; +} + + +/* Caller handles carry beyond low 16 bits of counter. */ +void ASM_FUNC_ATTR_NOINLINE FUNC_ATTR_OPT_O2 +_gcry_sm4_riscv_zvksed_ctr_enc (const u32 *rk, byte *out, const byte *in, + byte *ctr, size_t nblocks) +{ + u64 ctr_lo = rol64(buf_get_be64(ctr + 0), 32); + u64 ctr_hi_noswap = buf_get_be64(ctr + 8); + u64 ctr_hi = rol64(ctr_hi_noswap, 32); + size_t max_blocks_m1 = __riscv_vsetvlmax_e32m1() / 4; + size_t max_blocks_m2 = max_blocks_m1 * 2; + size_t max_blocks_m4 = max_blocks_m1 * 4; + SM4_ROUND_KEY_VARIABLES; + + SM4_LOAD_ROUND_KEYS(rk); + + ctr[14] = (ctr_hi_noswap + nblocks) >> 8; + ctr[15] = (ctr_hi_noswap + nblocks) >> 0; + + if (nblocks > max_blocks_m2) + { + vbool16_t even_m4 = even_mask_m4(); + vuint64m4_t ctr_m4 = ctr_init_m4(even_m4, ctr_lo, ctr_hi); + vuint64m4_t inc_m4 = ctr_inc_m4(even_m4, max_blocks_m4); + + while (nblocks >= max_blocks_m4) + { + sm4_ctr_blocks_m4(SM4_ROUND_KEYS, even_m4, &ctr_m4, inc_m4, out, in, + max_blocks_m4); + ctr_hi += (u64)max_blocks_m4 << 32; + SM4_NEXT_BLOCKS(max_blocks_m4); + } + if (nblocks > max_blocks_m2) + { + sm4_ctr_blocks_m4(SM4_ROUND_KEYS, even_m4, &ctr_m4, inc_m4, out, in, + nblocks); + ctr_hi += (u64)nblocks << 32; + SM4_NEXT_BLOCKS(nblocks); + } + } + if (nblocks > max_blocks_m1) + { + vbool32_t even_m2 = even_mask_m2(); + vuint64m2_t ctr_m2 = ctr_init_m2(even_m2, ctr_lo, ctr_hi); + + sm4_ctr_blocks_m2(SM4_ROUND_KEYS, even_m2, &ctr_m2, + ctr_inc_m2(even_m2, max_blocks_m2), out, in, nblocks); + } + else if (nblocks) + { + vbool64_t even_m1 = even_mask_m1(); + vuint64m1_t ctr_m1 = ctr_init_m1(even_m1, ctr_lo, ctr_hi); + + sm4_ctr_blocks_m1(SM4_ROUND_KEYS, even_m1, &ctr_m1, + ctr_inc_m1(even_m1, max_blocks_m1), out, in, nblocks); + } + + clear_vec_regs(); +} + + +void ASM_FUNC_ATTR_NOINLINE FUNC_ATTR_OPT_O2 +_gcry_sm4_riscv_zvksed_cbc_dec (const u32 *rk, byte *out, const byte *in, + byte *iv_arg, size_t nblocks) +{ + size_t max_blocks_m1 = __riscv_vsetvlmax_e32m1() / 4; + size_t max_blocks_m2 = max_blocks_m1 * 2; + size_t max_blocks_m4 = max_blocks_m1 * 4; + vuint32m1_t iv; + SM4_ROUND_KEY_VARIABLES; + + SM4_LOAD_ROUND_KEYS(rk); + + iv = load_u32_m1(iv_arg, 4); + + if (nblocks > max_blocks_m2) + { + vbool16_t even_m4 = even_mask_m4(); + vuint32m4_t ivs_m4 = __riscv_vundefined_u32m4(); + + while (nblocks >= max_blocks_m4) + { + sm4_cbc_dec_blocks_m4(SM4_ROUND_KEYS, even_m4, &iv, &ivs_m4, out, in, + max_blocks_m4); + SM4_NEXT_BLOCKS(max_blocks_m4); + } + if (nblocks > max_blocks_m2) + { + sm4_cbc_dec_blocks_m4(SM4_ROUND_KEYS, even_m4, &iv, &ivs_m4, out, in, + nblocks); + SM4_NEXT_BLOCKS(nblocks); + } + } + if (nblocks > max_blocks_m1) + { + vuint32m2_t ivs_m2 = __riscv_vundefined_u32m2(); + + sm4_cbc_dec_blocks_m2(SM4_ROUND_KEYS, even_mask_m2(), &iv, &ivs_m2, out, + in, nblocks); + } + else if (nblocks) + { + vuint32m1_t ivs_m1 = __riscv_vundefined_u32m1(); + + sm4_cbc_dec_blocks_m1(SM4_ROUND_KEYS, even_mask_m1(), &iv, &ivs_m1, out, + in, nblocks); + } + + store_u32_m1(iv_arg, iv, 4); + + clear_vec_regs(); +} + + +void ASM_FUNC_ATTR_NOINLINE FUNC_ATTR_OPT_O2 +_gcry_sm4_riscv_zvksed_cfb_dec (const u32 *rk, byte *out, const byte *in, + byte *iv_arg, size_t nblocks) +{ + size_t max_blocks_m1 = __riscv_vsetvlmax_e32m1() / 4; + size_t max_blocks_m2 = max_blocks_m1 * 2; + size_t max_blocks_m4 = max_blocks_m1 * 4; + vuint32m1_t iv; + SM4_ROUND_KEY_VARIABLES; + + SM4_LOAD_ROUND_KEYS(rk); + + iv = load_u32_m1(iv_arg, 4); + + if (nblocks > max_blocks_m2) + { + vbool16_t even_m4 = even_mask_m4(); + vuint32m4_t ivs_m4 = __riscv_vundefined_u32m4(); + + while (nblocks >= max_blocks_m4) + { + sm4_cfb_dec_blocks_m4(SM4_ROUND_KEYS, even_m4, &iv, &ivs_m4, out, in, + max_blocks_m4); + SM4_NEXT_BLOCKS(max_blocks_m4); + } + if (nblocks > max_blocks_m2) + { + sm4_cfb_dec_blocks_m4(SM4_ROUND_KEYS, even_m4, &iv, &ivs_m4, out, in, + nblocks); + SM4_NEXT_BLOCKS(nblocks); + } + } + if (nblocks > max_blocks_m1) + { + vuint32m2_t ivs_m2 = __riscv_vundefined_u32m2(); + + sm4_cfb_dec_blocks_m2(SM4_ROUND_KEYS, even_mask_m2(), &iv, &ivs_m2, out, + in, nblocks); + } + else if (nblocks) + { + vuint32m1_t ivs_m1 = __riscv_vundefined_u32m1(); + + sm4_cfb_dec_blocks_m1(SM4_ROUND_KEYS, even_mask_m1(), &iv, &ivs_m1, out, + in, nblocks); + } + + store_u32_m1(iv_arg, iv, 4); + + clear_vec_regs(); +} + + +#define SM4_EXPAND_KEY_ROUNDS(rnd) \ + k = __riscv_vsm4k_vi_u32m1(k, rnd, vl); \ + __riscv_vse32_v_u32m1(rk_enc + (rnd) * 4, k, vl); \ + __riscv_vsse32_v_u32m1(rk_dec + 31 - (rnd) * 4, -4, k, vl) + +void ASM_FUNC_ATTR_NOINLINE FUNC_ATTR_OPT_O2 +_gcry_sm4_riscv_zvksed_expand_key (const byte *key, u32 *rk_enc, u32 *rk_dec, + const u32 *fk) +{ + size_t vl = 4; + vuint32m1_t k; + + k = __riscv_vrev8_v_u32m1(load_u32_m1(key, vl), vl); + k = __riscv_vxor_vv_u32m1(k, __riscv_vle32_v_u32m1(fk, vl), vl); + + SM4_EXPAND_KEY_ROUNDS(0); + SM4_EXPAND_KEY_ROUNDS(1); + SM4_EXPAND_KEY_ROUNDS(2); + SM4_EXPAND_KEY_ROUNDS(3); + SM4_EXPAND_KEY_ROUNDS(4); + SM4_EXPAND_KEY_ROUNDS(5); + SM4_EXPAND_KEY_ROUNDS(6); + SM4_EXPAND_KEY_ROUNDS(7); + + clear_vec_regs(); +} + + +unsigned int ASM_FUNC_ATTR FUNC_ATTR_OPT_O2 +_gcry_sm4_riscv_v_check_hw(void) +{ + return (__riscv_vsetvl_e32m1(4) == 4); +} + +#endif /* HAVE_COMPATIBLE_CC_RISCV_VECTOR_INTRINSICS */ diff --git a/cipher/sm4-riscv-zvksed-zvkb.h b/cipher/sm4-riscv-zvksed-zvkb.h new file mode 100644 index 00000000..38e9b825 --- /dev/null +++ b/cipher/sm4-riscv-zvksed-zvkb.h @@ -0,0 +1,210 @@ +/* sm4-riscv-zvksed-zvkb.h - RISC-V vector crypto SM4 LMUL template + * Copyright (C) 2026 Jussi Kivilinna + * + * This file is part of Libgcrypt. + * + * Libgcrypt is free software; you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as + * published by the Free Software Foundation; either version 2.1 of + * the License, or (at your option) any later version. + * + * Libgcrypt is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this program; if not, see . + */ + +/* Included once for each LMUL with SM4_LMUL and SM4_MASK64_T set. */ + +#define CAT_(a, b) a##b +#define CAT(a, b) CAT_(a, b) + +#define FN(name) CAT(name##_, SM4_LMUL) +#define V32(name) CAT(__riscv_##name##_u32, SM4_LMUL) +#define V64(name) CAT(__riscv_##name##_u64, SM4_LMUL) +#define V8(name) CAT(__riscv_##name##_u8, SM4_LMUL) +#define VSETVLMAX_E64() CAT(__riscv_vsetvlmax_e64, SM4_LMUL)() +#define VEC32_T CAT(CAT(vuint32, SM4_LMUL), _t) +#define VEC64_T CAT(CAT(vuint64, SM4_LMUL), _t) +#define MASK64_T SM4_MASK64_T + +#define SET_IV_m1(ivs, iv) (iv) +#define SET_IV_m2(ivs, iv) __riscv_vset(ivs, 0, iv) +#define SET_IV_m4(ivs, iv) __riscv_vset(ivs, 0, iv) +#define SET_IV(ivs, iv) CAT(SET_IV_, SM4_LMUL)(ivs, iv) + + +static ASM_FUNC_ATTR_INLINE VEC32_T +FN(load_u32) (const byte *p, size_t vl) +{ +#ifdef RVV_UNALIGNED_NOT_ALLOWED + return V32(vreinterpret)(V8(vle8_v)(p, vl * 4)); +#else + return V32(vle32_v)((const void *)p, vl); +#endif +} + +static ASM_FUNC_ATTR_INLINE void +FN(store_u32) (byte *p, VEC32_T v, size_t vl) +{ +#ifdef RVV_UNALIGNED_NOT_ALLOWED + __riscv_vse8(p, V8(vreinterpret)(v), vl * 4); +#else + __riscv_vse32((u32 *)(void *)p, v, vl); +#endif +} + +static ASM_FUNC_ATTR_INLINE MASK64_T +FN(even_mask) (void) +{ + size_t vl64 = VSETVLMAX_E64(); + + return __riscv_vmseq(__riscv_vand(V64(vid_v)(vl64), 1, vl64), 0, vl64); +} + +/* Returns { ctr_lo, ctr_hi + (0 << 32), ctr_lo, ctr_hi + (1 << 32), + * ctr_lo, ctr_hi + (2 << 32), ... }. */ +static ASM_FUNC_ATTR_INLINE VEC64_T +FN(ctr_init) (MASK64_T even, u64 ctr_lo, u64 ctr_hi) +{ + size_t vl64 = VSETVLMAX_E64(); + VEC64_T c, offs; + + offs = __riscv_vsll(__riscv_vsrl(V64(vid_v)(vl64), 1, vl64), 32, vl64); + offs = __riscv_vmerge(offs, 0, even, vl64); + c = V64(vmv_v_x)(ctr_hi, vl64); + c = __riscv_vmerge(c, ctr_lo, even, vl64); + return __riscv_vadd(c, offs, vl64); +} + +/* Returns { 0, nblocks << 32, 0, nblocks << 32, ... }. */ +static ASM_FUNC_ATTR_INLINE VEC64_T +FN(ctr_inc) (MASK64_T even, size_t nblocks) +{ + size_t vl64 = VSETVLMAX_E64(); + + return __riscv_vmerge(V64(vmv_v_x)((u64)nblocks << 32, vl64), 0, even, + vl64); +} + +static ASM_FUNC_ATTR_INLINE VEC32_T +FN(sm4_crypt) (SM4_ROUND_KEY_PARAMS, MASK64_T even, VEC32_T x, size_t vl) +{ + size_t vl64 = vl / 2; + VEC64_T t, y; + +#ifdef HAVE_BROKEN_VSM4R_VS_INTRINSIC + asm ( "vsetvli zero,%[vl],e32," STR2(SM4_LMUL) ",ta,ma;\n\t" + "vsm4r.vs %[x],%[rk0];\n\t" + "vsm4r.vs %[x],%[rk1];\n\t" + "vsm4r.vs %[x],%[rk2];\n\t" + "vsm4r.vs %[x],%[rk3];\n\t" + "vsm4r.vs %[x],%[rk4];\n\t" + "vsm4r.vs %[x],%[rk5];\n\t" + "vsm4r.vs %[x],%[rk6];\n\t" + "vsm4r.vs %[x],%[rk7];\n\t" + : [x] "+vr" (x) + : [vl] "r" (vl), + [rk0] "vr" (rk0), [rk1] "vr" (rk1), [rk2] "vr" (rk2), + [rk3] "vr" (rk3), [rk4] "vr" (rk4), [rk5] "vr" (rk5), + [rk6] "vr" (rk6), [rk7] "vr" (rk7) + : "vl", "vtype", "memory"); +#else + x = __riscv_vsm4r_vs(x, rk0, vl); + x = __riscv_vsm4r_vs(x, rk1, vl); + x = __riscv_vsm4r_vs(x, rk2, vl); + x = __riscv_vsm4r_vs(x, rk3, vl); + x = __riscv_vsm4r_vs(x, rk4, vl); + x = __riscv_vsm4r_vs(x, rk5, vl); + x = __riscv_vsm4r_vs(x, rk6, vl); + x = __riscv_vsm4r_vs(x, rk7, vl); +#endif + + t = __riscv_vrev8(V64(vreinterpret)(x), vl64); + y = __riscv_vslide1up(t, 0, vl64); + y = __riscv_vslidedown_mu(even, y, t, 1, vl64); + return V32(vreinterpret)(y); +} + +static ASM_FUNC_ATTR_INLINE void +FN(sm4_ecb_blocks) (SM4_ROUND_KEY_PARAMS, MASK64_T even, byte *out, + const byte *in, size_t n) +{ + size_t vl = n * 4; + VEC32_T x; + + x = __riscv_vrev8(FN(load_u32)(in, vl), vl); + x = FN(sm4_crypt)(SM4_ROUND_KEYS, even, x, vl); + FN(store_u32)(out, x, vl); +} + +static ASM_FUNC_ATTR_INLINE void +FN(sm4_ctr_blocks) (SM4_ROUND_KEY_PARAMS, MASK64_T even, VEC64_T *ctr, + VEC64_T inc, byte *out, const byte *in, size_t n) +{ + size_t vl = n * 4; + size_t vl64 = n * 2; + VEC32_T x, d; + + d = FN(load_u32)(in, vl); + x = V32(vreinterpret)(*ctr); + *ctr = __riscv_vadd(*ctr, inc, vl64); + + x = FN(sm4_crypt)(SM4_ROUND_KEYS, even, x, vl); + x = __riscv_vxor(x, d, vl); + FN(store_u32)(out, x, vl); +} + +static ASM_FUNC_ATTR_INLINE void +FN(sm4_cbc_dec_blocks) (SM4_ROUND_KEY_PARAMS, MASK64_T even, vuint32m1_t *iv, + VEC32_T *ivs, byte *out, const byte *in, size_t n) +{ + size_t vl = n * 4; + VEC32_T d, x; + + d = FN(load_u32)(in, vl); + *ivs = __riscv_vslideup(SET_IV(*ivs, *iv), d, 4, vl); + *iv = load_u32_m1(in + (n - 1) * 16, 4); + + x = __riscv_vrev8(d, vl); + x = FN(sm4_crypt)(SM4_ROUND_KEYS, even, x, vl); + x = __riscv_vxor(x, *ivs, vl); + FN(store_u32)(out, x, vl); +} + +static ASM_FUNC_ATTR_INLINE void +FN(sm4_cfb_dec_blocks) (SM4_ROUND_KEY_PARAMS, MASK64_T even, vuint32m1_t *iv, + VEC32_T *ivs, byte *out, const byte *in, size_t n) +{ + size_t vl = n * 4; + VEC32_T d, x; + + d = FN(load_u32)(in, vl); + *ivs = __riscv_vslideup(SET_IV(*ivs, *iv), d, 4, vl); + *iv = load_u32_m1(in + (n - 1) * 16, 4); + + x = __riscv_vrev8(*ivs, vl); + x = FN(sm4_crypt)(SM4_ROUND_KEYS, even, x, vl); + x = __riscv_vxor(x, d, vl); + FN(store_u32)(out, x, vl); +} + +#undef SET_IV +#undef SET_IV_m4 +#undef SET_IV_m2 +#undef SET_IV_m1 +#undef MASK64_T +#undef VEC64_T +#undef VEC32_T +#undef VSETVLMAX_E64 +#undef V8 +#undef V64 +#undef V32 +#undef FN +#undef CAT +#undef CAT_ +#undef SM4_MASK64_T +#undef SM4_LMUL diff --git a/cipher/sm4.c b/cipher/sm4.c index 58aaea10..95cabdf3 100644 --- a/cipher/sm4.c +++ b/cipher/sm4.c @@ -142,6 +142,15 @@ # define USE_PPC_CRYPTO 1 #endif +/* USE_RISCV_V_CRYPTO indicates whether to enable RISC-V vector cryptography + * extension code. */ +#undef USE_RISCV_V_CRYPTO +#if defined (__riscv) && \ + defined(HAVE_COMPATIBLE_CC_RISCV_VECTOR_INTRINSICS) && \ + defined(HAVE_COMPATIBLE_CC_RISCV_VECTOR_CRYPTO_INTRINSICS) +# define USE_RISCV_V_CRYPTO 1 +#endif + static const char *sm4_selftest (void); static void _gcry_sm4_ctr_enc (void *context, unsigned char *ctr, @@ -216,6 +225,9 @@ typedef struct unsigned int use_ppc8le:1; unsigned int use_ppc9le:1; #endif +#ifdef USE_RISCV_V_CRYPTO + unsigned int use_riscv_zvksed:1; +#endif #ifdef ASM_FUNC_ABI unsigned int ocb_crypt_blk_init:1; unsigned int ocb_auth_blk_init:1; @@ -756,6 +768,36 @@ sm4_ppc9le_crypt_blk1_16(void *rk, byte *out, const byte *in, size_t num_blks) } #endif /* USE_PPC_CRYPTO */ +#ifdef USE_RISCV_V_CRYPTO +extern unsigned int _gcry_sm4_riscv_v_check_hw(void); + +extern void _gcry_sm4_riscv_zvksed_expand_key(const byte *key, u32 *rk_enc, + u32 *rk_dec, const u32 *fk); + +extern unsigned int _gcry_sm4_riscv_zvksed_crypt(const u32 *rk, byte *out, + const byte *in, + size_t num_blks); + +extern void _gcry_sm4_riscv_zvksed_ctr_enc(const u32 *rk_enc, byte *out, + const byte *in, byte *ctr, + size_t nblocks); + +extern void _gcry_sm4_riscv_zvksed_cbc_dec(const u32 *rk_dec, byte *out, + const byte *in, byte *iv, + size_t nblocks); + +extern void _gcry_sm4_riscv_zvksed_cfb_dec(const u32 *rk_enc, byte *out, + const byte *in, byte *iv, + size_t nblocks); + +static inline unsigned int +sm4_riscv_zvksed_crypt_blk1_16(void *rk, byte *out, const byte *in, + size_t num_blks) +{ + return _gcry_sm4_riscv_zvksed_crypt(rk, out, in, num_blks); +} +#endif /* USE_RISCV_V_CRYPTO */ + static inline void prefetch_sbox_table(void) { const volatile byte *vtab = (void *)&sbox_table.S[0]; @@ -870,6 +912,15 @@ sm4_expand_key (SM4_context *ctx, const byte *key) } #endif +#ifdef USE_RISCV_V_CRYPTO + if (ctx->use_riscv_zvksed) + { + _gcry_sm4_riscv_zvksed_expand_key (key, ctx->rkey_enc, ctx->rkey_dec, + fk); + return; + } +#endif + prefetch_sbox_table (); rk[0] = buf_get_be32(key + 4 * 0) ^ fk[0]; @@ -953,6 +1004,14 @@ sm4_setkey (void *context, const byte *key, const unsigned keylen, ctx->use_ppc8le = (hwf & HWF_PPC_VCRYPTO) != 0; ctx->use_ppc9le = (hwf & HWF_PPC_VCRYPTO) && (hwf & HWF_PPC_ARCH_3_00); #endif +#ifdef USE_RISCV_V_CRYPTO + ctx->use_riscv_zvksed = (hwf & HWF_RISCV_IMAFDC) + && (hwf & HWF_RISCV_B) /* Mandatory in RVA23U64 */ + && (hwf & HWF_RISCV_V) /* Mandatory in RVA23U64 */ + && (hwf & HWF_RISCV_ZVKB) /* Mandatory in RVA23U64 (Zvbb) */ + && (hwf & HWF_RISCV_ZVKSED) /* Optional in RVA23U64 (Zvks) */ + && _gcry_sm4_riscv_v_check_hw(); +#endif #ifdef USE_GFNI_AVX2 if (ctx->use_gfni_avx2) @@ -1056,6 +1115,11 @@ sm4_encrypt (void *context, byte *outbuf, const byte *inbuf) return sm4_armv8_ce_crypt_blk1_16(ctx->rkey_enc, outbuf, inbuf, 1); #endif +#ifdef USE_RISCV_V_CRYPTO + if (ctx->use_riscv_zvksed) + return sm4_riscv_zvksed_crypt_blk1_16(ctx->rkey_enc, outbuf, inbuf, 1); +#endif + prefetch_sbox_table (); return sm4_do_crypt (ctx->rkey_enc, outbuf, inbuf); @@ -1086,6 +1150,11 @@ sm4_decrypt (void *context, byte *outbuf, const byte *inbuf) return sm4_armv8_ce_crypt_blk1_16(ctx->rkey_dec, outbuf, inbuf, 1); #endif +#ifdef USE_RISCV_V_CRYPTO + if (ctx->use_riscv_zvksed) + return sm4_riscv_zvksed_crypt_blk1_16(ctx->rkey_dec, outbuf, inbuf, 1); +#endif + prefetch_sbox_table (); return sm4_do_crypt (ctx->rkey_dec, outbuf, inbuf); @@ -1232,6 +1301,12 @@ sm4_get_crypt_blk1_16_fn(SM4_context *ctx) { return &sm4_ppc8le_crypt_blk1_16; } +#endif +#ifdef USE_RISCV_V_CRYPTO + else if (ctx->use_riscv_zvksed) + { + return &sm4_riscv_zvksed_crypt_blk1_16; + } #endif else { @@ -1364,6 +1439,15 @@ _gcry_sm4_ctr_enc(void *context, unsigned char *ctr, } #endif +#ifdef USE_RISCV_V_CRYPTO + if (ctx->use_riscv_zvksed) + { + /* Process all blocks at a time. */ + _gcry_sm4_riscv_zvksed_ctr_enc(ctx->rkey_enc, outbuf, inbuf, ctr, nblocks); + nblocks = 0; + } +#endif + #ifdef USE_ARM_CE if (ctx->use_arm_ce) { @@ -1543,6 +1627,15 @@ _gcry_sm4_cbc_dec(void *context, unsigned char *iv, } #endif +#ifdef USE_RISCV_V_CRYPTO + if (ctx->use_riscv_zvksed) + { + /* Process all blocks at a time. */ + _gcry_sm4_riscv_zvksed_cbc_dec(ctx->rkey_dec, outbuf, inbuf, iv, nblocks); + nblocks = 0; + } +#endif + #ifdef USE_ARM_CE if (ctx->use_arm_ce) { @@ -1722,6 +1815,15 @@ _gcry_sm4_cfb_dec(void *context, unsigned char *iv, } #endif +#ifdef USE_RISCV_V_CRYPTO + if (ctx->use_riscv_zvksed) + { + /* Process all blocks at a time. */ + _gcry_sm4_riscv_zvksed_cfb_dec(ctx->rkey_enc, outbuf, inbuf, iv, nblocks); + nblocks = 0; + } +#endif + #ifdef USE_ARM_CE if (ctx->use_arm_ce) { @@ -1806,6 +1908,12 @@ sm4_crypt_blk1_32 (SM4_context *ctx, byte *outbuf, const byte *inbuf, return 0; } #endif +#ifdef USE_RISCV_V_CRYPTO + if (ctx->use_riscv_zvksed) + { + return _gcry_sm4_riscv_zvksed_crypt (rk, outbuf, inbuf, num_blks); + } +#endif if (num_blks > 16) { diff --git a/configure.ac b/configure.ac index a6b1f325..b10af44c 100644 --- a/configure.ac +++ b/configure.ac @@ -3054,6 +3054,15 @@ m4_define([GCRY_RISCV_VECTOR_CRYPTO_INTRINSICS_TEST], e = __riscv_vsm3c_vi_u32m1(e, g, 31, vl); __riscv_vse32_v_u32m1(ptr + 0 * vl, e, vl); } + void test_sm4(unsigned int *ptr) + { + int vl = __riscv_vsetvl_e32m4 (16); + vuint32m1_t a = __riscv_vle32_v_u32m1(ptr + 0 * vl, 4); + vuint32m4_t b = __riscv_vle32_v_u32m4(ptr + 1 * vl, vl); + vuint32m1_t c = __riscv_vsm4k_vi_u32m1(a, 7, 4); + vuint32m4_t d = __riscv_vsm4r_vs_u32m1_u32m4(b, c, vl); + __riscv_vse32_v_u32m4(ptr + 0 * vl, d, vl); + } void test_rev8(unsigned int *ptr) { int vl = __riscv_vsetvl_e32m1 (4); @@ -3093,6 +3102,7 @@ m4_define([GCRY_RISCV_VECTOR_CRYPTO_INTRINSICS_TEST], test_aes_crypt(buf); test_ghash(buf); test_sm3(buf); + test_sm4(buf); test_rev8(buf); test_rotate(buf); test_inline_vec_asm(buf); @@ -3254,6 +3264,56 @@ EOF CFLAGS=$_gcc_cflags_save; fi +# +# Check whether compiler has working RISC-V vector __riscv_vsm4r_vs intrinsics +# +# Same GCC issue as with __riscv_vaes*_vs intrinsics above, 'vsetvli' is +# generated with 'm1' LMUL instead of 'mX' for __riscv_vsm4r_vs_u32m1_u32mX. +# +if test "$gcry_cv_cc_riscv_vector_crypto_intrinsics" = "yes" || + test "$gcry_cv_cc_riscv_vector_crypto_intrinsics_cflags" = "yes"; then + + # Setup flags for test. + _gcc_cflags_save=$CFLAGS + if test "$gcry_cv_cc_riscv_vector_crypto_intrinsics_cflags" = "yes"; then + CFLAGS="$CFLAGS -O2 -march=$MARCH_RVA23U64_WITH_VEC_CRYPTO" + else + CFLAGS="$CFLAGS -O2" + fi + + # Disable LTO optimization for following check to generate + # RISC-V instructions in compiler output + CFLAGS="$CFLAGS -fno-lto" + + AC_CACHE_CHECK([whether compiler has working RISC-V __riscv_vsm4r_vs intrinsics], + [gcry_cv_riscv_vsm4r_vs_intrinsics_work], + [gcry_cv_riscv_vsm4r_vs_intrinsics_work=no + cat > conftest.c < +vuint32m4_t test_fn(vuint32m4_t a, vuint32m1_t b, int vl) +{ + /* This intrinsic should result in a 'vsetvli' with m4 */ + return __riscv_vsm4r_vs_u32m1_u32m4(a, b, vl); +} +EOF + + if $CC $CFLAGS -S conftest.c -o conftest.s >&5 2>&5; then + if grep -E 'vsetvli.*,[[[:space:]]]*m4[[[:space:]]]*,' conftest.s >/dev/null 2>&1; then + gcry_cv_riscv_vsm4r_vs_intrinsics_work=yes + fi + fi + rm -f conftest.* + ]) + + if test "$gcry_cv_riscv_vsm4r_vs_intrinsics_work" = "no"; then + AC_DEFINE([HAVE_BROKEN_VSM4R_VS_INTRINSIC], [1], + [Define to 1 if __riscv_vsm4r_vs intrinsics are broken]) + fi + + # Restore flags. + CFLAGS=$_gcc_cflags_save; +fi + ####################################### #### Checks for library functions. #### @@ -3973,6 +4033,10 @@ if test "$found" = "1" ; then # Build with the ppc64le vector implementation GCRYPT_ASM_CIPHERS="$GCRYPT_ASM_CIPHERS sm4-ppc.lo" ;; + riscv64-*-*) + # Build with the RISC-V vector cryptography implementation + GCRYPT_ASM_CIPHERS="$GCRYPT_ASM_CIPHERS sm4-riscv-zvksed-zvkb.lo" + ;; esac fi diff --git a/doc/gcrypt.texi b/doc/gcrypt.texi index 613920d8..18a6f24c 100644 --- a/doc/gcrypt.texi +++ b/doc/gcrypt.texi @@ -623,6 +623,7 @@ are @item riscv-zvkned @item riscv-zvknha @item riscv-zvknhb + at item riscv-zvksed @item riscv-zvksh @end table diff --git a/src/g10lib.h b/src/g10lib.h index 8a2d706c..a9c18db6 100644 --- a/src/g10lib.h +++ b/src/g10lib.h @@ -288,7 +288,8 @@ char **_gcry_strtokenize (const char *string, const char *delim); #define HWF_RISCV_ZVKNED (1 << 7) #define HWF_RISCV_ZVKNHA (1 << 8) #define HWF_RISCV_ZVKNHB (1 << 9) -#define HWF_RISCV_ZVKSH (1 << 10) +#define HWF_RISCV_ZVKSED (1 << 10) +#define HWF_RISCV_ZVKSH (1 << 11) #endif diff --git a/src/hwf-riscv.c b/src/hwf-riscv.c index adec2b02..192ab031 100644 --- a/src/hwf-riscv.c +++ b/src/hwf-riscv.c @@ -197,6 +197,7 @@ detect_riscv_at_hwcap(void) #define HWF_RISCV_HWPROBE_EXT_ZVKNED (1U << 21) #define HWF_RISCV_HWPROBE_EXT_ZVKNHA (1U << 22) #define HWF_RISCV_HWPROBE_EXT_ZVKNHB (1U << 23) +#define HWF_RISCV_HWPROBE_EXT_ZVKSED (1U << 24) #define HWF_RISCV_HWPROBE_EXT_ZVKSH (1U << 25) #define HWF_RISCV_HWPROBE_EXT_ZICOND (U64_C(1) << 35) @@ -228,6 +229,7 @@ static const struct hwprobe_feature_map_s hwprobe_features[] = { HWF_RISCV_HWPROBE_EXT_ZVKNED, HWF_RISCV_ZVKNED }, { HWF_RISCV_HWPROBE_EXT_ZVKNHA, HWF_RISCV_ZVKNHA }, { HWF_RISCV_HWPROBE_EXT_ZVKNHB, HWF_RISCV_ZVKNHB }, + { HWF_RISCV_HWPROBE_EXT_ZVKSED, HWF_RISCV_ZVKSED }, { HWF_RISCV_HWPROBE_EXT_ZVKSH, HWF_RISCV_ZVKSH }, }; diff --git a/src/hwfeatures.c b/src/hwfeatures.c index 2f836079..3067ee86 100644 --- a/src/hwfeatures.c +++ b/src/hwfeatures.c @@ -127,6 +127,7 @@ static struct { HWF_RISCV_ZVKNED, "riscv-zvkned" }, { HWF_RISCV_ZVKNHA, "riscv-zvknha" }, { HWF_RISCV_ZVKNHB, "riscv-zvknhb" }, + { HWF_RISCV_ZVKSED, "riscv-zvksed" }, { HWF_RISCV_ZVKSH, "riscv-zvksh" }, #endif }; -- 2.53.0 From jussi.kivilinna at iki.fi Sun Oct 11 19:23:21 2026 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sun, 11 Oct 2026 20:23:21 +0300 Subject: [PATCH 12/12] simd-common-riscv: faster vector register clearing In-Reply-To: <20261011172321.997183-1-jussi.kivilinna@iki.fi> References: <20261011172321.997183-1-jussi.kivilinna@iki.fi> Message-ID: <20261011172321.997183-12-jussi.kivilinna@iki.fi> * cipher/simd-common-riscv.h (clear_vec_regs): Clear registers with four independent LMUL=8 'vmv.v.i' instructions. -- Faster clearing with independent vector splats instead of chain of whole register moves. Signed-off-by: Jussi Kivilinna --- cipher/simd-common-riscv.h | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/cipher/simd-common-riscv.h b/cipher/simd-common-riscv.h index 8ddd5056..98361f49 100644 --- a/cipher/simd-common-riscv.h +++ b/cipher/simd-common-riscv.h @@ -38,14 +38,11 @@ #define memory_barrier_with_vec(a) __asm__("" : "+vr"(a) :: "memory") -#define clear_vec_regs() __asm__ volatile("vsetvli zero, %0, e8, m1, ta, ma;\n" \ +#define clear_vec_regs() __asm__ volatile("vsetvli zero, %0, e32, m8, ta, ma;\n" \ "vmv.v.i v0, 0;\n" \ - "vmv.v.i v1, 0;\n" \ - "vmv2r.v v2, v0;\n" \ - "vmv4r.v v4, v0;\n" \ - "vmv8r.v v8, v0;\n" \ - "vmv8r.v v16, v0;\n" \ - "vmv8r.v v24, v0;\n" \ + "vmv.v.i v8, 0;\n" \ + "vmv.v.i v16, 0;\n" \ + "vmv.v.i v24, 0;\n" \ : \ : "r" (~0) \ : "memory", "vl", "vtype", \ -- 2.53.0 From jussi.kivilinna at iki.fi Sun Oct 11 19:23:20 2026 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sun, 11 Oct 2026 20:23:20 +0300 Subject: [PATCH 11/12] cipher-gcm-riscv-zvkg: use fractional LMUL for POLYVAL byte reversal In-Reply-To: <20261011172321.997183-1-jussi.kivilinna@iki.fi> References: <20261011172321.997183-1-jussi.kivilinna@iki.fi> Message-ID: <20261011172321.997183-11-jussi.kivilinna@iki.fi> * cipher/cipher-gcm-riscv-zvkg.c (cast_u32m1_u8m1, bswap128_u32m1): Remove. (cast_u8mf2_u32m1, cast_u8mf4_u32m1, cast_u8mf8_u32m1, bswap128_arr) (POLYVAL_LOOP): New. (_gcry_polyval_riscv_zvkg): Use POLYVAL_LOOP with LMUL selected by VLEN. -- Byte reversal of POLYVAL input blocks was done with vrgather at LMUL=1, which is slow with VLEN larger than 128 bits. Now vrgather is done with LMUL that gives 128-bit register group, selected by VLEN. Benchmark on SpacemiT K3, X100 core (out-of-order, VLEN=256): Before: AES | nanosecs/byte mebibytes/sec cycles/byte GCM-SIV enc | 0.586 ns/B 1627 MiB/s 1.17 c/B GCM-SIV dec | 0.634 ns/B 1504 MiB/s 1.27 c/B GCM-SIV auth | 0.224 ns/B 4249 MiB/s 0.449 c/B After (GCM-SIV auth ~1.7x faster): AES | nanosecs/byte mebibytes/sec cycles/byte GCM-SIV enc | 0.494 ns/B 1929 MiB/s 0.989 c/B GCM-SIV dec | 0.537 ns/B 1777 MiB/s 1.07 c/B GCM-SIV auth | 0.129 ns/B 7418 MiB/s 0.257 c/B Benchmark on SpacemiT K3, A100 core (in-order, VLEN=1024): Before: AES | nanosecs/byte mebibytes/sec cycles/byte GCM-SIV enc | 3.71 ns/B 256.7 MiB/s 6.69 c/B GCM-SIV dec | 3.92 ns/B 243.5 MiB/s 7.05 c/B GCM-SIV auth | 3.44 ns/B 276.9 MiB/s 6.20 c/B After (GCM-SIV auth ~3.0x faster): AES | nanosecs/byte mebibytes/sec cycles/byte GCM-SIV enc | 1.53 ns/B 622.4 MiB/s 2.76 c/B GCM-SIV dec | 1.54 ns/B 617.3 MiB/s 2.78 c/B GCM-SIV auth | 1.13 ns/B 842.8 MiB/s 2.04 c/B Signed-off-by: Jussi Kivilinna --- cipher/cipher-gcm-riscv-zvkg.c | 57 ++++++++++++++++++++-------------- 1 file changed, 34 insertions(+), 23 deletions(-) diff --git a/cipher/cipher-gcm-riscv-zvkg.c b/cipher/cipher-gcm-riscv-zvkg.c index 1f0b3797..76be974f 100644 --- a/cipher/cipher-gcm-riscv-zvkg.c +++ b/cipher/cipher-gcm-riscv-zvkg.c @@ -38,7 +38,9 @@ #define cast_u8m1_u32m1(a) __riscv_vreinterpret_v_u8m1_u32m1(a) -#define cast_u32m1_u8m1(a) __riscv_vreinterpret_v_u32m1_u8m1(a) +#define cast_u8mf2_u32m1(a) cast_u8m1_u32m1(__riscv_vlmul_ext_v_u8mf2_u8m1(a)) +#define cast_u8mf4_u32m1(a) cast_u8m1_u32m1(__riscv_vlmul_ext_v_u8mf4_u8m1(a)) +#define cast_u8mf8_u32m1(a) cast_u8m1_u32m1(__riscv_vlmul_ext_v_u8mf8_u8m1(a)) static ASM_FUNC_ATTR_INLINE vuint32m1_t @@ -52,19 +54,6 @@ unaligned_load_u32m1(const void *ptr, size_t vl_u32) #endif } -static ASM_FUNC_ATTR_INLINE vuint32m1_t -bswap128_u32m1(vuint32m1_t vec, size_t vl_u32) -{ - static const byte bswap128_arr[16] = - { 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 }; - size_t vl_bytes = vl_u32 * 4; - vuint8m1_t bswap128 = __riscv_vle8_v_u8m1(bswap128_arr, vl_bytes); - - return cast_u8m1_u32m1( - __riscv_vrgather_vv_u8m1(cast_u32m1_u8m1(vec), bswap128, vl_bytes)); -} - - ASM_FUNC_ATTR_NOINLINE int _gcry_ghash_setup_riscv_zvkg(gcry_cipher_hd_t c) { @@ -104,24 +93,46 @@ _gcry_ghash_riscv_zvkg(gcry_cipher_hd_t c, byte *result, const byte *buf, return 0; } +static const byte bswap128_arr[16] = + { 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 }; + +#define POLYVAL_LOOP(lmul) \ + do \ + { \ + vuint8##lmul##_t bswap128 = __riscv_vle8_v_u8##lmul(bswap128_arr, 16); \ + \ + while (nblocks) \ + { \ + vuint8##lmul##_t data = __riscv_vle8_v_u8##lmul(buf, 16); \ + buf += 16; \ + nblocks--; \ + \ + data = __riscv_vrgather_vv_u8##lmul(data, bswap128, 16); \ + rhash = __riscv_vghsh_vv_u32m1(rhash, rh1, \ + cast_u8##lmul##_u32m1(data), vl); \ + } \ + } \ + while (0) + ASM_FUNC_ATTR_NOINLINE unsigned int _gcry_polyval_riscv_zvkg(gcry_cipher_hd_t c, byte *result, const byte *buf, - size_t nblocks) + size_t nblocks) { u32 *result_u32 = (void *)result; const u32 *key_u32 = (void *)c->u_mode.gcm.u_ghash_key.key; size_t vl = 4; + size_t vlenb = __riscv_vsetvlmax_e8m1(); vuint32m1_t rhash = __riscv_vle32_v_u32m1(result_u32, vl); vuint32m1_t rh1 = __riscv_vle32_v_u32m1(key_u32, vl); - while (nblocks) - { - vuint32m1_t data = bswap128_u32m1(unaligned_load_u32m1(buf, vl), vl); - buf += 16; - nblocks--; - - rhash = __riscv_vghsh_vv_u32m1(rhash, rh1, data, vl); - } + if (vlenb >= 128) + POLYVAL_LOOP(mf8); + else if (vlenb >= 64) + POLYVAL_LOOP(mf4); + else if (vlenb >= 32) + POLYVAL_LOOP(mf2); + else + POLYVAL_LOOP(m1); __riscv_vse32_v_u32m1(result_u32, rhash, vl); -- 2.53.0 From jussi.kivilinna at iki.fi Sun Oct 11 19:23:11 2026 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sun, 11 Oct 2026 20:23:11 +0300 Subject: [PATCH 02/12] chacha20: add RISC-V Zvkb intrinsics implementation In-Reply-To: <20261011172321.997183-1-jussi.kivilinna@iki.fi> References: <20261011172321.997183-1-jussi.kivilinna@iki.fi> Message-ID: <20261011172321.997183-2-jussi.kivilinna@iki.fi> * cipher/Makefile.am: Add 'chacha20-riscv-zvkb.c' and 'chacha20-riscv-rvv.h'; add 'riscv_vector_crypto_zvkb_cflags' handling for 'chacha20-riscv-zvkb.o' and 'chacha20-riscv-zvkb.lo'. * cipher/chacha20-riscv-rvv.h: New, implementation moved from 'chacha20-riscv-v.c'. (LOAD_GATHER_ROTATE_CONSTANTS): New. * cipher/chacha20-riscv-v.c: Move implementation to 'chacha20-riscv-rvv.h'. * cipher/chacha20-riscv-zvkb.c: New. * cipher/chacha20.c (USE_RISCV_ZVKB): New. (CHACHA20_context_s): Add 'use_riscv_zvkb'. [USE_RISCV_ZVKB] (_gcry_chacha20_riscv_zvkb_blocks): New. (chacha20_blocks) [USE_RISCV_ZVKB]: Add RISC-V Zvkb code path. (chacha20_do_setkey) [USE_RISCV_ZVKB]: Add HW feature detection for RISC-V Zvkb implementation. * configure.ac (GCRY_RISCV_VECTOR_CRYPTO_INTRINSICS_TEST): Add vector rotate test. Add 'chacha20-riscv-zvkb.lo'. -- Zvkb implementation uses vector rotate instructions instead of vrgather and shift/add sequences. Benchmark on SpacemiT K3, X100 core (2000 Mhz, out-of-order, VLEN=256): Before: CHACHA20 | nanosecs/byte mebibytes/sec cycles/byte STREAM enc | 2.10 ns/B 453.6 MiB/s 4.20 c/B POLY1305 enc | 2.91 ns/B 327.2 MiB/s 5.83 c/B After (STREAM ~1.38x, POLY1305 ~1.25x faster): CHACHA20 | nanosecs/byte mebibytes/sec cycles/byte STREAM enc | 1.52 ns/B 627.3 MiB/s 3.04 c/B POLY1305 enc | 2.33 ns/B 409.9 MiB/s 4.65 c/B Benchmark on SpacemiT K3, A100 core (1800 Mhz, in-order, VLEN=1024): Before: CHACHA20 | nanosecs/byte mebibytes/sec cycles/byte STREAM enc | 6.85 ns/B 139.2 MiB/s 12.33 c/B POLY1305 enc | 10.07 ns/B 94.73 MiB/s 18.12 c/B After (STREAM ~1.81x, POLY1305 ~1.32x faster): CHACHA20 | nanosecs/byte mebibytes/sec cycles/byte STREAM enc | 3.78 ns/B 252.6 MiB/s 6.80 c/B POLY1305 enc | 7.64 ns/B 124.9 MiB/s 13.75 c/B Signed-off-by: Jussi Kivilinna --- cipher/Makefile.am | 11 + cipher/chacha20-riscv-rvv.h | 548 +++++++++++++++++++++++++++++++++++ cipher/chacha20-riscv-v.c | 517 +-------------------------------- cipher/chacha20-riscv-zvkb.c | 43 +++ cipher/chacha20.c | 28 ++ configure.ac | 12 + 6 files changed, 645 insertions(+), 514 deletions(-) create mode 100644 cipher/chacha20-riscv-rvv.h create mode 100644 cipher/chacha20-riscv-zvkb.c diff --git a/cipher/Makefile.am b/cipher/Makefile.am index 89c7a269..a3d012ba 100644 --- a/cipher/Makefile.am +++ b/cipher/Makefile.am @@ -90,6 +90,7 @@ EXTRA_libcipher_la_SOURCES = \ chacha20-amd64-avx512.S chacha20-armv7-neon.S chacha20-aarch64.S \ chacha20-ppc.c chacha20-s390x.S chacha20-p10le-8x.s \ chacha20-riscv-v.c \ + chacha20-riscv-zvkb.c chacha20-riscv-rvv.h \ cipher-gcm-ppc.c cipher-gcm-intel-pclmul.c \ cipher-gcm-aarch64-simd.c cipher-gcm-armv7-neon.S \ cipher-gcm-armv8-aarch32-ce.S cipher-gcm-armv8-aarch64-ce.S \ @@ -455,15 +456,25 @@ rijndael-vp-riscv.lo: $(srcdir)/rijndael-vp-riscv.c Makefile $(custom_depfile_lo) if ENABLE_RISCV_VECTOR_CRYPTO_INTRINSICS_EXTRA_CFLAGS +riscv_vector_crypto_zvkb_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvkb riscv_vector_crypto_aes_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvkned_zvkb riscv_vector_crypto_sha_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvknha_zvknhb_zvkb riscv_vector_crypto_gcm_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvkg_zvkb else +riscv_vector_crypto_zvkb_cflags = riscv_vector_crypto_aes_cflags = riscv_vector_crypto_sha_cflags = riscv_vector_crypto_gcm_cflags = endif +chacha20-riscv-zvkb.o: $(srcdir)/chacha20-riscv-zvkb.c Makefile + `echo $(COMPILE) $(riscv_vector_crypto_zvkb_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) + +chacha20-riscv-zvkb.lo: $(srcdir)/chacha20-riscv-zvkb.c Makefile + `echo $(LTCOMPILE) $(riscv_vector_crypto_zvkb_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) + cipher-gcm-riscv-zvkg.o: $(srcdir)/cipher-gcm-riscv-zvkg.c Makefile `echo $(COMPILE) $(riscv_vector_crypto_gcm_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` $(custom_depfile_o) diff --git a/cipher/chacha20-riscv-rvv.h b/cipher/chacha20-riscv-rvv.h new file mode 100644 index 00000000..abd53c99 --- /dev/null +++ b/cipher/chacha20-riscv-rvv.h @@ -0,0 +1,548 @@ +/* chacha20-riscv-rvv.h - RISC-V vector implementation of ChaCha20 + * Copyright (C) 2025-2026 Jussi Kivilinna + * + * This file is part of Libgcrypt. + * + * Libgcrypt is free software; you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as + * published by the Free Software Foundation; either version 2.1 of + * the License, or (at your option) any later version. + * + * Libgcrypt is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this program; if not, see . + */ + +#ifndef GCRY_CHACHA20_RISCV_RVV_H +#define GCRY_CHACHA20_RISCV_RVV_H + +#include "simd-common-riscv.h" +#include +#include "bufhelp.h" + + +#define ALWAYS_INLINE inline __attribute__((always_inline)) +#define NO_INLINE __attribute__((noinline)) +#define NO_INSTRUMENT_FUNCTION __attribute__((no_instrument_function)) + +#define ASM_FUNC_ATTR NO_INSTRUMENT_FUNCTION +#define ASM_FUNC_ATTR_INLINE ASM_FUNC_ATTR ALWAYS_INLINE +#define ASM_FUNC_ATTR_NOINLINE ASM_FUNC_ATTR NO_INLINE + +#ifdef HAVE_GCC_ATTRIBUTE_OPTIMIZE +# define FUNC_ATTR_OPT_O2 __attribute__((optimize("-O2"))) +#else +# define FUNC_ATTR_OPT_O2 +#endif + + +/********************************************************************** + RISC-V vector extension chacha20 + **********************************************************************/ + +#define XOR(v, w) __riscv_vxor_vv_u32m1((v), (w), vl) +#define PLUS(v, w) __riscv_vadd_vv_u32m1((v), (w), vl) +#define WORD_ROL(v, c) __riscv_vrgather_vv_u32m1((v), (rol##c), vl) + +#define QUARTERROUND_4(a0, b0, c0, d0, a1, b1, c1, d1, \ + a2, b2, c2, d2, a3, b3, c3, d3) \ + a0 = PLUS(a0, b0); a1 = PLUS(a1, b1); \ + a2 = PLUS(a2, b2); a3 = PLUS(a3, b3); \ + d0 = XOR(d0, a0); d1 = XOR(d1, a1); \ + d2 = XOR(d2, a2); d3 = XOR(d3, a3); \ + d0 = ROTATE16(d0); d1 = ROTATE16(d1); \ + d2 = ROTATE16(d2); d3 = ROTATE16(d3); \ + c0 = PLUS(c0, d0); c1 = PLUS(c1, d1); \ + c2 = PLUS(c2, d2); c3 = PLUS(c3, d3); \ + b0 = XOR(b0, c0); b1 = XOR(b1, c1); \ + b2 = XOR(b2, c2); b3 = XOR(b3, c3); \ + b0 = ROTATE(b0, 12); b1 = ROTATE(b1, 12); \ + b2 = ROTATE(b2, 12); b3 = ROTATE(b3, 12); \ + a0 = PLUS(a0, b0); a1 = PLUS(a1, b1); \ + a2 = PLUS(a2, b2); a3 = PLUS(a3, b3); \ + d0 = XOR(d0, a0); d1 = XOR(d1, a1); \ + d2 = XOR(d2, a2); d3 = XOR(d3, a3); \ + d0 = ROTATE8(d0); d1 = ROTATE8(d1); \ + d2 = ROTATE8(d2); d3 = ROTATE8(d3); \ + c0 = PLUS(c0, d0); c1 = PLUS(c1, d1); \ + c2 = PLUS(c2, d2); c3 = PLUS(c3, d3); \ + b0 = XOR(b0, c0); b1 = XOR(b1, c1); \ + b2 = XOR(b2, c2); b3 = XOR(b3, c3); \ + b0 = ROTATE(b0, 7); b1 = ROTATE(b1, 7); \ + b2 = ROTATE(b2, 7); b3 = ROTATE(b3, 7); + +#define QUARTERROUND4_2(x0, x1, x2, x3, y0, y1, y2, y3, rol_x1, rol_x2, rol_x3) \ + x0 = PLUS(x0, x1); y0 = PLUS(y0, y1); \ + x3 = XOR(x3, x0); y3 = XOR(y3, y0); \ + x3 = ROTATE16(x3); y3 = ROTATE16(y3); \ + x2 = PLUS(x2, x3); y2 = PLUS(y2, y3); \ + x1 = XOR(x1, x2); y1 = XOR(y1, y2); \ + x1 = ROTATE(x1, 12); y1 = ROTATE(y1, 12); \ + x0 = PLUS(x0, x1); y0 = PLUS(y0, y1); \ + x3 = XOR(x3, x0); y3 = XOR(y3, y0); \ + x3 = ROTATE8(x3); y3 = ROTATE8(y3); \ + x2 = PLUS(x2, x3); y2 = PLUS(y2, y3); \ + x3 = WORD_ROL(x3, rol_x3); y3 = WORD_ROL(y3, rol_x3);\ + x1 = XOR(x1, x2); y1 = XOR(y1, y2); \ + x2 = WORD_ROL(x2, rol_x2); y2 = WORD_ROL(y2, rol_x2); \ + x1 = ROTATE(x1, 7); y1 = ROTATE(y1, 7); \ + x1 = WORD_ROL(x1, rol_x1); y1 = WORD_ROL(y1, rol_x1); + +#define QUARTERROUND4(x0, x1, x2, x3, rol_x1, rol_x2, rol_x3) \ + x0 = PLUS(x0, x1); x3 = XOR(x3, x0); x3 = ROTATE16(x3); \ + x2 = PLUS(x2, x3); x1 = XOR(x1, x2); x1 = ROTATE(x1, 12); \ + x0 = PLUS(x0, x1); x3 = XOR(x3, x0); x3 = ROTATE8(x3); \ + x2 = PLUS(x2, x3); \ + x3 = WORD_ROL(x3, rol_x3); \ + x1 = XOR(x1, x2); \ + x2 = WORD_ROL(x2, rol_x2); \ + x1= ROTATE(x1, 7); \ + x1 = WORD_ROL(x1, rol_x1); + +#define ADD_U64(a, b) __riscv_vreinterpret_v_u64m1_u32m1( \ + __riscv_vadd_vv_u64m1( \ + __riscv_vreinterpret_v_u32m1_u64m1(a), \ + __riscv_vreinterpret_v_u32m1_u64m1(b), vl / 2)) + +#define vxor_v_u32m1_u32m1x8(data, idx, vs, vl) \ + __riscv_vset_v_u32m1_u32m1x8((data), (idx), \ + __riscv_vxor_vv_u32m1( \ + __riscv_vget_v_u32m1x8_u32m1((data), (idx)), (vs), (vl))) + +#ifdef LOAD_GATHER_ROTATE_CONSTANTS + +static ASM_FUNC_ATTR_INLINE vuint16m1_t +gen_rot16(size_t vl) +{ + return __riscv_vxor_vx_u16m1(__riscv_vid_v_u16m1(vl * 2), 1, vl * 2); +} + +static ASM_FUNC_ATTR_INLINE vuint8m1_t +gen_rot8(size_t vl) +{ + vuint8m1_t rot8, rot8_hi; + + rot8 = __riscv_vid_v_u8m1(vl * 4); + rot8_hi = __riscv_vand_vx_u8m1(rot8, ~3, vl * 4); + rot8 = __riscv_vadd_vx_u8m1(rot8, 3, vl * 4); + rot8 = __riscv_vand_vx_u8m1(rot8, 3, vl * 4); + rot8 = __riscv_vadd_vv_u8m1(rot8, rot8_hi, vl * 4); + + return rot8; +} + +#endif + +static ASM_FUNC_ATTR_INLINE vuint16m2_t +gen_indexes(size_t vl, size_t stride) +{ + vuint16m2_t idx = __riscv_vid_v_u16m2(vl * 4); + vuint16m2_t idx_lo = __riscv_vand_vx_u16m2(idx, 3, vl * 4); + vuint16m2_t idx_hi = __riscv_vsrl_vx_u16m2(idx, 2, vl * 4); + idx_hi = __riscv_vmul_vx_u16m2(idx_hi, stride, vl * 4); + return __riscv_vadd_vv_u16m2(idx_hi, idx_lo, vl * 4); +} + +static ASM_FUNC_ATTR_INLINE vuint32m1_t +unaligned_load_u32m1(const void *src, size_t vl) +{ +#ifdef RVV_UNALIGNED_NOT_ALLOWED + return __riscv_vreinterpret_v_u8m1_u32m1(__riscv_vle8_v_u8m1(src, vl * 4)); +#else + return __riscv_vle32_v_u32m1(src, vl); +#endif +} + +static ASM_FUNC_ATTR_INLINE void +unaligned_store_u32m1(void *dst, vuint32m1_t vec, size_t vl) +{ +#ifdef RVV_UNALIGNED_NOT_ALLOWED + __riscv_vse8_v_u8m1(dst, __riscv_vreinterpret_v_u32m1_u8m1(vec), vl * 4); +#else + __riscv_vse32_v_u32m1(dst, vec, vl); +#endif +} + +static ASM_FUNC_ATTR_INLINE vuint32m1x8_t +unaligned_vlsseg8e32_v_u32m1x8(const void *src, size_t vl) +{ +#ifdef RVV_UNALIGNED_NOT_ALLOWED + if (UNLIKELY(((uintptr_t)src & 3) != 0)) + { + const byte *bsrc = src; + vuint16m2_t indexes; + vuint8m1_t b0, b1, b2, b3, b4, b5, b6, b7; + vuint32m1x8_t data; + + indexes = gen_indexes(4 * vl, 64); + + b0 = __riscv_vluxei16_v_u8m1(bsrc + 0 * 4, indexes, vl * 4); + b1 = __riscv_vluxei16_v_u8m1(bsrc + 1 * 4, indexes, vl * 4); + b2 = __riscv_vluxei16_v_u8m1(bsrc + 2 * 4, indexes, vl * 4); + b3 = __riscv_vluxei16_v_u8m1(bsrc + 3 * 4, indexes, vl * 4); + b4 = __riscv_vluxei16_v_u8m1(bsrc + 4 * 4, indexes, vl * 4); + b5 = __riscv_vluxei16_v_u8m1(bsrc + 5 * 4, indexes, vl * 4); + b6 = __riscv_vluxei16_v_u8m1(bsrc + 6 * 4, indexes, vl * 4); + b7 = __riscv_vluxei16_v_u8m1(bsrc + 7 * 4, indexes, vl * 4); + + data = __riscv_vundefined_u32m1x8(); + data = __riscv_vset_v_u32m1_u32m1x8( + data, 0, __riscv_vreinterpret_v_u8m1_u32m1(b0)); + data = __riscv_vset_v_u32m1_u32m1x8( + data, 1, __riscv_vreinterpret_v_u8m1_u32m1(b1)); + data = __riscv_vset_v_u32m1_u32m1x8( + data, 2, __riscv_vreinterpret_v_u8m1_u32m1(b2)); + data = __riscv_vset_v_u32m1_u32m1x8( + data, 3, __riscv_vreinterpret_v_u8m1_u32m1(b3)); + data = __riscv_vset_v_u32m1_u32m1x8( + data, 4, __riscv_vreinterpret_v_u8m1_u32m1(b4)); + data = __riscv_vset_v_u32m1_u32m1x8( + data, 5, __riscv_vreinterpret_v_u8m1_u32m1(b5)); + data = __riscv_vset_v_u32m1_u32m1x8( + data, 6, __riscv_vreinterpret_v_u8m1_u32m1(b6)); + data = __riscv_vset_v_u32m1_u32m1x8( + data, 7, __riscv_vreinterpret_v_u8m1_u32m1(b7)); + + return data; + } + else +#endif + { + /* Fast path for 32-bit aligned loads. */ + return __riscv_vlsseg8e32_v_u32m1x8(src, 64, vl); + } +} + +static ASM_FUNC_ATTR_INLINE void +unaligned_vssseg8e32_v_u32m1x8(void *dst, vuint32m1x8_t data, size_t vl) +{ +#ifdef RVV_UNALIGNED_NOT_ALLOWED + if (UNLIKELY(((uintptr_t)dst & 3) != 0)) + { + byte *bdst = dst; + vuint16m2_t indexes; + vuint8m1_t b0, b1, b2, b3, b4, b5, b6, b7; + + indexes = gen_indexes(4 * vl, 64); + + b0 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x8_u32m1(data, 0)); + b1 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x8_u32m1(data, 1)); + b2 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x8_u32m1(data, 2)); + b3 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x8_u32m1(data, 3)); + b4 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x8_u32m1(data, 4)); + b5 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x8_u32m1(data, 5)); + b6 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x8_u32m1(data, 6)); + b7 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x8_u32m1(data, 7)); + + __riscv_vsuxei16_v_u8m1(bdst + 0 * 4, indexes, b0, vl * 4); + __riscv_vsuxei16_v_u8m1(bdst + 1 * 4, indexes, b1, vl * 4); + __riscv_vsuxei16_v_u8m1(bdst + 2 * 4, indexes, b2, vl * 4); + __riscv_vsuxei16_v_u8m1(bdst + 3 * 4, indexes, b3, vl * 4); + __riscv_vsuxei16_v_u8m1(bdst + 4 * 4, indexes, b4, vl * 4); + __riscv_vsuxei16_v_u8m1(bdst + 5 * 4, indexes, b5, vl * 4); + __riscv_vsuxei16_v_u8m1(bdst + 6 * 4, indexes, b6, vl * 4); + __riscv_vsuxei16_v_u8m1(bdst + 7 * 4, indexes, b7, vl * 4); + } + else +#endif + { + /* Fast path for 32-bit aligned stores. */ + __riscv_vssseg8e32_v_u32m1x8(dst, 64, data, vl); + return; + } +} + +static ASM_FUNC_ATTR_INLINE unsigned int +chacha20_rvv_blocks(u32 *input, byte *dst, const byte *src, size_t nblks) +{ + unsigned int i; + + if (nblks == 0) + return 0; + + /* Try use vector implementation when there is 4 or more blocks. */ + if (nblks >= 4) + { + size_t vl = __riscv_vsetvl_e32m1(nblks) < 4 + ? __riscv_vsetvl_e32m1(4) : __riscv_vsetvl_e32m1(nblks); + vuint32m1_t x0, x1, x2, x3, x4, x5, x6, x7; + vuint32m1_t x8, x9, x10, x11, x12, x13, x14, x15; + u32 s0, s1, s2, s3, s4, s5, s6, s7; + u32 s8, s9, s10, s11, s12, s13, s14, s15; +#ifdef LOAD_GATHER_ROTATE_CONSTANTS + vuint16m1_t rot16 = gen_rot16(vl); + vuint8m1_t rot8 = gen_rot8(vl); +#endif + + s0 = input[0]; + s1 = input[1]; + s2 = input[2]; + s3 = input[3]; + s4 = input[4]; + s5 = input[5]; + s6 = input[6]; + s7 = input[7]; + s8 = input[8]; + s9 = input[9]; + s10 = input[10]; + s11 = input[11]; + s12 = input[12]; + s13 = input[13]; + s14 = input[14]; + s15 = input[15]; + + while (nblks >= 4) + { + vuint32m1_t ctr; + vbool32_t carry; + vuint32m1x8_t data; + + if (vl < 4) + break; + + x0 = __riscv_vmv_v_x_u32m1(s0, vl); + x1 = __riscv_vmv_v_x_u32m1(s1, vl); + x2 = __riscv_vmv_v_x_u32m1(s2, vl); + x3 = __riscv_vmv_v_x_u32m1(s3, vl); + x4 = __riscv_vmv_v_x_u32m1(s4, vl); + x5 = __riscv_vmv_v_x_u32m1(s5, vl); + x6 = __riscv_vmv_v_x_u32m1(s6, vl); + x7 = __riscv_vmv_v_x_u32m1(s7, vl); + x8 = __riscv_vmv_v_x_u32m1(s8, vl); + x9 = __riscv_vmv_v_x_u32m1(s9, vl); + x10 = __riscv_vmv_v_x_u32m1(s10, vl); + x11 = __riscv_vmv_v_x_u32m1(s11, vl); + x13 = __riscv_vmv_v_x_u32m1(s13, vl); + x14 = __riscv_vmv_v_x_u32m1(s14, vl); + x15 = __riscv_vmv_v_x_u32m1(s15, vl); + + ctr = __riscv_vid_v_u32m1(vl); + carry = __riscv_vmadc_vx_u32m1_b32(ctr, s12, vl); + ctr = __riscv_vadd_vx_u32m1(ctr, s12, vl); + x12 = ctr; + x13 = __riscv_vadc_vxm_u32m1(x13, 0, carry, vl); + + for (i = 20; i > 0; i -= 2) + { + QUARTERROUND_4(x0, x4, x8, x12, + x1, x5, x9, x13, + x2, x6, x10, x14, + x3, x7, x11, x15); + QUARTERROUND_4(x0, x5, x10, x15, + x1, x6, x11, x12, + x2, x7, x8, x13, + x3, x4, x9, x14); + } + + x0 = __riscv_vadd_vx_u32m1(x0, s0, vl); + x1 = __riscv_vadd_vx_u32m1(x1, s1, vl); + x2 = __riscv_vadd_vx_u32m1(x2, s2, vl); + x3 = __riscv_vadd_vx_u32m1(x3, s3, vl); + x4 = __riscv_vadd_vx_u32m1(x4, s4, vl); + x5 = __riscv_vadd_vx_u32m1(x5, s5, vl); + x6 = __riscv_vadd_vx_u32m1(x6, s6, vl); + x7 = __riscv_vadd_vx_u32m1(x7, s7, vl); + x8 = __riscv_vadd_vx_u32m1(x8, s8, vl); + x9 = __riscv_vadd_vx_u32m1(x9, s9, vl); + x10 = __riscv_vadd_vx_u32m1(x10, s10, vl); + x11 = __riscv_vadd_vx_u32m1(x11, s11, vl); + x12 = __riscv_vadd_vv_u32m1(x12, ctr, vl); + x13 = __riscv_vadc_vxm_u32m1(x13, s13, carry, vl); + x14 = __riscv_vadd_vx_u32m1(x14, s14, vl); + x15 = __riscv_vadd_vx_u32m1(x15, s15, vl); + + s12 += vl; + s13 += s12 < vl; + + data = unaligned_vlsseg8e32_v_u32m1x8((const void *)src, vl); + + data = vxor_v_u32m1_u32m1x8(data, 0, x0, vl); + data = vxor_v_u32m1_u32m1x8(data, 1, x1, vl); + data = vxor_v_u32m1_u32m1x8(data, 2, x2, vl); + data = vxor_v_u32m1_u32m1x8(data, 3, x3, vl); + data = vxor_v_u32m1_u32m1x8(data, 4, x4, vl); + data = vxor_v_u32m1_u32m1x8(data, 5, x5, vl); + data = vxor_v_u32m1_u32m1x8(data, 6, x6, vl); + data = vxor_v_u32m1_u32m1x8(data, 7, x7, vl); + + unaligned_vssseg8e32_v_u32m1x8((void *)dst, data, vl); + + data = unaligned_vlsseg8e32_v_u32m1x8((const void *)(src + 32), vl); + + data = vxor_v_u32m1_u32m1x8(data, 0, x8, vl); + data = vxor_v_u32m1_u32m1x8(data, 1, x9, vl); + data = vxor_v_u32m1_u32m1x8(data, 2, x10, vl); + data = vxor_v_u32m1_u32m1x8(data, 3, x11, vl); + data = vxor_v_u32m1_u32m1x8(data, 4, x12, vl); + data = vxor_v_u32m1_u32m1x8(data, 5, x13, vl); + data = vxor_v_u32m1_u32m1x8(data, 6, x14, vl); + data = vxor_v_u32m1_u32m1x8(data, 7, x15, vl); + + unaligned_vssseg8e32_v_u32m1x8((void *)(dst + 32), data, vl); + + src += vl * 64; + dst += vl * 64; + nblks -= vl; + vl = __riscv_vsetvl_e32m1(nblks) < 4 + ? __riscv_vsetvl_e32m1(4) : __riscv_vsetvl_e32m1(nblks); + } + + input[12] = s12; + input[13] = s13; + } + + /* Use SIMD implementation for remaining blocks. */ + if (nblks > 0) + { + static const u32 rol_const[3][4] = + { + { 1, 2, 3, 0 }, + { 2, 3, 0, 1 }, + { 3, 0, 1, 2 } + }; + static const u32 one_u64_const[4] = { 1, 0, 0, 0 }; + size_t vl = 4; + vuint32m1_t rol1, rol2, rol3; + vuint32m1_t one_u64; + vuint32m1_t v0, v1, v2, v3; + vuint32m1_t v4, v5, v6, v7; + vuint32m1_t state0, state1, state2, state3; + vuint32m1_t i0, i1, i2, i3; + vuint32m1_t i4, i5, i6, i7; +#ifdef LOAD_GATHER_ROTATE_CONSTANTS + vuint16m1_t rot16 = gen_rot16(vl); + vuint8m1_t rot8 = gen_rot8(vl); +#endif + + rol1 = __riscv_vle32_v_u32m1(rol_const[0], vl); + rol2 = __riscv_vle32_v_u32m1(rol_const[1], vl); + rol3 = __riscv_vle32_v_u32m1(rol_const[2], vl); + one_u64 = __riscv_vle32_v_u32m1(one_u64_const, vl); + + state0 = __riscv_vle32_v_u32m1(&input[0], vl); + state1 = __riscv_vle32_v_u32m1(&input[4], vl); + state2 = __riscv_vle32_v_u32m1(&input[8], vl); + state3 = __riscv_vle32_v_u32m1(&input[12], vl); + + input[12] += nblks; + input[13] += input[12] < nblks; + + /* SIMD 2x block implementation */ + while (nblks >= 2) + { + v0 = state0; + v1 = state1; + v2 = state2; + v3 = state3; + + v4 = state0; + v5 = state1; + v6 = state2; + v7 = state3; + v7 = ADD_U64(v7, one_u64); + + i0 = unaligned_load_u32m1(src + 0 * 16, vl); + i1 = unaligned_load_u32m1(src + 1 * 16, vl); + i2 = unaligned_load_u32m1(src + 2 * 16, vl); + i3 = unaligned_load_u32m1(src + 3 * 16, vl); + + for (i = 20; i > 0; i -= 2) + { + QUARTERROUND4_2(v0, v1, v2, v3, v4, v5, v6, v7, 1, 2, 3); + QUARTERROUND4_2(v0, v1, v2, v3, v4, v5, v6, v7, 3, 2, 1); + } + + v0 = __riscv_vadd_vv_u32m1(v0, state0, vl); + v1 = __riscv_vadd_vv_u32m1(v1, state1, vl); + v2 = __riscv_vadd_vv_u32m1(v2, state2, vl); + v3 = __riscv_vadd_vv_u32m1(v3, state3, vl); + state3 = ADD_U64(state3, one_u64); + + v0 = __riscv_vxor_vv_u32m1(i0, v0, vl); + v1 = __riscv_vxor_vv_u32m1(i1, v1, vl); + v2 = __riscv_vxor_vv_u32m1(i2, v2, vl); + v3 = __riscv_vxor_vv_u32m1(i3, v3, vl); + + v4 = __riscv_vadd_vv_u32m1(v4, state0, vl); + v5 = __riscv_vadd_vv_u32m1(v5, state1, vl); + v6 = __riscv_vadd_vv_u32m1(v6, state2, vl); + v7 = __riscv_vadd_vv_u32m1(v7, state3, vl); + state3 = ADD_U64(state3, one_u64); + + i4 = unaligned_load_u32m1(src + 4 * 16, vl); + i5 = unaligned_load_u32m1(src + 5 * 16, vl); + i6 = unaligned_load_u32m1(src + 6 * 16, vl); + i7 = unaligned_load_u32m1(src + 7 * 16, vl); + + unaligned_store_u32m1(dst + 0 * 16, v0, vl); + unaligned_store_u32m1(dst + 1 * 16, v1, vl); + unaligned_store_u32m1(dst + 2 * 16, v2, vl); + unaligned_store_u32m1(dst + 3 * 16, v3, vl); + + v4 = __riscv_vxor_vv_u32m1(i4, v4, vl); + v5 = __riscv_vxor_vv_u32m1(i5, v5, vl); + v6 = __riscv_vxor_vv_u32m1(i6, v6, vl); + v7 = __riscv_vxor_vv_u32m1(i7, v7, vl); + + unaligned_store_u32m1(dst + 4 * 16, v4, vl); + unaligned_store_u32m1(dst + 5 * 16, v5, vl); + unaligned_store_u32m1(dst + 6 * 16, v6, vl); + unaligned_store_u32m1(dst + 7 * 16, v7, vl); + + src += 2 * 64; + dst += 2 * 64; + + nblks -= 2; + } + + /* 1x block implementation */ + while (nblks) + { + v0 = state0; + v1 = state1; + v2 = state2; + v3 = state3; + + i0 = unaligned_load_u32m1(src + 0 * 16, vl); + i1 = unaligned_load_u32m1(src + 1 * 16, vl); + i2 = unaligned_load_u32m1(src + 2 * 16, vl); + i3 = unaligned_load_u32m1(src + 3 * 16, vl); + + for (i = 20; i > 0; i -= 2) + { + QUARTERROUND4(v0, v1, v2, v3, 1, 2, 3); + QUARTERROUND4(v0, v1, v2, v3, 3, 2, 1); + } + + v0 = __riscv_vadd_vv_u32m1(v0, state0, vl); + v1 = __riscv_vadd_vv_u32m1(v1, state1, vl); + v2 = __riscv_vadd_vv_u32m1(v2, state2, vl); + v3 = __riscv_vadd_vv_u32m1(v3, state3, vl); + + state3 = ADD_U64(state3, one_u64); + + v0 = __riscv_vxor_vv_u32m1(i0, v0, vl); + v1 = __riscv_vxor_vv_u32m1(i1, v1, vl); + v2 = __riscv_vxor_vv_u32m1(i2, v2, vl); + v3 = __riscv_vxor_vv_u32m1(i3, v3, vl); + unaligned_store_u32m1(dst + 0 * 16, v0, vl); + unaligned_store_u32m1(dst + 1 * 16, v1, vl); + unaligned_store_u32m1(dst + 2 * 16, v2, vl); + unaligned_store_u32m1(dst + 3 * 16, v3, vl); + src += 64; + dst += 64; + + nblks--; + } + } + + clear_vec_regs(); + + return 0; +} + +#endif /* GCRY_CHACHA20_RISCV_RVV_H */ diff --git a/cipher/chacha20-riscv-v.c b/cipher/chacha20-riscv-v.c index 94026305..32fc09cc 100644 --- a/cipher/chacha20-riscv-v.c +++ b/cipher/chacha20-riscv-v.c @@ -1,5 +1,5 @@ /* chacha20-riscv-v.c - RISC-V vector implementation of ChaCha20 - * Copyright (C) 2025 Jussi Kivilinna + * Copyright (C) 2025-2026 Jussi Kivilinna * * This file is part of Libgcrypt. * @@ -23,23 +23,8 @@ defined(HAVE_COMPATIBLE_CC_RISCV_VECTOR_INTRINSICS) && \ defined(USE_CHACHA20) -#include "simd-common-riscv.h" -#include -#include "bufhelp.h" - -#define ALWAYS_INLINE inline __attribute__((always_inline)) -#define NO_INLINE __attribute__((noinline)) -#define NO_INSTRUMENT_FUNCTION __attribute__((no_instrument_function)) - -#define ASM_FUNC_ATTR NO_INSTRUMENT_FUNCTION -#define ASM_FUNC_ATTR_INLINE ASM_FUNC_ATTR ALWAYS_INLINE -#define ASM_FUNC_ATTR_NOINLINE ASM_FUNC_ATTR NO_INLINE - - -/********************************************************************** - RISC-V vector extension chacha20 - **********************************************************************/ +#define LOAD_GATHER_ROTATE_CONSTANTS 1 #define ROTATE16(v) __riscv_vreinterpret_v_u16m1_u32m1( \ __riscv_vrgather_vv_u16m1( \ @@ -52,505 +37,9 @@ #define ROTATE(v, c) __riscv_vadd_vv_u32m1( \ __riscv_vsll_vx_u32m1((v), (c), vl), \ __riscv_vsrl_vx_u32m1((v), 32 - (c), vl), vl) -#define XOR(v, w) __riscv_vxor_vv_u32m1((v), (w), vl) -#define PLUS(v, w) __riscv_vadd_vv_u32m1((v), (w), vl) -#define WORD_ROL(v, c) __riscv_vrgather_vv_u32m1((v), (rol##c), vl) - -#define QUARTERROUND_4(a0, b0, c0, d0, a1, b1, c1, d1, \ - a2, b2, c2, d2, a3, b3, c3, d3) \ - a0 = PLUS(a0, b0); a1 = PLUS(a1, b1); \ - a2 = PLUS(a2, b2); a3 = PLUS(a3, b3); \ - d0 = XOR(d0, a0); d1 = XOR(d1, a1); \ - d2 = XOR(d2, a2); d3 = XOR(d3, a3); \ - d0 = ROTATE16(d0); d1 = ROTATE16(d1); \ - d2 = ROTATE16(d2); d3 = ROTATE16(d3); \ - c0 = PLUS(c0, d0); c1 = PLUS(c1, d1); \ - c2 = PLUS(c2, d2); c3 = PLUS(c3, d3); \ - b0 = XOR(b0, c0); b1 = XOR(b1, c1); \ - b2 = XOR(b2, c2); b3 = XOR(b3, c3); \ - b0 = ROTATE(b0, 12); b1 = ROTATE(b1, 12); \ - b2 = ROTATE(b2, 12); b3 = ROTATE(b3, 12); \ - a0 = PLUS(a0, b0); a1 = PLUS(a1, b1); \ - a2 = PLUS(a2, b2); a3 = PLUS(a3, b3); \ - d0 = XOR(d0, a0); d1 = XOR(d1, a1); \ - d2 = XOR(d2, a2); d3 = XOR(d3, a3); \ - d0 = ROTATE8(d0); d1 = ROTATE8(d1); \ - d2 = ROTATE8(d2); d3 = ROTATE8(d3); \ - c0 = PLUS(c0, d0); c1 = PLUS(c1, d1); \ - c2 = PLUS(c2, d2); c3 = PLUS(c3, d3); \ - b0 = XOR(b0, c0); b1 = XOR(b1, c1); \ - b2 = XOR(b2, c2); b3 = XOR(b3, c3); \ - b0 = ROTATE(b0, 7); b1 = ROTATE(b1, 7); \ - b2 = ROTATE(b2, 7); b3 = ROTATE(b3, 7); - -#define QUARTERROUND4_2(x0, x1, x2, x3, y0, y1, y2, y3, rol_x1, rol_x2, rol_x3) \ - x0 = PLUS(x0, x1); y0 = PLUS(y0, y1); \ - x3 = XOR(x3, x0); y3 = XOR(y3, y0); \ - x3 = ROTATE16(x3); y3 = ROTATE16(y3); \ - x2 = PLUS(x2, x3); y2 = PLUS(y2, y3); \ - x1 = XOR(x1, x2); y1 = XOR(y1, y2); \ - x1 = ROTATE(x1, 12); y1 = ROTATE(y1, 12); \ - x0 = PLUS(x0, x1); y0 = PLUS(y0, y1); \ - x3 = XOR(x3, x0); y3 = XOR(y3, y0); \ - x3 = ROTATE8(x3); y3 = ROTATE8(y3); \ - x2 = PLUS(x2, x3); y2 = PLUS(y2, y3); \ - x3 = WORD_ROL(x3, rol_x3); y3 = WORD_ROL(y3, rol_x3);\ - x1 = XOR(x1, x2); y1 = XOR(y1, y2); \ - x2 = WORD_ROL(x2, rol_x2); y2 = WORD_ROL(y2, rol_x2); \ - x1 = ROTATE(x1, 7); y1 = ROTATE(y1, 7); \ - x1 = WORD_ROL(x1, rol_x1); y1 = WORD_ROL(y1, rol_x1); - -#define QUARTERROUND4(x0, x1, x2, x3, rol_x1, rol_x2, rol_x3) \ - x0 = PLUS(x0, x1); x3 = XOR(x3, x0); x3 = ROTATE16(x3); \ - x2 = PLUS(x2, x3); x1 = XOR(x1, x2); x1 = ROTATE(x1, 12); \ - x0 = PLUS(x0, x1); x3 = XOR(x3, x0); x3 = ROTATE8(x3); \ - x2 = PLUS(x2, x3); \ - x3 = WORD_ROL(x3, rol_x3); \ - x1 = XOR(x1, x2); \ - x2 = WORD_ROL(x2, rol_x2); \ - x1= ROTATE(x1, 7); \ - x1 = WORD_ROL(x1, rol_x1); - -#define ADD_U64(a, b) __riscv_vreinterpret_v_u64m1_u32m1( \ - __riscv_vadd_vv_u64m1( \ - __riscv_vreinterpret_v_u32m1_u64m1(a), \ - __riscv_vreinterpret_v_u32m1_u64m1(b), vl / 2)) - -#define vxor_v_u32m1_u32m1x8(data, idx, vs, vl) \ - __riscv_vset_v_u32m1_u32m1x8((data), (idx), \ - __riscv_vxor_vv_u32m1( \ - __riscv_vget_v_u32m1x8_u32m1((data), (idx)), (vs), (vl))) - -static ASM_FUNC_ATTR_INLINE vuint16m1_t -gen_rot16(size_t vl) -{ - return __riscv_vxor_vx_u16m1(__riscv_vid_v_u16m1(vl * 2), 1, vl * 2); -} - -static ASM_FUNC_ATTR_INLINE vuint8m1_t -gen_rot8(size_t vl) -{ - vuint8m1_t rot8, rot8_hi; - - rot8 = __riscv_vid_v_u8m1(vl * 4); - rot8_hi = __riscv_vand_vx_u8m1(rot8, ~3, vl * 4); - rot8 = __riscv_vadd_vx_u8m1(rot8, 3, vl * 4); - rot8 = __riscv_vand_vx_u8m1(rot8, 3, vl * 4); - rot8 = __riscv_vadd_vv_u8m1(rot8, rot8_hi, vl * 4); - - return rot8; -} - -static ASM_FUNC_ATTR_INLINE vuint16m2_t -gen_indexes(size_t vl, size_t stride) -{ - vuint16m2_t idx = __riscv_vid_v_u16m2(vl * 4); - vuint16m2_t idx_lo = __riscv_vand_vx_u16m2(idx, 3, vl * 4); - vuint16m2_t idx_hi = __riscv_vsrl_vx_u16m2(idx, 2, vl * 4); - idx_hi = __riscv_vmul_vx_u16m2(idx_hi, stride, vl * 4); - return __riscv_vadd_vv_u16m2(idx_hi, idx_lo, vl * 4); -} - -static ASM_FUNC_ATTR_INLINE vuint32m1_t -unaligned_load_u32m1(const void *src, size_t vl) -{ -#ifdef RVV_UNALIGNED_NOT_ALLOWED - return __riscv_vreinterpret_v_u8m1_u32m1(__riscv_vle8_v_u8m1(src, vl * 4)); -#else - return __riscv_vle32_v_u32m1(src, vl); -#endif -} - -static ASM_FUNC_ATTR_INLINE void -unaligned_store_u32m1(void *dst, vuint32m1_t vec, size_t vl) -{ -#ifdef RVV_UNALIGNED_NOT_ALLOWED - __riscv_vse8_v_u8m1(dst, __riscv_vreinterpret_v_u32m1_u8m1(vec), vl * 4); -#else - __riscv_vse32_v_u32m1(dst, vec, vl); -#endif -} - -static ASM_FUNC_ATTR_INLINE vuint32m1x8_t -unaligned_vlsseg8e32_v_u32m1x8(const void *src, size_t vl) -{ -#ifdef RVV_UNALIGNED_NOT_ALLOWED - if (UNLIKELY(((uintptr_t)src & 3) != 0)) - { - const byte *bsrc = src; - vuint16m2_t indexes; - vuint8m1_t b0, b1, b2, b3, b4, b5, b6, b7; - vuint32m1x8_t data; - - indexes = gen_indexes(4 * vl, 64); - - b0 = __riscv_vluxei16_v_u8m1(bsrc + 0 * 4, indexes, vl * 4); - b1 = __riscv_vluxei16_v_u8m1(bsrc + 1 * 4, indexes, vl * 4); - b2 = __riscv_vluxei16_v_u8m1(bsrc + 2 * 4, indexes, vl * 4); - b3 = __riscv_vluxei16_v_u8m1(bsrc + 3 * 4, indexes, vl * 4); - b4 = __riscv_vluxei16_v_u8m1(bsrc + 4 * 4, indexes, vl * 4); - b5 = __riscv_vluxei16_v_u8m1(bsrc + 5 * 4, indexes, vl * 4); - b6 = __riscv_vluxei16_v_u8m1(bsrc + 6 * 4, indexes, vl * 4); - b7 = __riscv_vluxei16_v_u8m1(bsrc + 7 * 4, indexes, vl * 4); - - data = __riscv_vundefined_u32m1x8(); - data = __riscv_vset_v_u32m1_u32m1x8( - data, 0, __riscv_vreinterpret_v_u8m1_u32m1(b0)); - data = __riscv_vset_v_u32m1_u32m1x8( - data, 1, __riscv_vreinterpret_v_u8m1_u32m1(b1)); - data = __riscv_vset_v_u32m1_u32m1x8( - data, 2, __riscv_vreinterpret_v_u8m1_u32m1(b2)); - data = __riscv_vset_v_u32m1_u32m1x8( - data, 3, __riscv_vreinterpret_v_u8m1_u32m1(b3)); - data = __riscv_vset_v_u32m1_u32m1x8( - data, 4, __riscv_vreinterpret_v_u8m1_u32m1(b4)); - data = __riscv_vset_v_u32m1_u32m1x8( - data, 5, __riscv_vreinterpret_v_u8m1_u32m1(b5)); - data = __riscv_vset_v_u32m1_u32m1x8( - data, 6, __riscv_vreinterpret_v_u8m1_u32m1(b6)); - data = __riscv_vset_v_u32m1_u32m1x8( - data, 7, __riscv_vreinterpret_v_u8m1_u32m1(b7)); - - return data; - } - else -#endif - { - /* Fast path for 32-bit aligned loads. */ - return __riscv_vlsseg8e32_v_u32m1x8(src, 64, vl); - } -} - -static ASM_FUNC_ATTR_INLINE void -unaligned_vssseg8e32_v_u32m1x8(void *dst, vuint32m1x8_t data, size_t vl) -{ -#ifdef RVV_UNALIGNED_NOT_ALLOWED - if (UNLIKELY(((uintptr_t)dst & 3) != 0)) - { - byte *bdst = dst; - vuint16m2_t indexes; - vuint8m1_t b0, b1, b2, b3, b4, b5, b6, b7; - - indexes = gen_indexes(4 * vl, 64); - - b0 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x8_u32m1(data, 0)); - b1 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x8_u32m1(data, 1)); - b2 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x8_u32m1(data, 2)); - b3 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x8_u32m1(data, 3)); - b4 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x8_u32m1(data, 4)); - b5 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x8_u32m1(data, 5)); - b6 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x8_u32m1(data, 6)); - b7 = __riscv_vreinterpret_v_u32m1_u8m1(__riscv_vget_v_u32m1x8_u32m1(data, 7)); - - __riscv_vsuxei16_v_u8m1(bdst + 0 * 4, indexes, b0, vl * 4); - __riscv_vsuxei16_v_u8m1(bdst + 1 * 4, indexes, b1, vl * 4); - __riscv_vsuxei16_v_u8m1(bdst + 2 * 4, indexes, b2, vl * 4); - __riscv_vsuxei16_v_u8m1(bdst + 3 * 4, indexes, b3, vl * 4); - __riscv_vsuxei16_v_u8m1(bdst + 4 * 4, indexes, b4, vl * 4); - __riscv_vsuxei16_v_u8m1(bdst + 5 * 4, indexes, b5, vl * 4); - __riscv_vsuxei16_v_u8m1(bdst + 6 * 4, indexes, b6, vl * 4); - __riscv_vsuxei16_v_u8m1(bdst + 7 * 4, indexes, b7, vl * 4); - } - else -#endif - { - /* Fast path for 32-bit aligned stores. */ - __riscv_vssseg8e32_v_u32m1x8(dst, 64, data, vl); - return; - } -} - -static ASM_FUNC_ATTR_INLINE unsigned int -chacha20_rvv_blocks(u32 *input, byte *dst, const byte *src, size_t nblks) -{ - unsigned int i; - - if (nblks == 0) - return 0; - - /* Try use vector implementation when there is 4 or more blocks. */ - if (nblks >= 4) - { - size_t vl = __riscv_vsetvl_e32m1(nblks) < 4 - ? __riscv_vsetvl_e32m1(4) : __riscv_vsetvl_e32m1(nblks); - vuint32m1_t x0, x1, x2, x3, x4, x5, x6, x7; - vuint32m1_t x8, x9, x10, x11, x12, x13, x14, x15; - u32 s0, s1, s2, s3, s4, s5, s6, s7; - u32 s8, s9, s10, s11, s12, s13, s14, s15; - vuint16m1_t rot16 = gen_rot16(vl); - vuint8m1_t rot8 = gen_rot8(vl); - - s0 = input[0]; - s1 = input[1]; - s2 = input[2]; - s3 = input[3]; - s4 = input[4]; - s5 = input[5]; - s6 = input[6]; - s7 = input[7]; - s8 = input[8]; - s9 = input[9]; - s10 = input[10]; - s11 = input[11]; - s12 = input[12]; - s13 = input[13]; - s14 = input[14]; - s15 = input[15]; - - while (nblks >= 4) - { - vuint32m1_t ctr; - vbool32_t carry; - vuint32m1x8_t data; - - if (vl < 4) - break; - - x0 = __riscv_vmv_v_x_u32m1(s0, vl); - x1 = __riscv_vmv_v_x_u32m1(s1, vl); - x2 = __riscv_vmv_v_x_u32m1(s2, vl); - x3 = __riscv_vmv_v_x_u32m1(s3, vl); - x4 = __riscv_vmv_v_x_u32m1(s4, vl); - x5 = __riscv_vmv_v_x_u32m1(s5, vl); - x6 = __riscv_vmv_v_x_u32m1(s6, vl); - x7 = __riscv_vmv_v_x_u32m1(s7, vl); - x8 = __riscv_vmv_v_x_u32m1(s8, vl); - x9 = __riscv_vmv_v_x_u32m1(s9, vl); - x10 = __riscv_vmv_v_x_u32m1(s10, vl); - x11 = __riscv_vmv_v_x_u32m1(s11, vl); - x13 = __riscv_vmv_v_x_u32m1(s13, vl); - x14 = __riscv_vmv_v_x_u32m1(s14, vl); - x15 = __riscv_vmv_v_x_u32m1(s15, vl); - - ctr = __riscv_vid_v_u32m1(vl); - carry = __riscv_vmadc_vx_u32m1_b32(ctr, s12, vl); - ctr = __riscv_vadd_vx_u32m1(ctr, s12, vl); - x12 = ctr; - x13 = __riscv_vadc_vxm_u32m1(x13, 0, carry, vl); - - for (i = 20; i > 0; i -= 2) - { - QUARTERROUND_4(x0, x4, x8, x12, - x1, x5, x9, x13, - x2, x6, x10, x14, - x3, x7, x11, x15); - QUARTERROUND_4(x0, x5, x10, x15, - x1, x6, x11, x12, - x2, x7, x8, x13, - x3, x4, x9, x14); - } - - x0 = __riscv_vadd_vx_u32m1(x0, s0, vl); - x1 = __riscv_vadd_vx_u32m1(x1, s1, vl); - x2 = __riscv_vadd_vx_u32m1(x2, s2, vl); - x3 = __riscv_vadd_vx_u32m1(x3, s3, vl); - x4 = __riscv_vadd_vx_u32m1(x4, s4, vl); - x5 = __riscv_vadd_vx_u32m1(x5, s5, vl); - x6 = __riscv_vadd_vx_u32m1(x6, s6, vl); - x7 = __riscv_vadd_vx_u32m1(x7, s7, vl); - x8 = __riscv_vadd_vx_u32m1(x8, s8, vl); - x9 = __riscv_vadd_vx_u32m1(x9, s9, vl); - x10 = __riscv_vadd_vx_u32m1(x10, s10, vl); - x11 = __riscv_vadd_vx_u32m1(x11, s11, vl); - x12 = __riscv_vadd_vv_u32m1(x12, ctr, vl); - x13 = __riscv_vadc_vxm_u32m1(x13, s13, carry, vl); - x14 = __riscv_vadd_vx_u32m1(x14, s14, vl); - x15 = __riscv_vadd_vx_u32m1(x15, s15, vl); - - s12 += vl; - s13 += s12 < vl; - - data = unaligned_vlsseg8e32_v_u32m1x8((const void *)src, vl); - - data = vxor_v_u32m1_u32m1x8(data, 0, x0, vl); - data = vxor_v_u32m1_u32m1x8(data, 1, x1, vl); - data = vxor_v_u32m1_u32m1x8(data, 2, x2, vl); - data = vxor_v_u32m1_u32m1x8(data, 3, x3, vl); - data = vxor_v_u32m1_u32m1x8(data, 4, x4, vl); - data = vxor_v_u32m1_u32m1x8(data, 5, x5, vl); - data = vxor_v_u32m1_u32m1x8(data, 6, x6, vl); - data = vxor_v_u32m1_u32m1x8(data, 7, x7, vl); - - unaligned_vssseg8e32_v_u32m1x8((void *)dst, data, vl); - - data = unaligned_vlsseg8e32_v_u32m1x8((const void *)(src + 32), vl); - - data = vxor_v_u32m1_u32m1x8(data, 0, x8, vl); - data = vxor_v_u32m1_u32m1x8(data, 1, x9, vl); - data = vxor_v_u32m1_u32m1x8(data, 2, x10, vl); - data = vxor_v_u32m1_u32m1x8(data, 3, x11, vl); - data = vxor_v_u32m1_u32m1x8(data, 4, x12, vl); - data = vxor_v_u32m1_u32m1x8(data, 5, x13, vl); - data = vxor_v_u32m1_u32m1x8(data, 6, x14, vl); - data = vxor_v_u32m1_u32m1x8(data, 7, x15, vl); - - unaligned_vssseg8e32_v_u32m1x8((void *)(dst + 32), data, vl); - - src += vl * 64; - dst += vl * 64; - nblks -= vl; - vl = __riscv_vsetvl_e32m1(nblks) < 4 - ? __riscv_vsetvl_e32m1(4) : __riscv_vsetvl_e32m1(nblks); - } - - input[12] = s12; - input[13] = s13; - } - - /* Use SIMD implementation for remaining blocks. */ - if (nblks > 0) - { - static const u32 rol_const[3][4] = - { - { 1, 2, 3, 0 }, - { 2, 3, 0, 1 }, - { 3, 0, 1, 2 } - }; - static const u32 one_u64_const[4] = { 1, 0, 0, 0 }; - size_t vl = 4; - vuint32m1_t rol1, rol2, rol3; - vuint32m1_t one_u64; - vuint32m1_t v0, v1, v2, v3; - vuint32m1_t v4, v5, v6, v7; - vuint32m1_t state0, state1, state2, state3; - vuint32m1_t i0, i1, i2, i3; - vuint32m1_t i4, i5, i6, i7; - vuint16m1_t rot16 = gen_rot16(vl); - vuint8m1_t rot8 = gen_rot8(vl); - - rol1 = __riscv_vle32_v_u32m1(rol_const[0], vl); - rol2 = __riscv_vle32_v_u32m1(rol_const[1], vl); - rol3 = __riscv_vle32_v_u32m1(rol_const[2], vl); - one_u64 = __riscv_vle32_v_u32m1(one_u64_const, vl); - - state0 = __riscv_vle32_v_u32m1(&input[0], vl); - state1 = __riscv_vle32_v_u32m1(&input[4], vl); - state2 = __riscv_vle32_v_u32m1(&input[8], vl); - state3 = __riscv_vle32_v_u32m1(&input[12], vl); - - input[12] += nblks; - input[13] += input[12] < nblks; - - /* SIMD 2x block implementation */ - while (nblks >= 2) - { - v0 = state0; - v1 = state1; - v2 = state2; - v3 = state3; - - v4 = state0; - v5 = state1; - v6 = state2; - v7 = state3; - v7 = ADD_U64(v7, one_u64); - - i0 = unaligned_load_u32m1(src + 0 * 16, vl); - i1 = unaligned_load_u32m1(src + 1 * 16, vl); - i2 = unaligned_load_u32m1(src + 2 * 16, vl); - i3 = unaligned_load_u32m1(src + 3 * 16, vl); - - for (i = 20; i > 0; i -= 2) - { - QUARTERROUND4_2(v0, v1, v2, v3, v4, v5, v6, v7, 1, 2, 3); - QUARTERROUND4_2(v0, v1, v2, v3, v4, v5, v6, v7, 3, 2, 1); - } - - v0 = __riscv_vadd_vv_u32m1(v0, state0, vl); - v1 = __riscv_vadd_vv_u32m1(v1, state1, vl); - v2 = __riscv_vadd_vv_u32m1(v2, state2, vl); - v3 = __riscv_vadd_vv_u32m1(v3, state3, vl); - state3 = ADD_U64(state3, one_u64); - - v0 = __riscv_vxor_vv_u32m1(i0, v0, vl); - v1 = __riscv_vxor_vv_u32m1(i1, v1, vl); - v2 = __riscv_vxor_vv_u32m1(i2, v2, vl); - v3 = __riscv_vxor_vv_u32m1(i3, v3, vl); - - v4 = __riscv_vadd_vv_u32m1(v4, state0, vl); - v5 = __riscv_vadd_vv_u32m1(v5, state1, vl); - v6 = __riscv_vadd_vv_u32m1(v6, state2, vl); - v7 = __riscv_vadd_vv_u32m1(v7, state3, vl); - state3 = ADD_U64(state3, one_u64); - - i4 = unaligned_load_u32m1(src + 4 * 16, vl); - i5 = unaligned_load_u32m1(src + 5 * 16, vl); - i6 = unaligned_load_u32m1(src + 6 * 16, vl); - i7 = unaligned_load_u32m1(src + 7 * 16, vl); - - unaligned_store_u32m1(dst + 0 * 16, v0, vl); - unaligned_store_u32m1(dst + 1 * 16, v1, vl); - unaligned_store_u32m1(dst + 2 * 16, v2, vl); - unaligned_store_u32m1(dst + 3 * 16, v3, vl); - - v4 = __riscv_vxor_vv_u32m1(i4, v4, vl); - v5 = __riscv_vxor_vv_u32m1(i5, v5, vl); - v6 = __riscv_vxor_vv_u32m1(i6, v6, vl); - v7 = __riscv_vxor_vv_u32m1(i7, v7, vl); - - unaligned_store_u32m1(dst + 4 * 16, v4, vl); - unaligned_store_u32m1(dst + 5 * 16, v5, vl); - unaligned_store_u32m1(dst + 6 * 16, v6, vl); - unaligned_store_u32m1(dst + 7 * 16, v7, vl); - - src += 2 * 64; - dst += 2 * 64; - - nblks -= 2; - } - - /* 1x block implementation */ - while (nblks) - { - v0 = state0; - v1 = state1; - v2 = state2; - v3 = state3; - - i0 = unaligned_load_u32m1(src + 0 * 16, vl); - i1 = unaligned_load_u32m1(src + 1 * 16, vl); - i2 = unaligned_load_u32m1(src + 2 * 16, vl); - i3 = unaligned_load_u32m1(src + 3 * 16, vl); - - for (i = 20; i > 0; i -= 2) - { - QUARTERROUND4(v0, v1, v2, v3, 1, 2, 3); - QUARTERROUND4(v0, v1, v2, v3, 3, 2, 1); - } - - v0 = __riscv_vadd_vv_u32m1(v0, state0, vl); - v1 = __riscv_vadd_vv_u32m1(v1, state1, vl); - v2 = __riscv_vadd_vv_u32m1(v2, state2, vl); - v3 = __riscv_vadd_vv_u32m1(v3, state3, vl); - - state3 = ADD_U64(state3, one_u64); - - v0 = __riscv_vxor_vv_u32m1(i0, v0, vl); - v1 = __riscv_vxor_vv_u32m1(i1, v1, vl); - v2 = __riscv_vxor_vv_u32m1(i2, v2, vl); - v3 = __riscv_vxor_vv_u32m1(i3, v3, vl); - unaligned_store_u32m1(dst + 0 * 16, v0, vl); - unaligned_store_u32m1(dst + 1 * 16, v1, vl); - unaligned_store_u32m1(dst + 2 * 16, v2, vl); - unaligned_store_u32m1(dst + 3 * 16, v3, vl); - src += 64; - dst += 64; - - nblks--; - } - } - - clear_vec_regs(); - - return 0; -} -#ifdef HAVE_GCC_ATTRIBUTE_OPTIMIZE -# define FUNC_ATTR_OPT_O2 __attribute__((optimize("-O2"))) -#else -# define FUNC_ATTR_OPT_O2 -#endif +#include "chacha20-riscv-rvv.h" unsigned int ASM_FUNC_ATTR FUNC_ATTR_OPT_O2 diff --git a/cipher/chacha20-riscv-zvkb.c b/cipher/chacha20-riscv-zvkb.c new file mode 100644 index 00000000..abb34e4b --- /dev/null +++ b/cipher/chacha20-riscv-zvkb.c @@ -0,0 +1,43 @@ +/* chacha20-riscv-zvkb.c - RISC-V Zvkb implementation of ChaCha20 + * Copyright (C) 2025-2026 Jussi Kivilinna + * + * This file is part of Libgcrypt. + * + * Libgcrypt is free software; you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as + * published by the Free Software Foundation; either version 2.1 of + * the License, or (at your option) any later version. + * + * Libgcrypt is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this program; if not, see . + */ + +#include + +#if defined (__riscv) && \ + defined(HAVE_COMPATIBLE_CC_RISCV_VECTOR_INTRINSICS) && \ + defined(HAVE_COMPATIBLE_CC_RISCV_VECTOR_CRYPTO_INTRINSICS) && \ + defined(USE_CHACHA20) + + +#define ROTATE(v, c) __riscv_vror_vx_u32m1(v, 32 - (c), vl) +#define ROTATE16(v) ROTATE(v, 16) +#define ROTATE8(v) ROTATE(v, 8) + + +#include "chacha20-riscv-rvv.h" + + +unsigned int ASM_FUNC_ATTR FUNC_ATTR_OPT_O2 +_gcry_chacha20_riscv_zvkb_blocks(u32 *state, byte *dst, const byte *src, + size_t nblks) +{ + return chacha20_rvv_blocks(state, dst, src, nblks); +} + +#endif /* HAVE_COMPATIBLE_CC_RISCV_VECTOR_CRYPTO_INTRINSICS */ diff --git a/cipher/chacha20.c b/cipher/chacha20.c index 0bf31be7..64946835 100644 --- a/cipher/chacha20.c +++ b/cipher/chacha20.c @@ -120,6 +120,13 @@ # define USE_RISCV_V 1 #endif +/* USE_RISCV_ZVKB indicates whether to enable RISC-V Zvkb extension code. */ +#undef USE_RISCV_ZVKB +#if defined (USE_RISCV_V) \ + && defined(HAVE_COMPATIBLE_CC_RISCV_VECTOR_CRYPTO_INTRINSICS) +# define USE_RISCV_ZVKB 1 +#endif + /* Assembly implementations use SystemV ABI, ABI conversion and additional * stack to store XMM6-XMM15 needed on Win64. */ #undef ASM_FUNC_ABI @@ -148,6 +155,7 @@ typedef struct CHACHA20_context_s unsigned int use_p10:1; unsigned int use_s390x:1; unsigned int use_riscv_v:1; + unsigned int use_riscv_zvkb:1; } CHACHA20_context_t; @@ -280,6 +288,14 @@ unsigned int _gcry_chacha20_riscv_v_check_hw(void); #endif /* USE_RISCV_V */ +#ifdef USE_RISCV_ZVKB + +unsigned int _gcry_chacha20_riscv_zvkb_blocks(u32 *state, byte *dst, + const byte *src, + size_t nblks); + +#endif /* USE_RISCV_ZVKB */ + static const char *selftest (void); @@ -446,6 +462,13 @@ chacha20_blocks (CHACHA20_context_t *ctx, byte *dst, const byte *src, } #endif +#ifdef USE_RISCV_ZVKB + if (ctx->use_riscv_zvkb) + { + return _gcry_chacha20_riscv_zvkb_blocks(ctx->input, dst, src, nblks); + } +#endif + #ifdef USE_RISCV_V if (ctx->use_riscv_v) { @@ -608,6 +631,11 @@ chacha20_do_setkey (CHACHA20_context_t *ctx, && (features & HWF_RISCV_V) /* Optional in RVA22U64 */ && _gcry_chacha20_riscv_v_check_hw(); #endif +#ifdef USE_RISCV_ZVKB + ctx->use_riscv_zvkb = + ctx->use_riscv_v + && (features & HWF_RISCV_ZVKB) /* Mandatory in RVA23U64 (Zvbb) */; +#endif (void)features; diff --git a/configure.ac b/configure.ac index d9cbf5d2..fb778e38 100644 --- a/configure.ac +++ b/configure.ac @@ -3036,6 +3036,16 @@ m4_define([GCRY_RISCV_VECTOR_CRYPTO_INTRINSICS_TEST], vuint32m1_t b = __riscv_vrev8_v_u32m1(a, vl); __riscv_vse32_v_u32m1(ptr + 0 * vl, b, vl); } + void test_rotate(unsigned int *ptr) + { + int vl = __riscv_vsetvl_e32m1 (4); + vuint32m1_t a = __riscv_vle32_v_u32m1(ptr + 0 * vl, vl); + vuint32m1_t b = __riscv_vror_vx_u32m1(a, 3, vl); + b = __riscv_vror_vv_u32m1(b, a, vl); + b = __riscv_vrol_vx_u32m1(b, 3, vl); + b = __riscv_vrol_vv_u32m1(b, a, vl); + __riscv_vse32_v_u32m1(ptr + 0 * vl, b, vl); + } void test_inline_vec_asm(unsigned int *ptr) { int vl = __riscv_vsetvl_e32m1 (4); @@ -3058,6 +3068,7 @@ m4_define([GCRY_RISCV_VECTOR_CRYPTO_INTRINSICS_TEST], test_aes_crypt(buf); test_ghash(buf); test_rev8(buf); + test_rotate(buf); test_inline_vec_asm(buf); return 0; } @@ -3906,6 +3917,7 @@ if test "$found" = "1" ; then riscv64-*-*) # Build with the RISC-V vector implementation GCRYPT_ASM_CIPHERS="$GCRYPT_ASM_CIPHERS chacha20-riscv-v.lo" + GCRYPT_ASM_CIPHERS="$GCRYPT_ASM_CIPHERS chacha20-riscv-zvkb.lo" ;; esac fi -- 2.53.0 From jussi.kivilinna at iki.fi Sun Oct 11 19:23:19 2026 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sun, 11 Oct 2026 20:23:19 +0300 Subject: [PATCH 10/12] rijndael-vp-riscv: use LMUL that gives 128-bit register group In-Reply-To: <20261011172321.997183-1-jussi.kivilinna@iki.fi> References: <20261011172321.997183-1-jussi.kivilinna@iki.fi> Message-ID: <20261011172321.997183-10-jussi.kivilinna@iki.fi> * cipher/Makefile.am: Add 'rijndael-vp-riscv.h', 'rijndael-vp-riscv-mf2.c', 'rijndael-vp-riscv-mf4.c' and 'rijndael-vp-riscv-mf8.c'. * cipher/rijndael-internal.h (rijndael_setkeyfn_t): New. * cipher/rijndael-vp-riscv-mf2.c: New. * cipher/rijndael-vp-riscv-mf4.c: New. * cipher/rijndael-vp-riscv-mf8.c: New. * cipher/rijndael-vp-riscv.h: New. * cipher/rijndael-vp-riscv.c: Move SIMD128 macros and function names to 'rijndael-vp-riscv.h' and build LMUL=1 variant. (_gcry_aes_vp_riscv_setup_acceleration): Add 'hw_setkey' and 'bulk_ops' parameters and setup functions from variant selected by VLEN. * cipher/rijndael-vp-simd128.h [HAVE_SIMD256]: Remove. (FUNC_CFB_ENC, FUNC_CBC_ENC, FUNC_CTR_ENC, FUNC_CTR32LE_ENC) (FUNC_CFB_DEC, FUNC_CBC_DEC): Use bulk function prototypes. (aes_simd128_ecb_enc, aes_simd128_ecb_dec): Make static. * cipher/rijndael.c [USE_VP_RISCV] (_gcry_aes_vp_riscv_do_setkey) (_gcry_aes_vp_riscv_prepare_decryption, _gcry_aes_vp_riscv_encrypt) (_gcry_aes_vp_riscv_decrypt, _gcry_aes_vp_riscv_cfb_enc) (_gcry_aes_vp_riscv_cbc_enc, _gcry_aes_vp_riscv_ctr_enc) (_gcry_aes_vp_riscv_ctr32le_enc, _gcry_aes_vp_riscv_cfb_dec) (_gcry_aes_vp_riscv_cbc_dec, _gcry_aes_vp_riscv_ocb_crypt) (_gcry_aes_vp_riscv_ocb_auth, _gcry_aes_vp_riscv_ecb_crypt) (_gcry_aes_vp_riscv_xts_crypt): Remove. (_gcry_aes_vp_riscv_setup_acceleration): Add 'hw_setkey' and 'bulk_ops' parameters. (do_setkey) [USE_VP_RISCV]: Move function setup to _gcry_aes_vp_riscv_setup_acceleration. * configure.ac: Add 'rijndael-vp-riscv-mf2.lo', 'rijndael-vp-riscv-mf4.lo' and 'rijndael-vp-riscv-mf8.lo'. -- Vector permute AES used LMUL=1 for 128-bit vectors and, with VLEN>=256, LMUL=1 for 256-bit vectors. On SpacemiT K1 and K3, vrgather cost grows with register group size, which made shuffles slow with VLEN larger than 128 bits. Now implementation is built for LMUL=1, 1/2, 1/4 and 1/8 and variant that gives 128-bit register group is selected by VLEN. 256-bit code-path is removed as it was slower on SpacemiT K1. Benchmark on SpacemiT K1 (in-order, VLEN=256): Before: AES | nanosecs/byte mebibytes/sec cycles/byte auto Mhz ECB enc | 16.79 ns/B 56.80 MiB/s 26.82 c/B 1597 ECB dec | 20.12 ns/B 47.41 MiB/s 32.15 c/B 1598 CBC enc | 32.05 ns/B 29.76 MiB/s 51.18 c/B 1597 CBC dec | 20.71 ns/B 46.05 MiB/s 33.07 c/B 1597 CFB enc | 32.09 ns/B 29.72 MiB/s 51.25 c/B 1597 CFB dec | 16.86 ns/B 56.56 MiB/s 26.97 c/B 1600 CTR enc | 17.12 ns/B 55.70 MiB/s 27.37 c/B 1599 XTS enc | 17.46 ns/B 54.63 MiB/s 27.93 c/B 1600 OCB enc | 17.19 ns/B 55.49 MiB/s 27.50 c/B 1600 After (ECB enc ~1.2x, CTR ~1.3x, CBC enc ~1.6x faster): AES | nanosecs/byte mebibytes/sec cycles/byte auto Mhz ECB enc | 13.53 ns/B 70.51 MiB/s 21.64 c/B 1600 ECB dec | 16.32 ns/B 58.43 MiB/s 26.12 c/B 1600 CBC enc | 19.36 ns/B 49.27 MiB/s 30.96 c/B 1600 CBC dec | 16.74 ns/B 56.98 MiB/s 26.76 c/B 1599 CFB enc | 19.54 ns/B 48.81 MiB/s 31.26 c/B 1600 CFB dec | 12.66 ns/B 75.33 MiB/s 20.25 c/B 1600 CTR enc | 13.07 ns/B 72.96 MiB/s 20.91 c/B 1600 XTS enc | 14.15 ns/B 67.39 MiB/s 22.62 c/B 1599 OCB enc | 14.39 ns/B 66.28 MiB/s 23.02 c/B 1600 Signed-off-by: Jussi Kivilinna --- cipher/Makefile.am | 26 ++ cipher/rijndael-internal.h | 2 + cipher/rijndael-vp-riscv-mf2.c | 29 ++ cipher/rijndael-vp-riscv-mf4.c | 29 ++ cipher/rijndael-vp-riscv-mf8.c | 29 ++ cipher/rijndael-vp-riscv.c | 268 ++---------- cipher/rijndael-vp-riscv.h | 190 +++++++++ cipher/rijndael-vp-simd128.h | 743 ++------------------------------- cipher/rijndael.c | 73 +--- configure.ac | 3 + 10 files changed, 375 insertions(+), 1017 deletions(-) create mode 100644 cipher/rijndael-vp-riscv-mf2.c create mode 100644 cipher/rijndael-vp-riscv-mf4.c create mode 100644 cipher/rijndael-vp-riscv-mf8.c create mode 100644 cipher/rijndael-vp-riscv.h diff --git a/cipher/Makefile.am b/cipher/Makefile.am index 23c4655c..65b03d76 100644 --- a/cipher/Makefile.am +++ b/cipher/Makefile.am @@ -128,6 +128,8 @@ EXTRA_libcipher_la_SOURCES = \ rijndael-riscv-zvkned.c \ rijndael-s390x.c \ rijndael-vp-aarch64.c rijndael-vp-riscv.c \ + rijndael-vp-riscv.h rijndael-vp-riscv-mf2.c \ + rijndael-vp-riscv-mf4.c rijndael-vp-riscv-mf8.c \ rijndael-vp-simd128.h \ rmd160.c \ rsa.c \ @@ -466,6 +468,30 @@ rijndael-vp-riscv.lo: $(srcdir)/rijndael-vp-riscv.c Makefile `echo $(LTCOMPILE) $(riscv_vector_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` $(custom_depfile_lo) +rijndael-vp-riscv-mf2.o: $(srcdir)/rijndael-vp-riscv-mf2.c Makefile + `echo $(COMPILE) $(riscv_vector_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) + +rijndael-vp-riscv-mf2.lo: $(srcdir)/rijndael-vp-riscv-mf2.c Makefile + `echo $(LTCOMPILE) $(riscv_vector_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) + +rijndael-vp-riscv-mf4.o: $(srcdir)/rijndael-vp-riscv-mf4.c Makefile + `echo $(COMPILE) $(riscv_vector_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) + +rijndael-vp-riscv-mf4.lo: $(srcdir)/rijndael-vp-riscv-mf4.c Makefile + `echo $(LTCOMPILE) $(riscv_vector_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) + +rijndael-vp-riscv-mf8.o: $(srcdir)/rijndael-vp-riscv-mf8.c Makefile + `echo $(COMPILE) $(riscv_vector_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_o) + +rijndael-vp-riscv-mf8.lo: $(srcdir)/rijndael-vp-riscv-mf8.c Makefile + `echo $(LTCOMPILE) $(riscv_vector_cflags) $(custom_depflags) -c $< | $(instrumentation_munging) ` + $(custom_depfile_lo) + if ENABLE_RISCV_VECTOR_CRYPTO_INTRINSICS_EXTRA_CFLAGS riscv_vector_crypto_zvkb_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvkb riscv_vector_crypto_aes_cflags = -O2 -march=@MARCH_RVA23U64_BASE at _zvkned_zvkb diff --git a/cipher/rijndael-internal.h b/cipher/rijndael-internal.h index bb8f97a0..87b630ce 100644 --- a/cipher/rijndael-internal.h +++ b/cipher/rijndael-internal.h @@ -181,6 +181,8 @@ typedef unsigned int (*rijndael_cryptfn_t)(const struct RIJNDAEL_context_s *ctx, const unsigned char *ax); typedef void (*rijndael_prefetchfn_t)(void); typedef void (*rijndael_prepare_decfn_t)(struct RIJNDAEL_context_s *ctx); +typedef void (*rijndael_setkeyfn_t)(struct RIJNDAEL_context_s *ctx, + const byte *key); /* Our context object. */ typedef struct RIJNDAEL_context_s diff --git a/cipher/rijndael-vp-riscv-mf2.c b/cipher/rijndael-vp-riscv-mf2.c new file mode 100644 index 00000000..cf9aa577 --- /dev/null +++ b/cipher/rijndael-vp-riscv-mf2.c @@ -0,0 +1,29 @@ +/* rijndael-vp-riscv-mf2.c - RISC-V vector permutation AES, VLEN=256 variant + * Copyright (C) 2026 Jussi Kivilinna + * + * This file is part of Libgcrypt. + * + * Libgcrypt is free software; you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as + * published by the Free Software Foundation; either version 2.1 of + * the License, or (at your option) any later version. + * + * Libgcrypt is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this program; if not, see . + */ + +#include +#include "rijndael-internal.h" + +#ifdef USE_VP_RISCV + +#define VP_LMUL128 mf2 + +#include "rijndael-vp-riscv.h" + +#endif /* USE_VP_RISCV */ diff --git a/cipher/rijndael-vp-riscv-mf4.c b/cipher/rijndael-vp-riscv-mf4.c new file mode 100644 index 00000000..a0fb95ab --- /dev/null +++ b/cipher/rijndael-vp-riscv-mf4.c @@ -0,0 +1,29 @@ +/* rijndael-vp-riscv-mf4.c - RISC-V vector permutation AES, VLEN=512 variant + * Copyright (C) 2026 Jussi Kivilinna + * + * This file is part of Libgcrypt. + * + * Libgcrypt is free software; you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as + * published by the Free Software Foundation; either version 2.1 of + * the License, or (at your option) any later version. + * + * Libgcrypt is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this program; if not, see . + */ + +#include +#include "rijndael-internal.h" + +#ifdef USE_VP_RISCV + +#define VP_LMUL128 mf4 + +#include "rijndael-vp-riscv.h" + +#endif /* USE_VP_RISCV */ diff --git a/cipher/rijndael-vp-riscv-mf8.c b/cipher/rijndael-vp-riscv-mf8.c new file mode 100644 index 00000000..337e0098 --- /dev/null +++ b/cipher/rijndael-vp-riscv-mf8.c @@ -0,0 +1,29 @@ +/* rijndael-vp-riscv-mf8.c - RISC-V vector permutation AES, VLEN>=1024 variant + * Copyright (C) 2026 Jussi Kivilinna + * + * This file is part of Libgcrypt. + * + * Libgcrypt is free software; you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as + * published by the Free Software Foundation; either version 2.1 of + * the License, or (at your option) any later version. + * + * Libgcrypt is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this program; if not, see . + */ + +#include +#include "rijndael-internal.h" + +#ifdef USE_VP_RISCV + +#define VP_LMUL128 mf8 + +#include "rijndael-vp-riscv.h" + +#endif /* USE_VP_RISCV */ diff --git a/cipher/rijndael-vp-riscv.c b/cipher/rijndael-vp-riscv.c index 7a1da94f..28a5f698 100644 --- a/cipher/rijndael-vp-riscv.c +++ b/cipher/rijndael-vp-riscv.c @@ -1,5 +1,5 @@ /* RISC-V vector permutation AES for Libgcrypt - * Copyright (C) 2025 Jussi Kivilinna + * Copyright (C) 2025-2026 Jussi Kivilinna * * This file is part of Libgcrypt. * @@ -35,251 +35,45 @@ */ #include -#include -#include -#include /* for memcmp() */ - -#include "types.h" /* for byte and u32 typedefs */ -#include "g10lib.h" -#include "cipher.h" -#include "bufhelp.h" #include "rijndael-internal.h" -#include "cipher-internal.h" - #ifdef USE_VP_RISCV +#define VP_LMUL128 m1 -/********************************************************************** - AT&T x86 asm to intrinsics conversion macros (RISC-V) - **********************************************************************/ - -#include -#include "simd-common-riscv.h" - -/* - * SIMD128 - */ - -typedef vuint8m1_t __m128i; - -#define cast_m128i_to_s8(a) (__riscv_vreinterpret_v_u8m1_i8m1(a)) -#define cast_m128i_to_u32(a) (__riscv_vreinterpret_v_u8m1_u32m1(a)) -#define cast_m128i_to_u64(a) (__riscv_vreinterpret_v_u8m1_u64m1(a)) -#define cast_m128i_to_s64(a) (__riscv_vreinterpret_v_u64m1_i64m1(cast_m128i_to_u64(a))) - -#define cast_s8_to_m128i(a) (__riscv_vreinterpret_v_i8m1_u8m1(a)) -#define cast_u32_to_m128i(a) (__riscv_vreinterpret_v_u32m1_u8m1(a)) -#define cast_u64_to_m128i(a) (__riscv_vreinterpret_v_u64m1_u8m1(a)) -#define cast_s64_to_m128i(a) (cast_u64_to_m128i(__riscv_vreinterpret_v_i64m1_u64m1(a))) - -#define pand128(a, o) (o = __riscv_vand_vv_u8m1((o), (a), 16)) -#define pandn128(a, o) (o = __riscv_vand_vv_u8m1(__riscv_vnot_v_u8m1((o), 16), (a), 16)) -#define pxor128(a, o) (o = __riscv_vxor_vv_u8m1((o), (a), 16)) -#define paddb128(a, o) (o = __riscv_vadd_vv_u8m1((o), (a), 16)) -#define paddd128(a, o) (o = cast_u32_to_m128i(__riscv_vadd_vv_u32m1( \ - cast_m128i_to_u32(o), \ - cast_m128i_to_u32(a), 4))) -#define paddq128(a, o) (o = cast_u64_to_m128i(__riscv_vadd_vv_u64m1( \ - cast_m128i_to_u64(o), \ - cast_m128i_to_u64(a), 2))) - -#define psrld128(s, o) (o = cast_u32_to_m128i(__riscv_vsrl_vx_u32m1(cast_m128i_to_u32(o), (s), 4)) -#define psraq128(s, o) (o = cast_s64_to_m128i(__riscv_vsra_vx_i64m1(cast_m128i_to_s64(o), (s), 2))) -#define psrldq128(s, o) (o = __riscv_vslidedown_vx_u8m1((o), (s), 16)) -#define pslldq128(s, o) ({ vuint8m1_t __tmp = __riscv_vmv_v_x_u8m1(0, 16); \ - o = __riscv_vslideup_vx_u8m1(__tmp, (o), (s), 16); }) -#define psrl_byte_128(s, o) (o = __riscv_vsrl_vx_u8m1((o), (s), 16)) - -#define pshufb128(m8, o) (o = __riscv_vrgather_vv_u8m1((o), (m8), 16)) -#define pshufd128(m32, a, o) ({ static const __m128i_const __tmp1 = PSHUFD_MASK_TO_PSHUFB_MASK(m32); \ - __m128i __tmp2; \ - movdqa128(a, o); \ - movdqa128_memld(&__tmp1, __tmp2); \ - pshufb128(__tmp2, o); }) - -#define pshufd128_0x93(a, o) pshufd128(0x93, a, o) -#define pshufd128_0xFF(a, o) (o = cast_u32_to_m128i(__riscv_vrgather_vx_u32m1(cast_m128i_to_u32(a), 3, 4))) -#define pshufd128_0xFE(a, o) pshufd128(0xFE, a, o) -#define pshufd128_0x4E(a, o) pshufd128(0x4E, a, o) - -#define palignr128(s, a, o) (o = __riscv_vslideup_vx_u8m1(__riscv_vslidedown_vx_u8m1((a), (s), 16), (o), 16 - (s), 16)) - -#define movdqa128(a, o) (o = (a)) - -#define movdqa128_memld(a, o) (o = __riscv_vle8_v_u8m1((const void *)(a), 16)) - -#define pand128_amemld(m, o) pand128(__riscv_vle8_v_u8m1((const void *)(m), 16), (o)) -#define pxor128_amemld(m, o) pxor128(__riscv_vle8_v_u8m1((const void *)(m), 16), (o)) -#define paddq128_amemld(m, o) paddq128(__riscv_vle8_v_u8m1((const void *)(m), 16), (o)) -#define paddd128_amemld(m, o) paddd128(__riscv_vle8_v_u8m1((const void *)(m), 16), (o)) -#define pshufb128_amemld(m, o) pshufb128(__riscv_vle8_v_u8m1((const void *)(m), 16), (o)) - -/* Following operations may have unaligned memory input */ -#define movdqu128_memld(a, o) (o = __riscv_vle8_v_u8m1((const void *)(a), 16)) - -/* Following operations may have unaligned memory output */ -#define movdqu128_memst(a, o) (__riscv_vse8_v_u8m1((void *)(o), (a), 16)) - -/* - * SIMD256 - */ - -#define PSHUFD256_MASK_TO_PSHUFB256_MASK(m32) { \ - M128I_BYTE(((((m32) >> 0) & 0x03) * 4) + 0, \ - ((((m32) >> 0) & 0x03) * 4) + 1, \ - ((((m32) >> 0) & 0x03) * 4) + 2, \ - ((((m32) >> 0) & 0x03) * 4) + 3, \ - ((((m32) >> 2) & 0x03) * 4) + 0, \ - ((((m32) >> 2) & 0x03) * 4) + 1, \ - ((((m32) >> 2) & 0x03) * 4) + 2, \ - ((((m32) >> 2) & 0x03) * 4) + 3, \ - ((((m32) >> 4) & 0x03) * 4) + 0, \ - ((((m32) >> 4) & 0x03) * 4) + 1, \ - ((((m32) >> 4) & 0x03) * 4) + 2, \ - ((((m32) >> 4) & 0x03) * 4) + 3, \ - ((((m32) >> 6) & 0x03) * 4) + 0, \ - ((((m32) >> 6) & 0x03) * 4) + 1, \ - ((((m32) >> 6) & 0x03) * 4) + 2, \ - ((((m32) >> 6) & 0x03) * 4) + 3), \ - M128I_BYTE(((((m32) >> 0) & 0x03) * 4) + 1 + 16, \ - ((((m32) >> 0) & 0x03) * 4) + 1 + 16, \ - ((((m32) >> 0) & 0x03) * 4) + 2 + 16, \ - ((((m32) >> 0) & 0x03) * 4) + 3 + 16, \ - ((((m32) >> 2) & 0x03) * 4) + 0 + 16, \ - ((((m32) >> 2) & 0x03) * 4) + 1 + 16, \ - ((((m32) >> 2) & 0x03) * 4) + 2 + 16, \ - ((((m32) >> 2) & 0x03) * 4) + 3 + 16, \ - ((((m32) >> 4) & 0x03) * 4) + 0 + 16, \ - ((((m32) >> 4) & 0x03) * 4) + 1 + 16, \ - ((((m32) >> 4) & 0x03) * 4) + 2 + 16, \ - ((((m32) >> 4) & 0x03) * 4) + 3 + 16, \ - ((((m32) >> 6) & 0x03) * 4) + 0 + 16, \ - ((((m32) >> 6) & 0x03) * 4) + 1 + 16, \ - ((((m32) >> 6) & 0x03) * 4) + 2 + 16, \ - ((((m32) >> 6) & 0x03) * 4) + 3 + 16) } - -typedef vuint8m1_t __m256i; - -#define HAVE_SIMD256 1 - -#define check_simd256_support() (__riscv_vsetvl_e8m1(32) == 32) - -#define cast_m256i_to_s8(a) cast_m128i_to_s8(a) -#define cast_m256i_to_u32(a) cast_m128i_to_u32(a) -#define cast_m256i_to_u64(a) cast_m128i_to_u64(a) -#define cast_m256i_to_s64(a) cast_m128i_to_s64(a) - -#define cast_s8_to_m256i(a) (__riscv_vreinterpret_v_i8m1_u8m1(a)) -#define cast_u32_to_m256i(a) (__riscv_vreinterpret_v_u32m1_u8m1(a)) -#define cast_u64_to_m256i(a) (__riscv_vreinterpret_v_u64m1_u8m1(a)) -#define cast_s64_to_m256i(a) (cast_u64_to_m128i(__riscv_vreinterpret_v_i64m1_u64m1(a))) - -#define pand256(a, o) (o = __riscv_vand_vv_u8m1((o), (a), 32)) -#define pandn256(a, o) (o = __riscv_vand_vv_u8m1(__riscv_vnot_v_u8m1((o), 32), (a), 32)) -#define pxor256(a, o) (o = __riscv_vxor_vv_u8m1((o), (a), 32)) -#define paddb256(a, o) (o = __riscv_vadd_vv_u8m1((o), (a), 32)) -#define paddd256(a, o) (o = cast_u32_to_m256i(__riscv_vadd_vv_u32m1( \ - cast_m256i_to_u32(o), \ - cast_m256i_to_u32(a), 8))) -#define paddq256(a, o) (o = cast_u64_to_m256i(__riscv_vadd_vv_u64m1( \ - cast_m256i_to_u64(o), \ - cast_m256i_to_u64(a), 4))) - -#define psrld256(s, o) (o = cast_u32_to_m256i(__riscv_vsrl_vx_u32m1(cast_m256i_to_u32(o), (s), 8)) -#define psraq256(s, o) (o = cast_s64_to_m256i(__riscv_vsra_vx_i64m1(cast_m256i_to_s64(o), (s), 4))) -#define psrl_byte_256(s, o) (o = __riscv_vsrl_vx_u8m1((o), (s), 32)) - -/* Note: these are not PSHUFB equavalent as full 256-bit vector is used as - * 32 byte table. 256-bit PSHUFB on x86 handles 128-bit lanes separately as - * 128-bit 16 byte tables. */ - -/* tab32 variant: indexes have values 0..31. Used when 'm8' is constant and - * variable data is in 'o'. */ -#define pshufb256_tab32(m8, o) (o = __riscv_vrgather_vv_u8m1((o), (m8), 32)) - -/* tab16 variant: indexes have values 0..16 and only low 128-bit of 'o' is - * used. Used when 'o' is constant and variable data is in 'm8'. */ -#define pshufb256_tab16(m8, o) (o = __riscv_vrgather_vv_u8m1((o), (m8), 32)) - -/* Load 16 byte mask for 'pshufb256_tab32' usage as if 256-bit PSHUFB was to be - * used as on x86 (two separate 128-bit lanes). */ -#define load_tab32_mask(m, o) ({ __m128i __tmp_lo128; \ - __m128i __tmp_hi128; \ - movdqu128_memld(m, __tmp_lo128); \ - __tmp_hi128 = __riscv_vadd_vx_u8m1(__tmp_lo128, 16, 16); \ - o = __riscv_vslideup_vx_u8m1(__tmp_lo128, __tmp_hi128, 16, 32); }) - -#define broadcast128_256(a, o) (o = __riscv_vslideup_vx_u8m1((a), (a), 16, 32)) - -/* Load 16 byte table for 'pshufb256_tab16' usage. On x86 this would splat - * 128-bit table from memory to both 128-bit lanes of 256-bit register. - * On RISC-V this just loads memory to lower 128-bits. */ -#define load_tab16_table(m, o) movdqu128_memld(m, o) - -#define pshufd256(m32, a, o) ({ static const __m128i_const __tmp1 = PSHUFD_MASK_TO_PSHUFB_MASK(m32); \ - __m256i __tmp2; \ - movdqa256(a, o); \ - load_tab32_mask(&__tmp1, __tmp2); \ - pshufb256_tab32(__tmp2, o); }) - -#define pshufd256_0x93(a, o) pshufd256(0x93, a, o) - -#define insert256_hi128(x, o) (o = __riscv_vslideup_vx_u8m1((o), (x), 16, 32)) -#define extract256_hi128(y, o) (o = __riscv_vslidedown_vx_u8m1((y), 16, 32)) - -#define movdqa256(a, o) (o = (a)) - -#define movdqa128_256(a, o) (o = __riscv_vslideup_vx_u8m1((a), __riscv_vmv_v_x_u8m1(0, 16), 16, 32)) -#define movdqa256_128(a, o) (o = (a)) - -#define movdqa256_memld(a, o) (o = __riscv_vle8_v_u8m1((const void *)(a), 32)) - -#define pand256_amemld(m, o) pand128(__riscv_vle8_v_u8m1((const void *)(m), 32), (o)) -#define pxor256_amemld(m, o) pxor128(__riscv_vle8_v_u8m1((const void *)(m), 32), (o)) -#define paddq256_amemld(m, o) paddq128(__riscv_vle8_v_u8m1((const void *)(m), 32), (o)) -#define paddd256_amemld(m, o) paddd128(__riscv_vle8_v_u8m1((const void *)(m), 32), (o)) -#define pshufb256_amemld(m, o) pshufb128(__riscv_vle8_v_u8m1((const void *)(m), 32), (o)) -#define broadcast128_256_amemld(m, o) \ - broadcast128_256(__riscv_vle8_v_u8m1((const void *)(m), 32), (o)) - -/* Following operations may have unaligned memory input */ -#define movdqu256_memld(a, o) (o = __riscv_vle8_v_u8m1((const void *)(a), 32)) - -/* Following operations may have unaligned memory output */ -#define movdqu256_memst(a, o) (__riscv_vse8_v_u8m1((void *)(o), (a), 32)) - - -#ifdef HAVE_GCC_ATTRIBUTE_OPTIMIZE -# define FUNC_ATTR_OPT __attribute__((optimize("-O2"))) -#else -# define FUNC_ATTR_OPT -#endif - -#define SIMD128_OPT_ATTR FUNC_ATTR_OPT - -#define FUNC_ENCRYPT _gcry_aes_vp_riscv_encrypt -#define FUNC_DECRYPT _gcry_aes_vp_riscv_decrypt -#define FUNC_CFB_ENC _gcry_aes_vp_riscv_cfb_enc -#define FUNC_CFB_DEC _gcry_aes_vp_riscv_cfb_dec -#define FUNC_CBC_ENC _gcry_aes_vp_riscv_cbc_enc -#define FUNC_CBC_DEC _gcry_aes_vp_riscv_cbc_dec -#define FUNC_CTR_ENC _gcry_aes_vp_riscv_ctr_enc -#define FUNC_CTR32LE_ENC _gcry_aes_vp_riscv_ctr32le_enc -#define FUNC_OCB_CRYPT _gcry_aes_vp_riscv_ocb_crypt -#define FUNC_OCB_AUTH _gcry_aes_vp_riscv_ocb_auth -#define FUNC_ECB_CRYPT _gcry_aes_vp_riscv_ecb_crypt -#define FUNC_XTS_CRYPT _gcry_aes_vp_riscv_xts_crypt -#define FUNC_SETKEY _gcry_aes_vp_riscv_do_setkey -#define FUNC_PREPARE_DEC _gcry_aes_vp_riscv_prepare_decryption +#include "rijndael-vp-riscv.h" -#include "rijndael-vp-simd128.h" +extern void _gcry_aes_vp_riscv_mf2_setup (RIJNDAEL_context *ctx, + rijndael_setkeyfn_t *hw_setkey, + cipher_bulk_ops_t *bulk_ops); +extern void _gcry_aes_vp_riscv_mf4_setup (RIJNDAEL_context *ctx, + rijndael_setkeyfn_t *hw_setkey, + cipher_bulk_ops_t *bulk_ops); +extern void _gcry_aes_vp_riscv_mf8_setup (RIJNDAEL_context *ctx, + rijndael_setkeyfn_t *hw_setkey, + cipher_bulk_ops_t *bulk_ops); int -_gcry_aes_vp_riscv_setup_acceleration(RIJNDAEL_context *ctx) +_gcry_aes_vp_riscv_setup_acceleration (RIJNDAEL_context *ctx, + rijndael_setkeyfn_t *hw_setkey, + cipher_bulk_ops_t *bulk_ops) { - (void)ctx; - return (__riscv_vsetvl_e8m1(16) == 16); + size_t vlenb; + + if (__riscv_vsetvl_e8m1(16) != 16) + return 0; + + vlenb = __riscv_vsetvlmax_e8m1 (); + if (vlenb >= 128) + _gcry_aes_vp_riscv_mf8_setup (ctx, hw_setkey, bulk_ops); + else if (vlenb >= 64) + _gcry_aes_vp_riscv_mf4_setup (ctx, hw_setkey, bulk_ops); + else if (vlenb >= 32) + _gcry_aes_vp_riscv_mf2_setup (ctx, hw_setkey, bulk_ops); + else + _gcry_aes_vp_riscv_m1_setup (ctx, hw_setkey, bulk_ops); + + return 1; } #endif /* USE_VP_RISCV */ diff --git a/cipher/rijndael-vp-riscv.h b/cipher/rijndael-vp-riscv.h new file mode 100644 index 00000000..149ac70d --- /dev/null +++ b/cipher/rijndael-vp-riscv.h @@ -0,0 +1,190 @@ +/* RISC-V vector permutation AES for Libgcrypt + * Copyright (C) 2025-2026 Jussi Kivilinna + * + * This file is part of Libgcrypt. + * + * Libgcrypt is free software; you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as + * published by the Free Software Foundation; either version 2.1 of + * the License, or (at your option) any later version. + * + * Libgcrypt is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this program; if not, see . + * + * + * The code is based on the public domain library libvpaes version 0.5 + * available at http://crypto.stanford.edu/vpaes/ and which carries + * this notice: + * + * libvpaes: constant-time SSSE3 AES encryption and decryption. + * version 0.5 + * + * By Mike Hamburg, Stanford University, 2009. Public domain. + * I wrote essentially all of this code. I did not write the test + * vectors; they are the NIST known answer tests. I hereby release all + * the code and documentation here that I wrote into the public domain. + * + * This is an implementation of AES following my paper, + * "Accelerating AES with Vector Permute Instructions" + * CHES 2009; http://shiftleft.org/papers/vector_aes/ + */ + +#ifndef GCRY_RIJNDAEL_VP_RISCV_H +#define GCRY_RIJNDAEL_VP_RISCV_H + +#include +#include "types.h" +#include "g10lib.h" +#include "cipher.h" +#include "rijndael-internal.h" +#include "cipher-internal.h" + +#ifdef VP_LMUL128 + +/********************************************************************** + AT&T x86 asm to intrinsics conversion macros (RISC-V) + **********************************************************************/ + +/* VP_LMUL128 is LMUL that gives 128-bit register group. */ + +#include +#include "simd-common-riscv.h" + +#define VP_CAT_(a, b) a##b +#define VP_CAT(a, b) VP_CAT_(a, b) +#define VP_CAT3_(a, b, c) a##b##c +#define VP_CAT3(a, b, c) VP_CAT3_(a, b, c) + +#define VP_U8(fn, lmul) VP_CAT3(__riscv_, fn, VP_CAT(_u8, lmul)) + +#define VP_TO_M1_m1(a) (a) +#define VP_TO_M1_mf2(a) __riscv_vlmul_ext_v_u8mf2_u8m1(a) +#define VP_TO_M1_mf4(a) __riscv_vlmul_ext_v_u8mf4_u8m1(a) +#define VP_TO_M1_mf8(a) __riscv_vlmul_ext_v_u8mf8_u8m1(a) +#define VP_FROM_M1_m1(a) (a) +#define VP_FROM_M1_mf2(a) __riscv_vlmul_trunc_v_u8m1_u8mf2(a) +#define VP_FROM_M1_mf4(a) __riscv_vlmul_trunc_v_u8m1_u8mf4(a) +#define VP_FROM_M1_mf8(a) __riscv_vlmul_trunc_v_u8m1_u8mf8(a) +#define VP_TO_M1(lmul, a) VP_CAT(VP_TO_M1_, lmul)(a) +#define VP_FROM_M1(lmul, a) VP_CAT(VP_FROM_M1_, lmul)(a) + +/* + * SIMD128 + */ + +typedef VP_CAT3(vuint8, VP_LMUL128, _t) __m128i; + +#define V128(fn) VP_U8(fn, VP_LMUL128) + +#define cast_m128i_to_u32(a) (__riscv_vreinterpret_v_u8m1_u32m1(VP_TO_M1(VP_LMUL128, a))) +#define cast_m128i_to_u64(a) (__riscv_vreinterpret_v_u8m1_u64m1(VP_TO_M1(VP_LMUL128, a))) +#define cast_m128i_to_s64(a) (__riscv_vreinterpret_v_u64m1_i64m1(cast_m128i_to_u64(a))) + +#define cast_u32_to_m128i(a) (VP_FROM_M1(VP_LMUL128, __riscv_vreinterpret_v_u32m1_u8m1(a))) +#define cast_u64_to_m128i(a) (VP_FROM_M1(VP_LMUL128, __riscv_vreinterpret_v_u64m1_u8m1(a))) +#define cast_s64_to_m128i(a) (cast_u64_to_m128i(__riscv_vreinterpret_v_i64m1_u64m1(a))) + +#define pand128(a, o) (o = V128(vand_vv)((o), (a), 16)) +#define pandn128(a, o) (o = V128(vand_vv)(V128(vnot_v)((o), 16), (a), 16)) +#define pxor128(a, o) (o = V128(vxor_vv)((o), (a), 16)) +#define paddb128(a, o) (o = V128(vadd_vv)((o), (a), 16)) +#define paddd128(a, o) (o = cast_u32_to_m128i(__riscv_vadd_vv_u32m1( \ + cast_m128i_to_u32(o), \ + cast_m128i_to_u32(a), 4))) +#define paddq128(a, o) (o = cast_u64_to_m128i(__riscv_vadd_vv_u64m1( \ + cast_m128i_to_u64(o), \ + cast_m128i_to_u64(a), 2))) + +#define psrld128(s, o) (o = cast_u32_to_m128i(__riscv_vsrl_vx_u32m1(cast_m128i_to_u32(o), (s), 4)) +#define psraq128(s, o) (o = cast_s64_to_m128i(__riscv_vsra_vx_i64m1(cast_m128i_to_s64(o), (s), 2))) +#define psrldq128(s, o) (o = V128(vslidedown_vx)((o), (s), 16)) +#define pslldq128(s, o) ({ __m128i __tmp = V128(vmv_v_x)(0, 16); \ + o = V128(vslideup_vx)(__tmp, (o), (s), 16); }) +#define psrl_byte_128(s, o) (o = V128(vsrl_vx)((o), (s), 16)) + +#define pshufb128(m8, o) (o = V128(vrgather_vv)((o), (m8), 16)) +#define pshufd128(m32, a, o) ({ static const __m128i_const __tmp1 = PSHUFD_MASK_TO_PSHUFB_MASK(m32); \ + __m128i __tmp2; \ + movdqa128(a, o); \ + movdqa128_memld(&__tmp1, __tmp2); \ + pshufb128(__tmp2, o); }) + +#define pshufd128_0x93(a, o) pshufd128(0x93, a, o) +#define pshufd128_0xFF(a, o) pshufd128(0xFF, a, o) +#define pshufd128_0xFE(a, o) pshufd128(0xFE, a, o) +#define pshufd128_0x4E(a, o) pshufd128(0x4E, a, o) + +#define palignr128(s, a, o) (o = V128(vslideup_vx)(V128(vslidedown_vx)((a), (s), 16), (o), 16 - (s), 16)) + +#define movdqa128(a, o) (o = (a)) + +#define movdqa128_memld(a, o) (o = V128(vle8_v)((const void *)(a), 16)) + +#define pand128_amemld(m, o) pand128(V128(vle8_v)((const void *)(m), 16), (o)) +#define pxor128_amemld(m, o) pxor128(V128(vle8_v)((const void *)(m), 16), (o)) +#define paddq128_amemld(m, o) paddq128(V128(vle8_v)((const void *)(m), 16), (o)) +#define paddd128_amemld(m, o) paddd128(V128(vle8_v)((const void *)(m), 16), (o)) +#define pshufb128_amemld(m, o) pshufb128(V128(vle8_v)((const void *)(m), 16), (o)) + +/* Following operations may have unaligned memory input */ +#define movdqu128_memld(a, o) (o = V128(vle8_v)((const void *)(a), 16)) + +/* Following operations may have unaligned memory output */ +#define movdqu128_memst(a, o) (V128(vse8_v)((void *)(o), (a), 16)) + +#ifdef HAVE_GCC_ATTRIBUTE_OPTIMIZE +# define FUNC_ATTR_OPT __attribute__((optimize("-O2"))) +#else +# define FUNC_ATTR_OPT +#endif + +#define SIMD128_OPT_ATTR FUNC_ATTR_OPT + +#define VP_FUNC(name) VP_CAT3(_gcry_aes_vp_riscv_, VP_LMUL128, _##name) + +#define FUNC_ENCRYPT VP_FUNC(encrypt) +#define FUNC_DECRYPT VP_FUNC(decrypt) +#define FUNC_CFB_ENC VP_FUNC(cfb_enc) +#define FUNC_CFB_DEC VP_FUNC(cfb_dec) +#define FUNC_CBC_ENC VP_FUNC(cbc_enc) +#define FUNC_CBC_DEC VP_FUNC(cbc_dec) +#define FUNC_CTR_ENC VP_FUNC(ctr_enc) +#define FUNC_CTR32LE_ENC VP_FUNC(ctr32le_enc) +#define FUNC_OCB_CRYPT VP_FUNC(ocb_crypt) +#define FUNC_OCB_AUTH VP_FUNC(ocb_auth) +#define FUNC_ECB_CRYPT VP_FUNC(ecb_crypt) +#define FUNC_XTS_CRYPT VP_FUNC(xts_crypt) +#define FUNC_SETKEY VP_FUNC(do_setkey) +#define FUNC_PREPARE_DEC VP_FUNC(prepare_decryption) + +#include "rijndael-vp-simd128.h" + +void +VP_FUNC(setup) (RIJNDAEL_context *ctx, rijndael_setkeyfn_t *hw_setkey, + cipher_bulk_ops_t *bulk_ops) +{ + *hw_setkey = FUNC_SETKEY; + ctx->encrypt_fn = FUNC_ENCRYPT; + ctx->decrypt_fn = FUNC_DECRYPT; + ctx->prepare_decryption = FUNC_PREPARE_DEC; + + bulk_ops->cfb_enc = FUNC_CFB_ENC; + bulk_ops->cfb_dec = FUNC_CFB_DEC; + bulk_ops->cbc_enc = FUNC_CBC_ENC; + bulk_ops->cbc_dec = FUNC_CBC_DEC; + bulk_ops->ctr16be_enc = FUNC_CTR_ENC; + bulk_ops->ctr32le_enc = FUNC_CTR32LE_ENC; + bulk_ops->ocb_crypt = FUNC_OCB_CRYPT; + bulk_ops->ocb_auth = FUNC_OCB_AUTH; + bulk_ops->ecb_crypt = FUNC_ECB_CRYPT; + bulk_ops->xts_crypt = FUNC_XTS_CRYPT; +} + +#endif /* VP_LMUL128 */ + +#endif /* GCRY_RIJNDAEL_VP_RISCV_H */ diff --git a/cipher/rijndael-vp-simd128.h b/cipher/rijndael-vp-simd128.h index 84dab316..092f6ec5 100644 --- a/cipher/rijndael-vp-simd128.h +++ b/cipher/rijndael-vp-simd128.h @@ -1160,119 +1160,6 @@ aes_encrypt_core_2blks(__m128i *pxmm0_a, __m128i *pxmm0_b, *pxmm0_b = xmm0_b; } -#ifdef HAVE_SIMD256 - -static ASM_FUNC_ATTR_INLINE void -aes_encrypt_core_4blks_simd256(__m256i *pymm0_a, __m256i *pymm0_b, - struct vp_aes_config_s config, - __m128i xmm9, __m128i xmm10, __m128i xmm11, - __m128i xmm12, __m128i xmm13, __m128i xmm14, - __m128i xmm15) -{ - __m256i ymm9, ymm10, ymm11, ymm12, ymm13, ymm14, ymm15; - __m256i ymm0_a, ymm0_b; - __m256i ymm1_a, ymm2_a, ymm3_a, ymm4_a; - __m256i ymm1_b, ymm2_b, ymm3_b, ymm4_b; - __m256i ymm5, ymm6; - const byte *end_keys = config.sched_keys + 16 * config.nround; - unsigned int mc_pos = 1; - - broadcast128_256(xmm9, ymm9); - movdqa128_256(xmm10, ymm10); - movdqa128_256(xmm11, ymm11); - movdqa128_256(xmm12, ymm12); - movdqa128_256(xmm13, ymm13); - movdqa128_256(xmm14, ymm14); - movdqa128_256(xmm15, ymm15); - - ymm0_a = *pymm0_a; - ymm0_b = *pymm0_b; - - load_tab16_table(&k_iptlo, ymm2_a); movdqa256(ymm2_a, ymm2_b); - movdqa256(ymm9, ymm1_a); movdqa256(ymm9, ymm1_b); - pandn256(ymm0_a, ymm1_a); pandn256(ymm0_b, ymm1_b); - psrl_byte_256(4, ymm1_a); psrl_byte_256(4, ymm1_b); - pand256(ymm9, ymm0_a); pand256(ymm9, ymm0_b); - pshufb256_tab16(ymm0_a, ymm2_a); pshufb256_tab16(ymm0_b, ymm2_b); - load_tab16_table(&k_ipthi, ymm0_a); movdqa256(ymm0_a, ymm0_b); - - pshufb256_tab16(ymm1_a, ymm0_a); pshufb256_tab16(ymm1_b, ymm0_b); - broadcast128_256_amemld(config.sched_keys, ymm5); - pxor256(ymm5, ymm2_a); pxor256(ymm5, ymm2_b); - pxor256(ymm2_a, ymm0_a); pxor256(ymm2_b, ymm0_b); - - config.sched_keys += 16; - - while (1) - { - /* top of round */ - movdqa256(ymm9, ymm1_a); movdqa256(ymm9, ymm1_b); - pandn256(ymm0_a, ymm1_a); pandn256(ymm0_b, ymm1_b); - psrl_byte_256(4, ymm1_a); psrl_byte_256(4, ymm1_b); - pand256(ymm9, ymm0_a); pand256(ymm9, ymm0_b); - movdqa256(ymm11, ymm2_a); movdqa256(ymm11, ymm2_b); - pshufb256_tab16(ymm0_a, ymm2_a); pshufb256_tab16(ymm0_b, ymm2_b); - pxor256(ymm1_a, ymm0_a); pxor256(ymm1_b, ymm0_b); - movdqa256(ymm10, ymm3_a); movdqa256(ymm10, ymm3_b); - pshufb256_tab16(ymm1_a, ymm3_a); pshufb256_tab16(ymm1_b, ymm3_b); - pxor256(ymm2_a, ymm3_a); pxor256(ymm2_b, ymm3_b); - movdqa256(ymm10, ymm4_a); movdqa256(ymm10, ymm4_b); - pshufb256_tab16(ymm0_a, ymm4_a); pshufb256_tab16(ymm0_b, ymm4_b); - pxor256(ymm2_a, ymm4_a); pxor256(ymm2_b, ymm4_b); - movdqa256(ymm10, ymm2_a); movdqa256(ymm10, ymm2_b); - pshufb256_tab16(ymm3_a, ymm2_a); pshufb256_tab16(ymm3_b, ymm2_b); - pxor256(ymm0_a, ymm2_a); pxor256(ymm0_b, ymm2_b); - movdqa256(ymm10, ymm3_a); movdqa256(ymm10, ymm3_b); - pshufb256_tab16(ymm4_a, ymm3_a); pshufb256_tab16(ymm4_b, ymm3_b); - pxor256(ymm1_a, ymm3_a); pxor256(ymm1_b, ymm3_b); - - if (config.sched_keys == end_keys) - break; - - /* middle of middle round */ - movdqa256(ymm13, ymm4_a); movdqa256(ymm13, ymm4_b); - pshufb256_tab16(ymm2_a, ymm4_a); pshufb256_tab16(ymm2_b, ymm4_b); - broadcast128_256_amemld(config.sched_keys, ymm5); - pxor256(ymm5, ymm4_a); pxor256(ymm5, ymm4_b); - movdqa256(ymm12, ymm0_a); movdqa256(ymm12, ymm0_b); - pshufb256_tab16(ymm3_a, ymm0_a); pshufb256_tab16(ymm3_b, ymm0_b); - pxor256(ymm4_a, ymm0_a); pxor256(ymm4_b, ymm0_b); - movdqa256(ymm15, ymm4_a); movdqa256(ymm15, ymm4_b); - pshufb256_tab16(ymm2_a, ymm4_a); pshufb256_tab16(ymm2_b, ymm4_b); - load_tab32_mask(&k_mc_forward[mc_pos], ymm6); - movdqa256(ymm14, ymm2_a); movdqa256(ymm14, ymm2_b); - pshufb256_tab16(ymm3_a, ymm2_a); pshufb256_tab16(ymm3_b, ymm2_b); - pxor256(ymm4_a, ymm2_a); pxor256(ymm4_b, ymm2_b); - movdqa256(ymm0_a, ymm3_a); movdqa256(ymm0_b, ymm3_b); - pshufb256_tab32(ymm6, ymm0_a); pshufb256_tab32(ymm6, ymm0_b); - pxor256(ymm2_a, ymm0_a); pxor256(ymm2_b, ymm0_b); - load_tab32_mask(&k_mc_backward[mc_pos], ymm5); - pshufb256_tab32(ymm5, ymm3_a); pshufb256_tab32(ymm5, ymm3_b); - pxor256(ymm0_a, ymm3_a); pxor256(ymm0_b, ymm3_b); - pshufb256_tab32(ymm6, ymm0_a); pshufb256_tab32(ymm6, ymm0_b); - pxor256(ymm3_a, ymm0_a); pxor256(ymm3_b, ymm0_b); - - config.sched_keys += 16; - mc_pos = (mc_pos + 1) % 4; /* next mc mod 4 */ - } - - /* middle of last round */ - movdqa256_memld(&k_sbou, ymm4_a); movdqa256_memld(&k_sbou, ymm4_b); - pshufb256_tab16(ymm2_a, ymm4_a); pshufb256_tab16(ymm2_b, ymm4_b); - broadcast128_256_amemld(config.sched_keys, ymm5); - pxor256(ymm5, ymm4_a); pxor256(ymm5, ymm4_b); - movdqa256_memld(&k_sbot, ymm0_a); movdqa256_memld(&k_sbot, ymm0_b); - pshufb256_tab16(ymm3_a, ymm0_a); pshufb256_tab16(ymm3_b, ymm0_b); - pxor256(ymm4_a, ymm0_a); pxor256(ymm4_b, ymm0_b); - load_tab32_mask(&k_sr[mc_pos], ymm5); - pshufb256_tab32(ymm5, ymm0_a); pshufb256_tab32(ymm5, ymm0_b); - - *pymm0_a = ymm0_a; - *pymm0_b = ymm0_b; -} - -#endif /* HAVE_SIMD256 */ - static ASM_FUNC_ATTR_INLINE __m128i aes_decrypt_core(__m128i xmm0, struct vp_aes_config_s config, __m128i xmm9, __m128i xmm10, __m128i xmm11, __m128i xmm12, @@ -1480,130 +1367,6 @@ aes_decrypt_core_2blks(__m128i *pxmm0_a, __m128i *pxmm0_b, *pxmm0_b = xmm0_b; } -#ifdef HAVE_SIMD256 - -static ASM_FUNC_ATTR_INLINE void -aes_decrypt_core_4blks_simd256(__m256i *pymm0_a, __m256i *pymm0_b, - struct vp_aes_config_s config, - __m128i xmm9, __m128i xmm10, __m128i xmm11, - __m128i xmm12, __m128i xmm13, __m128i xmm14, - __m128i xmm15, __m128i xmm8) -{ - __m256i ymm9, ymm10, ymm11, ymm12, ymm13, ymm14, ymm15, ymm8; - __m256i ymm0_a, ymm0_b; - __m256i ymm1_a, ymm2_a, ymm3_a, ymm4_a; - __m256i ymm1_b, ymm2_b, ymm3_b, ymm4_b; - __m256i ymm5, ymm6; - const byte *end_keys = config.sched_keys + 16 * config.nround; - unsigned int mc_pos = config.nround % 4; - - broadcast128_256(xmm9, ymm9); - movdqa128_256(xmm10, ymm10); - movdqa128_256(xmm11, ymm11); - movdqa128_256(xmm12, ymm12); - movdqa128_256(xmm13, ymm13); - movdqa128_256(xmm14, ymm14); - movdqa128_256(xmm15, ymm15); - movdqa128_256(xmm8, ymm8); - - ymm0_a = *pymm0_a; - ymm0_b = *pymm0_b; - - load_tab16_table(&k_dipt[0], ymm2_a); movdqa256(ymm2_a, ymm2_b); - movdqa256(ymm9, ymm1_a); movdqa256(ymm9, ymm1_b); - pandn256(ymm0_a, ymm1_a); pandn256(ymm0_b, ymm1_b); - psrl_byte_256(4, ymm1_a); psrl_byte_256(4, ymm1_b); - pand256(ymm9, ymm0_a); pand256(ymm9, ymm0_b); - pshufb256_tab16(ymm0_a, ymm2_a); pshufb256_tab16(ymm0_b, ymm2_b); - load_tab16_table(&k_dipt[1], ymm0_a); movdqa256(ymm0_a, ymm0_b); - pshufb256_tab16(ymm1_a, ymm0_a); pshufb256_tab16(ymm1_b, ymm0_b); - broadcast128_256_amemld(config.sched_keys, ymm6); - pxor256(ymm6, ymm2_a); pxor256(ymm6, ymm2_b); - pxor256(ymm2_a, ymm0_a); pxor256(ymm2_b, ymm0_b); - load_tab32_mask(&k_mc_forward[3], ymm5); - - config.sched_keys += 16; - - while (1) - { - /* top of round */ - movdqa256(ymm9, ymm1_a); movdqa256(ymm9, ymm1_b); - pandn256(ymm0_a, ymm1_a); pandn256(ymm0_b, ymm1_b); - psrl_byte_256(4, ymm1_a); psrl_byte_256(4, ymm1_b); - pand256(ymm9, ymm0_a); pand256(ymm9, ymm0_b); - movdqa256(ymm11, ymm2_a); movdqa256(ymm11, ymm2_b); - pshufb256_tab16(ymm0_a, ymm2_a); pshufb256_tab16(ymm0_b, ymm2_b); - pxor256(ymm1_a, ymm0_a); pxor256(ymm1_b, ymm0_b); - movdqa256(ymm10, ymm3_a); movdqa256(ymm10, ymm3_b); - pshufb256_tab16(ymm1_a, ymm3_a); pshufb256_tab16(ymm1_b, ymm3_b); - pxor256(ymm2_a, ymm3_a); pxor256(ymm2_b, ymm3_b); - movdqa256(ymm10, ymm4_a); movdqa256(ymm10, ymm4_b); - pshufb256_tab16(ymm0_a, ymm4_a); pshufb256_tab16(ymm0_b, ymm4_b); - pxor256(ymm2_a, ymm4_a); pxor256(ymm2_b, ymm4_b); - movdqa256(ymm10, ymm2_a); movdqa256(ymm10, ymm2_b); - pshufb256_tab16(ymm3_a, ymm2_a); pshufb256_tab16(ymm3_b, ymm2_b); - pxor256(ymm0_a, ymm2_a); pxor256(ymm0_b, ymm2_b); - movdqa256(ymm10, ymm3_a); movdqa256(ymm10, ymm3_b); - pshufb256_tab16(ymm4_a, ymm3_a); pshufb256_tab16(ymm4_b, ymm3_b); - pxor256(ymm1_a, ymm3_a); pxor256(ymm1_b, ymm3_b); - - if (config.sched_keys == end_keys) - break; - - /* Inverse mix columns */ - movdqa256(ymm13, ymm4_a); movdqa256(ymm13, ymm4_b); - pshufb256_tab16(ymm2_a, ymm4_a); pshufb256_tab16(ymm2_b, ymm4_b); - broadcast128_256_amemld(config.sched_keys, ymm6); - pxor256(ymm6, ymm4_a); pxor256(ymm6, ymm4_b); - movdqa256(ymm12, ymm0_a); movdqa256(ymm12, ymm0_b); - pshufb256_tab16(ymm3_a, ymm0_a); pshufb256_tab16(ymm3_b, ymm0_b); - load_tab16_table(&k_dsbd[1], ymm1_a); movdqa256(ymm1_a, ymm1_b); - pxor256(ymm4_a, ymm0_a); pxor256(ymm4_b, ymm0_b); - - pshufb256_tab32(ymm5, ymm0_a); pshufb256_tab32(ymm5, ymm0_b); - movdqa256(ymm15, ymm4_a); movdqa256(ymm15, ymm4_b); - pshufb256_tab16(ymm2_a, ymm4_a); pshufb256_tab16(ymm2_b, ymm4_b); - pxor256(ymm0_a, ymm4_a); pxor256(ymm0_b, ymm4_b); - pshufb256_tab16(ymm3_a, ymm1_a); pshufb256_tab16(ymm3_b, ymm1_b); - pxor256(ymm4_a, ymm1_a); pxor256(ymm4_b, ymm1_b); - - pshufb256_tab32(ymm5, ymm1_a); pshufb256_tab32(ymm5, ymm1_b); - movdqa256(ymm14, ymm4_a); movdqa256(ymm14, ymm4_b); - pshufb256_tab16(ymm2_a, ymm4_a); pshufb256_tab16(ymm2_b, ymm4_b); - pxor256(ymm1_a, ymm4_a); pxor256(ymm1_b, ymm4_b); - load_tab16_table(&k_dsbb[1], ymm0_a); movdqa256(ymm0_a, ymm0_b); - pshufb256_tab16(ymm3_a, ymm0_a); pshufb256_tab16(ymm3_b, ymm0_b); - pxor256(ymm4_a, ymm0_a); pxor256(ymm4_b, ymm0_b); - - pshufb256_tab32(ymm5, ymm0_a); pshufb256_tab32(ymm5, ymm0_b); - movdqa256(ymm8, ymm4_a); movdqa256(ymm8, ymm4_b); - pshufb256_tab16(ymm2_a, ymm4_a); pshufb256_tab16(ymm2_b, ymm4_b); - pshufd256_0x93(ymm5, ymm5); - pxor256(ymm0_a, ymm4_a); pxor256(ymm0_b, ymm4_b); - load_tab16_table(&k_dsbe[1], ymm0_a); movdqa256(ymm0_a, ymm0_b); - pshufb256_tab16(ymm3_a, ymm0_a); pshufb256_tab16(ymm3_b, ymm0_b); - pxor256(ymm4_a, ymm0_a); pxor256(ymm4_b, ymm0_b); - - config.sched_keys += 16; - } - - /* middle of last round */ - load_tab16_table(&k_dsbo[0], ymm4_a); movdqa256(ymm4_a, ymm4_b); - pshufb256_tab16(ymm2_a, ymm4_a); pshufb256_tab16(ymm2_b, ymm4_b); - broadcast128_256_amemld(config.sched_keys, ymm6); - pxor256(ymm6, ymm4_a); pxor256(ymm6, ymm4_b); - load_tab16_table(&k_dsbo[1], ymm0_a); movdqa256(ymm0_a, ymm0_b); - pshufb256_tab16(ymm3_a, ymm0_a); pshufb256_tab16(ymm3_b, ymm0_b); - pxor256(ymm4_a, ymm0_a); pxor256(ymm4_b, ymm0_b); - load_tab32_mask(&k_sr[mc_pos], ymm5); - pshufb256_tab16(ymm5, ymm0_a); pshufb256_tab16(ymm5, ymm0_b); - - *pymm0_a = ymm0_a; - *pymm0_b = ymm0_b; -} - -#endif /* HAVE_SIMD256 */ - ASM_FUNC_ATTR_NOINLINE unsigned int FUNC_ENCRYPT (const RIJNDAEL_context *ctx, unsigned char *dst, const unsigned char *src) @@ -1653,10 +1416,12 @@ FUNC_DECRYPT (const RIJNDAEL_context *ctx, unsigned char *dst, } ASM_FUNC_ATTR_NOINLINE void -FUNC_CFB_ENC (RIJNDAEL_context *ctx, unsigned char *iv, - unsigned char *outbuf, const unsigned char *inbuf, - size_t nblocks) +FUNC_CFB_ENC (void *context, unsigned char *iv, void *outbuf_arg, + const void *inbuf_arg, size_t nblocks) { + RIJNDAEL_context *ctx = context; + unsigned char *outbuf = outbuf_arg; + const unsigned char *inbuf = inbuf_arg; __m128i xmm0, xmm1, xmm9, xmm10, xmm11, xmm12, xmm13, xmm14, xmm15; struct vp_aes_config_s config; @@ -1686,10 +1451,12 @@ FUNC_CFB_ENC (RIJNDAEL_context *ctx, unsigned char *iv, } ASM_FUNC_ATTR_NOINLINE void -FUNC_CBC_ENC (RIJNDAEL_context *ctx, unsigned char *iv, - unsigned char *outbuf, const unsigned char *inbuf, - size_t nblocks, int cbc_mac) +FUNC_CBC_ENC (void *context, unsigned char *iv, void *outbuf_arg, + const void *inbuf_arg, size_t nblocks, int cbc_mac) { + RIJNDAEL_context *ctx = context; + unsigned char *outbuf = outbuf_arg; + const unsigned char *inbuf = inbuf_arg; __m128i xmm0, xmm7, xmm9, xmm10, xmm11, xmm12, xmm13, xmm14, xmm15; struct vp_aes_config_s config; size_t outbuf_add = (!cbc_mac) * BLOCKSIZE; @@ -1722,10 +1489,12 @@ FUNC_CBC_ENC (RIJNDAEL_context *ctx, unsigned char *iv, } ASM_FUNC_ATTR_NOINLINE void -FUNC_CTR_ENC (RIJNDAEL_context *ctx, unsigned char *ctr, - unsigned char *outbuf, const unsigned char *inbuf, - size_t nblocks) +FUNC_CTR_ENC (void *context, unsigned char *ctr, void *outbuf_arg, + const void *inbuf_arg, size_t nblocks) { + RIJNDAEL_context *ctx = context; + unsigned char *outbuf = outbuf_arg; + const unsigned char *inbuf = inbuf_arg; __m128i xmm0, xmm1, xmm2, xmm3, xmm6, xmm7, xmm8; __m128i xmm9, xmm10, xmm11, xmm12, xmm13, xmm14, xmm15; static const __m128i_const be_mask = @@ -1745,62 +1514,6 @@ FUNC_CTR_ENC (RIJNDAEL_context *ctx, unsigned char *ctr, movdqu128_memld(ctr, xmm7); /* Preload CTR */ movdqa128_memld(&be_mask, xmm6); /* Preload mask */ -#ifdef HAVE_SIMD256 - if (check_simd256_support()) - { - __m256i ymm0, ymm1, ymm2, ymm3; - - for (; nblocks >= 4; nblocks -= 4) - { - movdqa128_256(xmm7, ymm0); - - /* detect if 8-bit carry handling is needed */ - if (UNLIKELY(((ctrlow += 4) & 0xff) <= 3)) - { - pshufb128(xmm6, xmm7); - paddd128_amemld(&littleendian_add, xmm7); - movdqa128(xmm7, xmm2); - pshufb128(xmm6, xmm2); - insert256_hi128(xmm2, ymm0); - paddd128_amemld(&littleendian_add, xmm7); - movdqa128(xmm7, xmm2); - pshufb128(xmm6, xmm2); - movdqa128_256(xmm2, ymm1); - paddd128_amemld(&littleendian_add, xmm7); - movdqa128(xmm7, xmm2); - pshufb128(xmm6, xmm2); - insert256_hi128(xmm2, ymm1); - paddd128_amemld(&littleendian_add, xmm7); - pshufb128(xmm6, xmm7); - } - else - { - paddb128(xmm8, xmm7); - insert256_hi128(xmm7, ymm0); - paddb128(xmm8, xmm7); - movdqa128_256(xmm7, ymm1); - paddb128(xmm8, xmm7); - insert256_hi128(xmm7, ymm1); - paddb128(xmm8, xmm7); - } - - aes_encrypt_core_4blks_simd256(&ymm0, &ymm1, config, - xmm9, xmm10, xmm11, xmm12, xmm13, xmm14, - xmm15); - - movdqu256_memld(inbuf + 0 * BLOCKSIZE, ymm2); - movdqu256_memld(inbuf + 2 * BLOCKSIZE, ymm3); - pxor256(ymm2, ymm0); - pxor256(ymm3, ymm1); - movdqu256_memst(ymm0, outbuf + 0 * BLOCKSIZE); - movdqu256_memst(ymm1, outbuf + 2 * BLOCKSIZE); - - outbuf += 4 * BLOCKSIZE; - inbuf += 4 * BLOCKSIZE; - } - } -#endif /* HAVE_SIMD256 */ - for (; nblocks >= 2; nblocks -= 2) { movdqa128(xmm7, xmm0); @@ -1869,10 +1582,12 @@ FUNC_CTR_ENC (RIJNDAEL_context *ctx, unsigned char *ctr, } ASM_FUNC_ATTR_NOINLINE void -FUNC_CTR32LE_ENC (RIJNDAEL_context *ctx, unsigned char *ctr, - unsigned char *outbuf, const unsigned char *inbuf, - size_t nblocks) +FUNC_CTR32LE_ENC (void *context, unsigned char *ctr, void *outbuf_arg, + const void *inbuf_arg, size_t nblocks) { + RIJNDAEL_context *ctx = context; + unsigned char *outbuf = outbuf_arg; + const unsigned char *inbuf = inbuf_arg; __m128i xmm0, xmm1, xmm2, xmm3, xmm7, xmm8; __m128i xmm9, xmm10, xmm11, xmm12, xmm13, xmm14, xmm15; static const __m128i_const add_one = M128I_U64(1, 0); @@ -1887,44 +1602,6 @@ FUNC_CTR32LE_ENC (RIJNDAEL_context *ctx, unsigned char *ctr, movdqa128_memld(&add_one, xmm8); /* Preload byte add */ movdqu128_memld(ctr, xmm7); /* Preload CTR */ -#ifdef HAVE_SIMD256 - if (check_simd256_support()) - { - __m256i ymm0, ymm1, ymm2, ymm3; - - for (; nblocks >= 4; nblocks -= 4) - { - movdqa128(xmm7, xmm0); - movdqa128(xmm7, xmm1); - paddd128(xmm8, xmm1); - paddd128_amemld(&add_two, xmm7); - movdqa128_256(xmm0, ymm0); - insert256_hi128(xmm1, ymm0); - - movdqa128(xmm7, xmm1); - movdqa128(xmm7, xmm2); - paddd128(xmm8, xmm2); - paddd128_amemld(&add_two, xmm7); - movdqa128_256(xmm1, ymm1); - insert256_hi128(xmm2, ymm1); - - aes_encrypt_core_4blks_simd256(&ymm0, &ymm1, config, - xmm9, xmm10, xmm11, xmm12, xmm13, - xmm14, xmm15); - - movdqu256_memld(inbuf + 0 * BLOCKSIZE, ymm2); - movdqu256_memld(inbuf + 2 * BLOCKSIZE, ymm3); - pxor256(ymm2, ymm0); - pxor256(ymm3, ymm1); - movdqu256_memst(ymm0, outbuf + 0 * BLOCKSIZE); - movdqu256_memst(ymm1, outbuf + 2 * BLOCKSIZE); - - outbuf += 4 * BLOCKSIZE; - inbuf += 4 * BLOCKSIZE; - } - } -#endif /* HAVE_SIMD256 */ - for (; nblocks >= 2; nblocks -= 2) { movdqa128(xmm7, xmm0); @@ -1968,10 +1645,12 @@ FUNC_CTR32LE_ENC (RIJNDAEL_context *ctx, unsigned char *ctr, } ASM_FUNC_ATTR_NOINLINE void -FUNC_CFB_DEC (RIJNDAEL_context *ctx, unsigned char *iv, - unsigned char *outbuf, const unsigned char *inbuf, - size_t nblocks) +FUNC_CFB_DEC (void *context, unsigned char *iv, void *outbuf_arg, + const void *inbuf_arg, size_t nblocks) { + RIJNDAEL_context *ctx = context; + unsigned char *outbuf = outbuf_arg; + const unsigned char *inbuf = inbuf_arg; __m128i xmm0, xmm1, xmm2, xmm6, xmm9; __m128i xmm10, xmm11, xmm12, xmm13, xmm14, xmm15; struct vp_aes_config_s config; @@ -1983,36 +1662,6 @@ FUNC_CFB_DEC (RIJNDAEL_context *ctx, unsigned char *iv, movdqu128_memld(iv, xmm0); -#ifdef HAVE_SIMD256 - if (check_simd256_support()) - { - __m256i ymm6, ymm1, ymm2, ymm3; - - for (; nblocks >= 4; nblocks -= 4) - { - movdqa128_256(xmm0, ymm6); - movdqu256_memld(inbuf + 0 * BLOCKSIZE, ymm2); - movdqa256_128(ymm2, xmm2); - insert256_hi128(xmm2, ymm6); - movdqu256_memld(inbuf + 1 * BLOCKSIZE, ymm1); - movdqu256_memld(inbuf + 2 * BLOCKSIZE, ymm3); - extract256_hi128(ymm3, xmm0); - - aes_encrypt_core_4blks_simd256(&ymm6, &ymm1, config, - xmm9, xmm10, xmm11, xmm12, xmm13, - xmm14, xmm15); - - pxor256(ymm2, ymm6); - pxor256(ymm3, ymm1); - movdqu256_memst(ymm6, outbuf + 0 * BLOCKSIZE); - movdqu256_memst(ymm1, outbuf + 2 * BLOCKSIZE); - - outbuf += 4 * BLOCKSIZE; - inbuf += 4 * BLOCKSIZE; - } - } -#endif /* HAVE_SIMD256 */ - for (; nblocks >= 2; nblocks -= 2) { movdqa128(xmm0, xmm1); @@ -2052,10 +1701,12 @@ FUNC_CFB_DEC (RIJNDAEL_context *ctx, unsigned char *iv, } ASM_FUNC_ATTR_NOINLINE void -FUNC_CBC_DEC (RIJNDAEL_context *ctx, unsigned char *iv, - unsigned char *outbuf, const unsigned char *inbuf, - size_t nblocks) +FUNC_CBC_DEC (void *context, unsigned char *iv, void *outbuf_arg, + const void *inbuf_arg, size_t nblocks) { + RIJNDAEL_context *ctx = context; + unsigned char *outbuf = outbuf_arg; + const unsigned char *inbuf = inbuf_arg; __m128i xmm0, xmm1, xmm5, xmm6, xmm7; __m128i xmm9, xmm10, xmm11, xmm12, xmm13, xmm14, xmm15, xmm8; struct vp_aes_config_s config; @@ -2073,36 +1724,6 @@ FUNC_CBC_DEC (RIJNDAEL_context *ctx, unsigned char *iv, movdqu128_memld(iv, xmm7); -#ifdef HAVE_SIMD256 - if (check_simd256_support()) - { - __m256i ymm0, ymm1, ymm2, ymm3; - - for (; nblocks >= 4; nblocks -= 4) - { - movdqu256_memld(inbuf + 0 * BLOCKSIZE, ymm0); - movdqu256_memld(inbuf + 2 * BLOCKSIZE, ymm1); - movdqa256_128(ymm0, xmm0); - movdqa128_256(xmm7, ymm2); - insert256_hi128(xmm0, ymm2); - movdqu256_memld(inbuf + 1 * BLOCKSIZE, ymm3); - extract256_hi128(ymm1, xmm7); - - aes_decrypt_core_4blks_simd256(&ymm0, &ymm1, config, - xmm9, xmm10, xmm11, xmm12, xmm13, - xmm14, xmm15, xmm8); - - pxor256(ymm2, ymm0); - pxor256(ymm3, ymm1); - movdqu256_memst(ymm0, outbuf + 0 * BLOCKSIZE); - movdqu256_memst(ymm1, outbuf + 2 * BLOCKSIZE); - - outbuf += 4 * BLOCKSIZE; - inbuf += 4 * BLOCKSIZE; - } - } -#endif /* HAVE_SIMD256 */ - for (; nblocks >= 2; nblocks -= 2) { movdqu128_memld(inbuf, xmm0); @@ -2167,68 +1788,6 @@ aes_simd128_ocb_enc (gcry_cipher_hd_t c, void *outbuf_arg, movdqu128_memld(c->u_iv.iv, xmm7); movdqu128_memld(c->u_ctr.ctr, xmm6); -#ifdef HAVE_SIMD256 - if (check_simd256_support() && nblocks >= 4) - { - __m256i ymm0, ymm1, ymm3, ymm6, ymm8; - - movdqa128_256(xmm6, ymm6); - - for (; nblocks >= 4; nblocks -= 4) - { - const unsigned char *l; - - movdqu256_memld(inbuf + 0 * BLOCKSIZE, ymm0); - movdqu256_memld(inbuf + 2 * BLOCKSIZE, ymm1); - - /* Offset_i = Offset_{i-1} xor L_{ntz(i)} */ - /* C_i = Offset_i xor ENCIPHER(K, P_i xor Offset_i) */ - l = ocb_get_l(c, ++n); - movdqu128_memld(l, xmm2); - pxor128(xmm2, xmm7); - movdqa128_256(xmm7, ymm3); - - l = ocb_get_l(c, ++n); - movdqu128_memld(l, xmm2); - pxor128(xmm2, xmm7); - insert256_hi128(xmm7, ymm3); - - l = ocb_get_l(c, ++n); - movdqu128_memld(l, xmm2); - pxor128(xmm2, xmm7); - movdqa128_256(xmm7, ymm8); - - l = ocb_get_l(c, ++n); - movdqu128_memld(l, xmm2); - pxor128(xmm2, xmm7); - insert256_hi128(xmm7, ymm8); - - /* Checksum_i = Checksum_{i-1} xor P_i */ - pxor256(ymm0, ymm6); - pxor256(ymm1, ymm6); - - pxor256(ymm3, ymm0); - pxor256(ymm8, ymm1); - - aes_encrypt_core_4blks_simd256(&ymm0, &ymm1, config, - xmm9, xmm10, xmm11, xmm12, xmm13, - xmm14, xmm15); - - pxor256(ymm3, ymm0); - pxor256(ymm8, ymm1); - movdqu256_memst(ymm0, outbuf + 0 * BLOCKSIZE); - movdqu256_memst(ymm1, outbuf + 2 * BLOCKSIZE); - - inbuf += 4 * BLOCKSIZE; - outbuf += 4 * BLOCKSIZE; - } - - extract256_hi128(ymm6, xmm0); - movdqa256_128(ymm6, xmm6); - pxor128(xmm0, xmm6); - } -#endif /* HAVE_SIMD256 */ - for (; nblocks >= 2; nblocks -= 2) { const unsigned char *l; @@ -2328,69 +1887,6 @@ aes_simd128_ocb_dec (gcry_cipher_hd_t c, void *outbuf_arg, movdqu128_memld(c->u_iv.iv, xmm7); movdqu128_memld(c->u_ctr.ctr, xmm6); -#ifdef HAVE_SIMD256 - if (check_simd256_support() && nblocks >= 4) - { - __m256i ymm0, ymm1, ymm3, ymm6, ymm8; - - movdqa128_256(xmm6, ymm6); - - for (; nblocks >= 4; nblocks -= 4) - { - const unsigned char *l; - - movdqu256_memld(inbuf + 0 * BLOCKSIZE, ymm0); - movdqu256_memld(inbuf + 2 * BLOCKSIZE, ymm1); - - /* Offset_i = Offset_{i-1} xor L_{ntz(i)} */ - /* C_i = Offset_i xor ENCIPHER(K, P_i xor Offset_i) */ - l = ocb_get_l(c, ++n); - movdqu128_memld(l, xmm2); - pxor128(xmm2, xmm7); - movdqa128_256(xmm7, ymm3); - - l = ocb_get_l(c, ++n); - movdqu128_memld(l, xmm2); - pxor128(xmm2, xmm7); - insert256_hi128(xmm7, ymm3); - - l = ocb_get_l(c, ++n); - movdqu128_memld(l, xmm2); - pxor128(xmm2, xmm7); - movdqa128_256(xmm7, ymm8); - - l = ocb_get_l(c, ++n); - movdqu128_memld(l, xmm2); - pxor128(xmm2, xmm7); - insert256_hi128(xmm7, ymm8); - - pxor256(ymm3, ymm0); - pxor256(ymm8, ymm1); - - aes_decrypt_core_4blks_simd256(&ymm0, &ymm1, config, - xmm9, xmm10, xmm11, xmm12, xmm13, - xmm14, xmm15, xmm8); - - pxor256(ymm3, ymm0); - pxor256(ymm8, ymm1); - - /* Checksum_i = Checksum_{i-1} xor P_i */ - pxor256(ymm0, ymm6); - pxor256(ymm1, ymm6); - - movdqu256_memst(ymm0, outbuf + 0 * BLOCKSIZE); - movdqu256_memst(ymm1, outbuf + 2 * BLOCKSIZE); - - inbuf += 4 * BLOCKSIZE; - outbuf += 4 * BLOCKSIZE; - } - - extract256_hi128(ymm6, xmm0); - movdqa256_128(ymm6, xmm6); - pxor128(xmm0, xmm6); - } -#endif /* HAVE_SIMD256 */ - for (; nblocks >= 2; nblocks -= 2) { const unsigned char *l; @@ -2493,61 +1989,6 @@ FUNC_OCB_AUTH(gcry_cipher_hd_t c, const void *abuf_arg, size_t nblocks) movdqu128_memld(c->u_mode.ocb.aad_offset, xmm7); movdqu128_memld(c->u_mode.ocb.aad_sum, xmm6); -#ifdef HAVE_SIMD256 - if (check_simd256_support() && nblocks >= 4) - { - __m256i ymm0, ymm1, ymm3, ymm6, ymm8; - - movdqa128_256(xmm6, ymm6); - - for (; nblocks >= 4; nblocks -= 4) - { - const unsigned char *l; - - movdqu256_memld(abuf + 0 * BLOCKSIZE, ymm0); - movdqu256_memld(abuf + 2 * BLOCKSIZE, ymm1); - - /* Offset_i = Offset_{i-1} xor L_{ntz(i)} */ - /* Sum_i = Sum_{i-1} xor ENCIPHER(K, A_i xor Offset_i) */ - l = ocb_get_l(c, ++n); - movdqu128_memld(l, xmm2); - pxor128(xmm2, xmm7); - movdqa128_256(xmm7, ymm3); - - l = ocb_get_l(c, ++n); - movdqu128_memld(l, xmm2); - pxor128(xmm2, xmm7); - insert256_hi128(xmm7, ymm3); - - l = ocb_get_l(c, ++n); - movdqu128_memld(l, xmm2); - pxor128(xmm2, xmm7); - movdqa128_256(xmm7, ymm8); - - l = ocb_get_l(c, ++n); - movdqu128_memld(l, xmm2); - pxor128(xmm2, xmm7); - insert256_hi128(xmm7, ymm8); - - pxor256(ymm3, ymm0); - pxor256(ymm8, ymm1); - - aes_encrypt_core_4blks_simd256(&ymm0, &ymm1, config, - xmm9, xmm10, xmm11, xmm12, xmm13, - xmm14, xmm15); - - pxor256(ymm0, ymm6); - pxor256(ymm1, ymm6); - - abuf += 4 * BLOCKSIZE; - } - - extract256_hi128(ymm6, xmm0); - movdqa256_128(ymm6, xmm6); - pxor128(xmm0, xmm6); - } -#endif /* HAVE_SIMD256 */ - for (; nblocks >= 2; nblocks -= 2) { const unsigned char *l; @@ -2606,7 +2047,7 @@ FUNC_OCB_AUTH(gcry_cipher_hd_t c, const void *abuf_arg, size_t nblocks) return 0; } -ASM_FUNC_ATTR_NOINLINE void +static ASM_FUNC_ATTR_NOINLINE void aes_simd128_ecb_enc (void *context, void *outbuf_arg, const void *inbuf_arg, size_t nblocks) { @@ -2621,29 +2062,6 @@ aes_simd128_ecb_enc (void *context, void *outbuf_arg, const void *inbuf_arg, enc_preload(xmm9, xmm10, xmm11, xmm12, xmm13, xmm14, xmm15); -#ifdef HAVE_SIMD256 - if (check_simd256_support()) - { - __m256i ymm0, ymm1; - - for (; nblocks >= 4; nblocks -= 4) - { - movdqu256_memld(inbuf + 0 * BLOCKSIZE, ymm0); - movdqu256_memld(inbuf + 2 * BLOCKSIZE, ymm1); - - aes_encrypt_core_4blks_simd256(&ymm0, &ymm1, config, - xmm9, xmm10, xmm11, xmm12, xmm13, - xmm14, xmm15); - - movdqu256_memst(ymm0, outbuf + 0 * BLOCKSIZE); - movdqu256_memst(ymm1, outbuf + 2 * BLOCKSIZE); - - inbuf += 4 * BLOCKSIZE; - outbuf += 4 * BLOCKSIZE; - } - } -#endif /* HAVE_SIMD256 */ - for (; nblocks >= 2; nblocks -= 2) { movdqu128_memld(inbuf + 0 * BLOCKSIZE, xmm0); @@ -2677,7 +2095,7 @@ aes_simd128_ecb_enc (void *context, void *outbuf_arg, const void *inbuf_arg, clear_vec_regs(); } -ASM_FUNC_ATTR_NOINLINE void +static ASM_FUNC_ATTR_NOINLINE void aes_simd128_ecb_dec (void *context, void *outbuf_arg, const void *inbuf_arg, size_t nblocks) { @@ -2698,29 +2116,6 @@ aes_simd128_ecb_dec (void *context, void *outbuf_arg, const void *inbuf_arg, dec_preload(xmm9, xmm10, xmm11, xmm12, xmm13, xmm14, xmm15, xmm8); -#ifdef HAVE_SIMD256 - if (check_simd256_support()) - { - __m256i ymm0, ymm1; - - for (; nblocks >= 4; nblocks -= 4) - { - movdqu256_memld(inbuf + 0 * BLOCKSIZE, ymm0); - movdqu256_memld(inbuf + 2 * BLOCKSIZE, ymm1); - - aes_decrypt_core_4blks_simd256(&ymm0, &ymm1, config, - xmm9, xmm10, xmm11, xmm12, xmm13, - xmm14, xmm15, xmm8); - - movdqu256_memst(ymm0, outbuf + 0 * BLOCKSIZE); - movdqu256_memst(ymm1, outbuf + 2 * BLOCKSIZE); - - inbuf += 4 * BLOCKSIZE; - outbuf += 4 * BLOCKSIZE; - } - } -#endif /* HAVE_SIMD256 */ - for (; nblocks >= 2; nblocks -= 2) { movdqu128_memld(inbuf + 0 * BLOCKSIZE, xmm0); @@ -2796,43 +2191,6 @@ aes_simd128_xts_enc (void *context, unsigned char *tweak, void *outbuf_arg, movdqu128_memld(tweak, xmm7); /* Preload tweak */ -#ifdef HAVE_SIMD256 - if (check_simd256_support()) - { - __m256i ymm0, ymm1, ymm2, ymm3; - - for (; nblocks >= 4; nblocks -= 4) - { - movdqu256_memld(inbuf + 0 * BLOCKSIZE, ymm0); - movdqu256_memld(inbuf + 2 * BLOCKSIZE, ymm1); - - movdqa128_256(xmm7, ymm2); - xmm7 = xts_gfmul_byA(xmm7); - insert256_hi128(xmm7, ymm2); - xmm7 = xts_gfmul_byA(xmm7); - movdqa128_256(xmm7, ymm3); - xmm7 = xts_gfmul_byA(xmm7); - insert256_hi128(xmm7, ymm3); - xmm7 = xts_gfmul_byA(xmm7); - - pxor256(ymm2, ymm0); - pxor256(ymm3, ymm1); - - aes_encrypt_core_4blks_simd256(&ymm0, &ymm1, config, - xmm9, xmm10, xmm11, xmm12, xmm13, - xmm14, xmm15); - - pxor256(ymm2, ymm0); - pxor256(ymm3, ymm1); - movdqu256_memst(ymm0, outbuf + 0 * BLOCKSIZE); - movdqu256_memst(ymm1, outbuf + 2 * BLOCKSIZE); - - outbuf += 4 * BLOCKSIZE; - inbuf += 4 * BLOCKSIZE; - } - } -#endif /* HAVE_SIMD256 */ - for (; nblocks >= 2; nblocks -= 2) { movdqu128_memld(inbuf, xmm0); @@ -2902,43 +2260,6 @@ aes_simd128_xts_dec (void *context, unsigned char *tweak, void *outbuf_arg, movdqu128_memld(tweak, xmm7); /* Preload tweak */ -#ifdef HAVE_SIMD256 - if (check_simd256_support()) - { - __m256i ymm0, ymm1, ymm2, ymm3; - - for (; nblocks >= 4; nblocks -= 4) - { - movdqu256_memld(inbuf + 0 * BLOCKSIZE, ymm0); - movdqu256_memld(inbuf + 2 * BLOCKSIZE, ymm1); - - movdqa128_256(xmm7, ymm2); - xmm7 = xts_gfmul_byA(xmm7); - insert256_hi128(xmm7, ymm2); - xmm7 = xts_gfmul_byA(xmm7); - movdqa128_256(xmm7, ymm3); - xmm7 = xts_gfmul_byA(xmm7); - insert256_hi128(xmm7, ymm3); - xmm7 = xts_gfmul_byA(xmm7); - - pxor256(ymm2, ymm0); - pxor256(ymm3, ymm1); - - aes_decrypt_core_4blks_simd256(&ymm0, &ymm1, config, - xmm9, xmm10, xmm11, xmm12, xmm13, - xmm14, xmm15, xmm8); - - pxor256(ymm2, ymm0); - pxor256(ymm3, ymm1); - movdqu256_memst(ymm0, outbuf + 0 * BLOCKSIZE); - movdqu256_memst(ymm1, outbuf + 2 * BLOCKSIZE); - - outbuf += 4 * BLOCKSIZE; - inbuf += 4 * BLOCKSIZE; - } - } -#endif /* HAVE_SIMD256 */ - for (; nblocks >= 2; nblocks -= 2) { movdqu128_memld(inbuf, xmm0); diff --git a/cipher/rijndael.c b/cipher/rijndael.c index 082b20ab..0586bfd2 100644 --- a/cipher/rijndael.c +++ b/cipher/rijndael.c @@ -227,58 +227,9 @@ extern void _gcry_aes_vp_aarch64_xts_crypt (void *context, unsigned char *tweak, #ifdef USE_VP_RISCV /* RISC-V vector permutation implementation of AES */ -extern int _gcry_aes_vp_riscv_setup_acceleration(RIJNDAEL_context *ctx); - -extern void _gcry_aes_vp_riscv_do_setkey(RIJNDAEL_context *ctx, - const byte *key); -extern void _gcry_aes_vp_riscv_prepare_decryption(RIJNDAEL_context *ctx); - -extern unsigned int _gcry_aes_vp_riscv_encrypt (const RIJNDAEL_context *ctx, - unsigned char *dst, - const unsigned char *src); -extern unsigned int _gcry_aes_vp_riscv_decrypt (const RIJNDAEL_context *ctx, - unsigned char *dst, - const unsigned char *src); -extern void _gcry_aes_vp_riscv_cfb_enc (void *context, unsigned char *iv, - void *outbuf_arg, - const void *inbuf_arg, - size_t nblocks); -extern void _gcry_aes_vp_riscv_cbc_enc (void *context, unsigned char *iv, - void *outbuf_arg, - const void *inbuf_arg, - size_t nblocks, - int cbc_mac); -extern void _gcry_aes_vp_riscv_ctr_enc (void *context, unsigned char *ctr, - void *outbuf_arg, - const void *inbuf_arg, - size_t nblocks); -extern void _gcry_aes_vp_riscv_ctr32le_enc (void *context, unsigned char *ctr, - void *outbuf_arg, - const void *inbuf_arg, - size_t nblocks); -extern void _gcry_aes_vp_riscv_cfb_dec (void *context, unsigned char *iv, - void *outbuf_arg, - const void *inbuf_arg, - size_t nblocks); -extern void _gcry_aes_vp_riscv_cbc_dec (void *context, unsigned char *iv, - void *outbuf_arg, - const void *inbuf_arg, - size_t nblocks); -extern size_t _gcry_aes_vp_riscv_ocb_crypt (gcry_cipher_hd_t c, - void *outbuf_arg, - const void *inbuf_arg, - size_t nblocks, - int encrypt); -extern size_t _gcry_aes_vp_riscv_ocb_auth (gcry_cipher_hd_t c, - const void *abuf_arg, - size_t nblocks); -extern void _gcry_aes_vp_riscv_ecb_crypt (void *context, void *outbuf_arg, - const void *inbuf_arg, - size_t nblocks, int encrypt); -extern void _gcry_aes_vp_riscv_xts_crypt (void *context, unsigned char *tweak, - void *outbuf_arg, - const void *inbuf_arg, - size_t nblocks, int encrypt); +extern int _gcry_aes_vp_riscv_setup_acceleration(RIJNDAEL_context *ctx, + rijndael_setkeyfn_t *hw_setkey, + cipher_bulk_ops_t *bulk_ops); #endif #ifdef USE_RISCV_V_CRYPTO @@ -871,26 +822,10 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen, else if ((hwfeatures & HWF_RISCV_IMAFDC) && (hwfeatures & HWF_RISCV_B) /* Mandatory in RVA22U64 */ && (hwfeatures & HWF_RISCV_V) /* Optional in RVA22U64 */ - && _gcry_aes_vp_riscv_setup_acceleration(ctx)) + && _gcry_aes_vp_riscv_setup_acceleration(ctx, &hw_setkey, bulk_ops)) { - hw_setkey = _gcry_aes_vp_riscv_do_setkey; - ctx->encrypt_fn = _gcry_aes_vp_riscv_encrypt; - ctx->decrypt_fn = _gcry_aes_vp_riscv_decrypt; ctx->prefetch_enc_fn = NULL; ctx->prefetch_dec_fn = NULL; - ctx->prepare_decryption = _gcry_aes_vp_riscv_prepare_decryption; - - /* Setup vector permute RISC-V bulk encryption routines. */ - bulk_ops->cfb_enc = _gcry_aes_vp_riscv_cfb_enc; - bulk_ops->cfb_dec = _gcry_aes_vp_riscv_cfb_dec; - bulk_ops->cbc_enc = _gcry_aes_vp_riscv_cbc_enc; - bulk_ops->cbc_dec = _gcry_aes_vp_riscv_cbc_dec; - bulk_ops->ctr16be_enc = _gcry_aes_vp_riscv_ctr_enc; - bulk_ops->ctr32le_enc = _gcry_aes_vp_riscv_ctr32le_enc; - bulk_ops->ocb_crypt = _gcry_aes_vp_riscv_ocb_crypt; - bulk_ops->ocb_auth = _gcry_aes_vp_riscv_ocb_auth; - bulk_ops->ecb_crypt = _gcry_aes_vp_riscv_ecb_crypt; - bulk_ops->xts_crypt = _gcry_aes_vp_riscv_xts_crypt; } #endif #ifdef USE_PPC_CRYPTO_WITH_PPC9LE diff --git a/configure.ac b/configure.ac index b10af44c..4299b433 100644 --- a/configure.ac +++ b/configure.ac @@ -3787,6 +3787,9 @@ if test "$found" = "1" ; then riscv64-*-*) # Build with the vector permute SIMD128 implementation GCRYPT_ASM_CIPHERS="$GCRYPT_ASM_CIPHERS rijndael-vp-riscv.lo" + GCRYPT_ASM_CIPHERS="$GCRYPT_ASM_CIPHERS rijndael-vp-riscv-mf2.lo" + GCRYPT_ASM_CIPHERS="$GCRYPT_ASM_CIPHERS rijndael-vp-riscv-mf4.lo" + GCRYPT_ASM_CIPHERS="$GCRYPT_ASM_CIPHERS rijndael-vp-riscv-mf8.lo" # Build with the RISC-V vector cryptography implementation GCRYPT_ASM_CIPHERS="$GCRYPT_ASM_CIPHERS rijndael-riscv-zvkned.lo" -- 2.53.0