From guidovranken at gmail.com  Thu Sep  3 14:19:22 2020
From: guidovranken at gmail.com (Guido Vranken)
Date: Thu, 3 Sep 2020 14:19:22 +0200
Subject: gcry_mpi_invm succeeds if the inverse does not exist
In-Reply-To: <87d06jmdzh.fsf@iwagami.gniibe.org>
References: <CAO5O-ELb_27vpXnAXeeJZxnovftEqe_aFe0JsbUs7KMhtEuD=Q@mail.gmail.com>
 <87zhbeh921.fsf@iwagami.gniibe.org>
 <CAO5O-EJipQu2VBbtTiO_bt=7mJ35aT_AFd-OJz4F9swR7SfWbg@mail.gmail.com>
 <51fc5b09-7b2f-ce2e-bb8c-f653ba907446@iki.fi>
 <CAO5O-EJwyViAdfUY-hnZ0hgBYWkLR7DK9M2-RGvbyT-t-BEsPg@mail.gmail.com>
 <87tv0k9vwj.fsf@jumper.gniibe.org>
 <6f38bc98-85fe-12b1-9cce-8e96e699e378@iki.fi>
 <87d06jmdzh.fsf@iwagami.gniibe.org>
Message-ID: <CAO5O-EJxRqvY=Y2ZJbe7e9261QB34qqw2-AQiaZPabCsUwm+zw@mail.gmail.com>

The following inputs to gcry_mpi_invm():

36fb5bdb5daa9864113ad8a49a41722fc7003a40b02a13daca6997859c2d8534192ff6c02447
25c88352cfa171fc728503df037c355a6d5588b22e3510b08f10848ad7c0980b400

produces the number:

66CAF1A9A03478A288760C2E05E237F11432BA70BECEE56D942ACCD337470E5D77

But this is incorrect (another library reports the modular inverse does not
exist).

----------

The following inputs to gcry_mpi_invm():

12cf3a8ca3d97bea2f080362600cee355
1c3fddf62aee0be2f6dc2ef8471f1be2e

produces the number:

60A6520F494E6EE6EE436283FB34B945

but it should produce:

1339462644931fd624528ea6b3fb1f985

On Mon, Jun 1, 2020 at 9:39 AM NIIBE Yutaka <gniibe at fsij.org> wrote:

> Jussi Kivilinna <jussi.kivilinna at iki.fi> wrote:
> > Cryptofuzz is reporting another heap-buffer-overflow issue in
> > _gcry_mpi_invm. I've attached reproducer, original from Guido and
> > as patch applied to tests/basic.c.
>
> My fix of 69b55f87053ce2494cd4b38dc600f867bc4355be was not enough.
> I just push another change:
>
>         6f8b1d4cb798375e6d830fd6b73c71da93ee5f3f
>
> Thank you for your report.
> --
>
> _______________________________________________
> Gcrypt-devel mailing list
> Gcrypt-devel at gnupg.org
> http://lists.gnupg.org/mailman/listinfo/gcrypt-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.gnupg.org/pipermail/gcrypt-devel/attachments/20200903/bf2ab177/attachment.html>

From tianjia.zhang at linux.alibaba.com  Fri Sep 18 11:39:55 2020
From: tianjia.zhang at linux.alibaba.com (Tianjia Zhang)
Date: Fri, 18 Sep 2020 17:39:55 +0800
Subject: https://git.gnupg.org/ website cannot be accessed
Message-ID: <64117fe0-681a-516f-f22d-c03441fd4189@linux.alibaba.com>

Hi,

As the title indicates, I don?t know if it?s a problem with my own 
network, but I can still access it some time ago.

Thanks,
Tianjia


From jussi.kivilinna at iki.fi  Fri Sep 18 17:14:17 2020
From: jussi.kivilinna at iki.fi (Jussi Kivilinna)
Date: Fri, 18 Sep 2020 18:14:17 +0300
Subject: [PATCH] rijndael-aesni: tweak x86_64 AES-NI for better performance on
 AMD Zen2
Message-ID: <20200918151417.1512754-1-jussi.kivilinna@iki.fi>

* cipher/rijndael-aesni.c (do_aesni_enc_vec8, do_aesni_dec_vec8): More
first round key xoring and last round out to caller.
(do_aesni_ctr_4): Change low 8-bit counter overflow check to 8-bit
addition to low-bits and detect overflow from carry flag; Adjust
slow path to restore counter.
(do_aesni_ctr_8): Same as above; Interleave first round key xoring and
first round with CTR generation on fast path; Interleave last round
with output xoring.
(_gcry_aes_aesni_cfb_dec, _gcry_aes_aesni_cbc_dec): Add first round
key xoring; Change order of last round xoring and output xoring
(shorten the dependency path).
(_gcry_aes_aesni_ocb_auth): Add first round key xoring and last round
handling.
--

Benchmark on Ryzen 7 3700X:

Before:
 AES            |  nanosecs/byte   mebibytes/sec   cycles/byte
        CBC dec |     0.113 ns/B      8445 MiB/s     0.407 c/B
        CFB dec |     0.114 ns/B      8337 MiB/s     0.412 c/B
        CTR enc |     0.112 ns/B      8505 MiB/s     0.404 c/B
        CTR dec |     0.113 ns/B      8476 MiB/s     0.405 c/B

After (CBC-dec +21%, CFB-dec +24%, CTR +8% faster):
 AES            |  nanosecs/byte   mebibytes/sec   cycles/byte
        CBC dec |     0.093 ns/B     10277 MiB/s     0.334 c/B
        CFB dec |     0.092 ns/B     10372 MiB/s     0.331 c/B
        CTR enc |     0.104 ns/B      9209 MiB/s     0.373 c/B
        CTR dec |     0.104 ns/B      9192 MiB/s     0.373 c/B

Performance remains the same on Intel Skylake.

Signed-off-by: Jussi Kivilinna <jussi.kivilinna at iki.fi>
---
 cipher/rijndael-aesni.c | 313 +++++++++++++++++++++++-----------------
 1 file changed, 183 insertions(+), 130 deletions(-)

diff --git a/cipher/rijndael-aesni.c b/cipher/rijndael-aesni.c
index b26449a7..747ef662 100644
--- a/cipher/rijndael-aesni.c
+++ b/cipher/rijndael-aesni.c
@@ -788,17 +788,7 @@ do_aesni_dec_vec4 (const RIJNDAEL_context *ctx)
 static ASM_FUNC_ATTR_INLINE void
 do_aesni_enc_vec8 (const RIJNDAEL_context *ctx)
 {
-  asm volatile ("movdqa (%[key]), %%xmm0\n\t"
-                "pxor   %%xmm0, %%xmm1\n\t"     /* xmm1 ^= key[0] */
-                "pxor   %%xmm0, %%xmm2\n\t"     /* xmm2 ^= key[0] */
-                "pxor   %%xmm0, %%xmm3\n\t"     /* xmm3 ^= key[0] */
-                "pxor   %%xmm0, %%xmm4\n\t"     /* xmm4 ^= key[0] */
-                "pxor   %%xmm0, %%xmm8\n\t"     /* xmm8 ^= key[0] */
-                "pxor   %%xmm0, %%xmm9\n\t"     /* xmm9 ^= key[0] */
-                "pxor   %%xmm0, %%xmm10\n\t"     /* xmm10 ^= key[0] */
-                "pxor   %%xmm0, %%xmm11\n\t"     /* xmm11 ^= key[0] */
-                "movdqa 0x10(%[key]), %%xmm0\n\t"
-                "cmpl $12, %[rounds]\n\t"
+  asm volatile ("movdqa 0x10(%[key]), %%xmm0\n\t"
                 "aesenc %%xmm0, %%xmm1\n\t"
                 "aesenc %%xmm0, %%xmm2\n\t"
                 "aesenc %%xmm0, %%xmm3\n\t"
@@ -808,6 +798,7 @@ do_aesni_enc_vec8 (const RIJNDAEL_context *ctx)
                 "aesenc %%xmm0, %%xmm10\n\t"
                 "aesenc %%xmm0, %%xmm11\n\t"
                 "movdqa 0x20(%[key]), %%xmm0\n\t"
+                "cmpl $12, %[rounds]\n\t"
                 "aesenc %%xmm0, %%xmm1\n\t"
                 "aesenc %%xmm0, %%xmm2\n\t"
                 "aesenc %%xmm0, %%xmm3\n\t"
@@ -920,14 +911,6 @@ do_aesni_enc_vec8 (const RIJNDAEL_context *ctx)
                 "movdqa 0xe0(%[key]), %%xmm0\n"
 
                 ".Ldeclast%=:\n\t"
-                "aesenclast %%xmm0, %%xmm1\n\t"
-                "aesenclast %%xmm0, %%xmm2\n\t"
-                "aesenclast %%xmm0, %%xmm3\n\t"
-                "aesenclast %%xmm0, %%xmm4\n\t"
-                "aesenclast %%xmm0, %%xmm8\n\t"
-                "aesenclast %%xmm0, %%xmm9\n\t"
-                "aesenclast %%xmm0, %%xmm10\n\t"
-                "aesenclast %%xmm0, %%xmm11\n\t"
                 : /* no output */
                 : [key] "r" (ctx->keyschenc),
                   [rounds] "r" (ctx->rounds)
@@ -940,16 +923,7 @@ do_aesni_enc_vec8 (const RIJNDAEL_context *ctx)
 static ASM_FUNC_ATTR_INLINE void
 do_aesni_dec_vec8 (const RIJNDAEL_context *ctx)
 {
-  asm volatile ("movdqa (%[key]), %%xmm0\n\t"
-                "pxor   %%xmm0, %%xmm1\n\t"     /* xmm1 ^= key[0] */
-                "pxor   %%xmm0, %%xmm2\n\t"     /* xmm2 ^= key[0] */
-                "pxor   %%xmm0, %%xmm3\n\t"     /* xmm3 ^= key[0] */
-                "pxor   %%xmm0, %%xmm4\n\t"     /* xmm4 ^= key[0] */
-                "pxor   %%xmm0, %%xmm8\n\t"     /* xmm8 ^= key[0] */
-                "pxor   %%xmm0, %%xmm9\n\t"     /* xmm9 ^= key[0] */
-                "pxor   %%xmm0, %%xmm10\n\t"    /* xmm10 ^= key[0] */
-                "pxor   %%xmm0, %%xmm11\n\t"    /* xmm11 ^= key[0] */
-                "movdqa 0x10(%[key]), %%xmm0\n\t"
+  asm volatile ("movdqa 0x10(%[key]), %%xmm0\n\t"
                 "cmpl $12, %[rounds]\n\t"
                 "aesdec %%xmm0, %%xmm1\n\t"
                 "aesdec %%xmm0, %%xmm2\n\t"
@@ -1072,14 +1046,6 @@ do_aesni_dec_vec8 (const RIJNDAEL_context *ctx)
                 "movdqa 0xe0(%[key]), %%xmm0\n"
 
                 ".Ldeclast%=:\n\t"
-                "aesdeclast %%xmm0, %%xmm1\n\t"
-                "aesdeclast %%xmm0, %%xmm2\n\t"
-                "aesdeclast %%xmm0, %%xmm3\n\t"
-                "aesdeclast %%xmm0, %%xmm4\n\t"
-                "aesdeclast %%xmm0, %%xmm8\n\t"
-                "aesdeclast %%xmm0, %%xmm9\n\t"
-                "aesdeclast %%xmm0, %%xmm10\n\t"
-                "aesdeclast %%xmm0, %%xmm11\n\t"
                 : /* no output */
                 : [key] "r" (ctx->keyschdec),
                   [rounds] "r" (ctx->rounds)
@@ -1204,8 +1170,8 @@ do_aesni_ctr_4 (const RIJNDAEL_context *ctx,
    */
 
   asm volatile (/* detect if 8-bit carry handling is needed */
-                "cmpb   $0xfb, 15(%[ctr])\n\t"
-                "ja     .Ladd32bit%=\n\t"
+                "addb   $4, 15(%[ctr])\n\t"
+                "jc     .Ladd32bit%=\n\t"
 
                 "movdqa %%xmm5, %%xmm0\n\t"     /* xmm0 := CTR (xmm5) */
                 "movdqa 0*16(%[addb]), %%xmm2\n\t"  /* xmm2 := be(1) */
@@ -1217,9 +1183,10 @@ do_aesni_ctr_4 (const RIJNDAEL_context *ctx,
                 "paddb  %%xmm0, %%xmm4\n\t"     /* xmm4 := be(3) + CTR (xmm0) */
                 "paddb  %%xmm0, %%xmm5\n\t"     /* xmm5 := be(4) + CTR (xmm0) */
                 "movdqa (%[key]), %%xmm1\n\t"   /* xmm1 := key[0] */
-                "jmp    .Lstore_ctr%=\n\t"
+                "jmp    .Ldone_ctr%=\n\t"
 
                 ".Ladd32bit%=:\n\t"
+                "movdqa %%xmm5, (%[ctr])\n\t"   /* Restore CTR.  */
                 "movdqa %%xmm5, %%xmm0\n\t"     /* xmm0, xmm2 := CTR (xmm5) */
                 "movdqa %%xmm0, %%xmm2\n\t"
                 "pcmpeqd %%xmm1, %%xmm1\n\t"
@@ -1265,8 +1232,9 @@ do_aesni_ctr_4 (const RIJNDAEL_context *ctx,
                 "pshufb %%xmm6, %%xmm4\n\t"     /* xmm4 := be(xmm4) */
                 "pshufb %%xmm6, %%xmm5\n\t"     /* xmm5 := be(xmm5) */
 
-                ".Lstore_ctr%=:\n\t"
                 "movdqa %%xmm5, (%[ctr])\n\t"   /* Update CTR (mem).  */
+
+                ".Ldone_ctr%=:\n\t"
                 :
                 : [ctr] "r" (ctr),
                   [key] "r" (ctx->keyschenc),
@@ -1428,30 +1396,50 @@ do_aesni_ctr_8 (const RIJNDAEL_context *ctx,
    */
 
   asm volatile (/* detect if 8-bit carry handling is needed */
-                "cmpb   $0xf7, 15(%[ctr])\n\t"
-                "ja     .Ladd32bit%=\n\t"
+                "addb   $8, 15(%[ctr])\n\t"
+                "jc     .Ladd32bit%=\n\t"
 
-                "movdqa %%xmm5, %%xmm0\n\t"     /* xmm0 := CTR (xmm5) */
-                "movdqa 0*16(%[addb]), %%xmm2\n\t"  /* xmm2 := be(1) */
-                "movdqa 1*16(%[addb]), %%xmm3\n\t"  /* xmm3 := be(2) */
-                "movdqa 2*16(%[addb]), %%xmm4\n\t"  /* xmm4 := be(3) */
-                "movdqa 3*16(%[addb]), %%xmm8\n\t"  /* xmm8 := be(4) */
-                "movdqa 4*16(%[addb]), %%xmm9\n\t"  /* xmm9 := be(5) */
-                "movdqa 5*16(%[addb]), %%xmm10\n\t" /* xmm10 := be(6) */
-                "movdqa 6*16(%[addb]), %%xmm11\n\t" /* xmm11 := be(7) */
-                "movdqa 7*16(%[addb]), %%xmm5\n\t"  /* xmm5 := be(8) */
                 "movdqa (%[key]), %%xmm1\n\t"   /* xmm1 := key[0] */
-                "paddb  %%xmm0, %%xmm2\n\t"     /* xmm2 := be(1) + CTR (xmm0) */
-                "paddb  %%xmm0, %%xmm3\n\t"     /* xmm3 := be(2) + CTR (xmm0) */
-                "paddb  %%xmm0, %%xmm4\n\t"     /* xmm4 := be(3) + CTR (xmm0) */
-                "paddb  %%xmm0, %%xmm8\n\t"     /* xmm8 := be(4) + CTR (xmm0) */
-                "paddb  %%xmm0, %%xmm9\n\t"     /* xmm9 := be(5) + CTR (xmm0) */
-                "paddb  %%xmm0, %%xmm10\n\t"    /* xmm10 := be(6) + CTR (xmm0) */
-                "paddb  %%xmm0, %%xmm11\n\t"    /* xmm11 := be(7) + CTR (xmm0) */
-                "paddb  %%xmm0, %%xmm5\n\t"     /* xmm5 := be(8) + CTR (xmm0) */
-                "jmp    .Lstore_ctr%=\n\t"
+                "movdqa 16(%[key]), %%xmm7\n\t" /* xmm7 := key[1] */
+
+                "movdqa %%xmm5, %%xmm0\n\t"     /* xmm0 := CTR (xmm5) */
+                "movdqa %%xmm5, %%xmm2\n\t"     /* xmm2 := CTR (xmm5) */
+                "movdqa %%xmm5, %%xmm3\n\t"     /* xmm3 := CTR (xmm5) */
+                "movdqa %%xmm5, %%xmm4\n\t"     /* xmm4 := CTR (xmm5) */
+                "paddb  0*16(%[addb]), %%xmm2\n\t" /* xmm2 := be(1) + CTR */
+                "paddb  1*16(%[addb]), %%xmm3\n\t" /* xmm3 := be(2) + CTR */
+                "paddb  2*16(%[addb]), %%xmm4\n\t" /* xmm4 := be(3) + CTR */
+                "pxor   %%xmm1, %%xmm0\n\t"     /* xmm0 ^= key[0]    */
+                "pxor   %%xmm1, %%xmm2\n\t"     /* xmm2 ^= key[0]    */
+                "pxor   %%xmm1, %%xmm3\n\t"     /* xmm3 ^= key[0]    */
+                "pxor   %%xmm1, %%xmm4\n\t"     /* xmm4 ^= key[0]    */
+                "aesenc %%xmm7, %%xmm0\n\t"
+                "aesenc %%xmm7, %%xmm2\n\t"
+                "aesenc %%xmm7, %%xmm3\n\t"
+                "aesenc %%xmm7, %%xmm4\n\t"
+                "movdqa %%xmm5, %%xmm8\n\t"     /* xmm8 := CTR (xmm5) */
+                "movdqa %%xmm5, %%xmm9\n\t"     /* xmm9 := CTR (xmm5) */
+                "movdqa %%xmm5, %%xmm10\n\t"    /* xmm10 := CTR (xmm5) */
+                "movdqa %%xmm5, %%xmm11\n\t"    /* xmm11 := CTR (xmm5) */
+                "paddb  3*16(%[addb]), %%xmm8\n\t"  /* xmm8 := be(4) + CTR */
+                "paddb  4*16(%[addb]), %%xmm9\n\t"  /* xmm9 := be(5) + CTR */
+                "paddb  5*16(%[addb]), %%xmm10\n\t" /* xmm10 := be(6) + CTR */
+                "paddb  6*16(%[addb]), %%xmm11\n\t" /* xmm11 := be(7) + CTR */
+                "pxor   %%xmm1, %%xmm8\n\t"     /* xmm8 ^= key[0]    */
+                "pxor   %%xmm1, %%xmm9\n\t"     /* xmm9 ^= key[0]    */
+                "pxor   %%xmm1, %%xmm10\n\t"    /* xmm10 ^= key[0]   */
+                "pxor   %%xmm1, %%xmm11\n\t"    /* xmm11 ^= key[0]   */
+                "aesenc %%xmm7, %%xmm8\n\t"
+                "aesenc %%xmm7, %%xmm9\n\t"
+                "aesenc %%xmm7, %%xmm10\n\t"
+                "aesenc %%xmm7, %%xmm11\n\t"
+
+                "paddb  7*16(%[addb]), %%xmm5\n\t" /* xmm5 := be(8) + CTR */
+
+                "jmp    .Ldone_ctr%=\n\t"
 
                 ".Ladd32bit%=:\n\t"
+                "movdqa %%xmm5, (%[ctr])\n\t"   /* Restore CTR. */
                 "movdqa %%xmm5, %%xmm0\n\t"     /* xmm0, xmm2 := CTR (xmm5) */
                 "movdqa %%xmm0, %%xmm2\n\t"
                 "pcmpeqd %%xmm1, %%xmm1\n\t"
@@ -1512,44 +1500,50 @@ do_aesni_ctr_8 (const RIJNDAEL_context *ctx,
                 "psubq   %%xmm1, %%xmm5\n\t"
 
                 ".Lno_carry%=:\n\t"
-                "movdqa (%[key]), %%xmm1\n\t"   /* xmm1 := key[0]    */
+                "movdqa (%[key]), %%xmm1\n\t"   /* xmm1 := key[0] */
+                "movdqa 16(%[key]), %%xmm7\n\t" /* xmm7 := key[1] */
 
                 "pshufb %%xmm6, %%xmm2\n\t"     /* xmm2 := be(xmm2) */
                 "pshufb %%xmm6, %%xmm3\n\t"     /* xmm3 := be(xmm3) */
                 "pshufb %%xmm6, %%xmm4\n\t"     /* xmm4 := be(xmm4) */
-                "pshufb %%xmm6, %%xmm5\n\t"     /* xmm5 := be(xmm5) */
+                "pxor   %%xmm1, %%xmm0\n\t"     /* xmm0 ^= key[0]    */
+                "pxor   %%xmm1, %%xmm2\n\t"     /* xmm2 ^= key[0]    */
+                "pxor   %%xmm1, %%xmm3\n\t"     /* xmm3 ^= key[0]    */
+                "pxor   %%xmm1, %%xmm4\n\t"     /* xmm4 ^= key[0]    */
+                "aesenc %%xmm7, %%xmm0\n\t"
+                "aesenc %%xmm7, %%xmm2\n\t"
+                "aesenc %%xmm7, %%xmm3\n\t"
+                "aesenc %%xmm7, %%xmm4\n\t"
                 "pshufb %%xmm6, %%xmm8\n\t"     /* xmm8 := be(xmm8) */
                 "pshufb %%xmm6, %%xmm9\n\t"     /* xmm9 := be(xmm9) */
                 "pshufb %%xmm6, %%xmm10\n\t"    /* xmm10 := be(xmm10) */
                 "pshufb %%xmm6, %%xmm11\n\t"    /* xmm11 := be(xmm11) */
+                "pxor   %%xmm1, %%xmm8\n\t"     /* xmm8 ^= key[0]    */
+                "pxor   %%xmm1, %%xmm9\n\t"     /* xmm9 ^= key[0]    */
+                "pxor   %%xmm1, %%xmm10\n\t"    /* xmm10 ^= key[0]   */
+                "pxor   %%xmm1, %%xmm11\n\t"    /* xmm11 ^= key[0]   */
+                "aesenc %%xmm7, %%xmm8\n\t"
+                "aesenc %%xmm7, %%xmm9\n\t"
+                "aesenc %%xmm7, %%xmm10\n\t"
+                "aesenc %%xmm7, %%xmm11\n\t"
 
-                ".Lstore_ctr%=:\n\t"
+                "pshufb %%xmm6, %%xmm5\n\t"     /* xmm5 := be(xmm5) */
                 "movdqa %%xmm5, (%[ctr])\n\t"   /* Update CTR (mem).  */
+
+                ".align 16\n\t"
+                ".Ldone_ctr%=:\n\t"
                 :
                 : [ctr] "r" (ctr),
                   [key] "r" (ctx->keyschenc),
                   [addb] "r" (bige_addb)
                 : "%esi", "cc", "memory");
 
-  asm volatile ("pxor   %%xmm1, %%xmm0\n\t"     /* xmm0 ^= key[0]    */
-                "pxor   %%xmm1, %%xmm2\n\t"     /* xmm2 ^= key[0]    */
-                "pxor   %%xmm1, %%xmm3\n\t"     /* xmm3 ^= key[0]    */
-                "pxor   %%xmm1, %%xmm4\n\t"     /* xmm4 ^= key[0]    */
-                "pxor   %%xmm1, %%xmm8\n\t"     /* xmm8 ^= key[0]    */
-                "pxor   %%xmm1, %%xmm9\n\t"     /* xmm9 ^= key[0]    */
-                "pxor   %%xmm1, %%xmm10\n\t"    /* xmm10 ^= key[0]   */
-                "pxor   %%xmm1, %%xmm11\n\t"    /* xmm11 ^= key[0]   */
-                "movdqa 0x10(%[key]), %%xmm1\n\t"
-                "cmpl $12, %[rounds]\n\t"
-                "aesenc %%xmm1, %%xmm0\n\t"
-                "aesenc %%xmm1, %%xmm2\n\t"
-                "aesenc %%xmm1, %%xmm3\n\t"
-                "aesenc %%xmm1, %%xmm4\n\t"
-                "aesenc %%xmm1, %%xmm8\n\t"
-                "aesenc %%xmm1, %%xmm9\n\t"
-                "aesenc %%xmm1, %%xmm10\n\t"
-                "aesenc %%xmm1, %%xmm11\n\t"
-                "movdqa 0x20(%[key]), %%xmm1\n\t"
+  asm volatile ("movdqa 0x20(%[key]), %%xmm1\n\t"
+                "movdqu 0*16(%[src]), %%xmm12\n\t" /* Get block 1.      */
+                "movdqu 1*16(%[src]), %%xmm13\n\t" /* Get block 2.      */
+                "movdqu 2*16(%[src]), %%xmm14\n\t" /* Get block 3.      */
+                "movdqu 3*16(%[src]), %%xmm15\n\t" /* Get block 4.      */
+                "movdqu 4*16(%[src]), %%xmm7\n\t"  /* Get block 5.      */
                 "aesenc %%xmm1, %%xmm0\n\t"
                 "aesenc %%xmm1, %%xmm2\n\t"
                 "aesenc %%xmm1, %%xmm3\n\t"
@@ -1558,6 +1552,7 @@ do_aesni_ctr_8 (const RIJNDAEL_context *ctx,
                 "aesenc %%xmm1, %%xmm9\n\t"
                 "aesenc %%xmm1, %%xmm10\n\t"
                 "aesenc %%xmm1, %%xmm11\n\t"
+                "cmpl $12, %[rounds]\n\t"
                 "movdqa 0x30(%[key]), %%xmm1\n\t"
                 "aesenc %%xmm1, %%xmm0\n\t"
                 "aesenc %%xmm1, %%xmm2\n\t"
@@ -1664,38 +1659,33 @@ do_aesni_ctr_8 (const RIJNDAEL_context *ctx,
                 ".Lenclast%=:\n\t"
                 :
                 : [key] "r" (ctx->keyschenc),
-                  [rounds] "r" (ctx->rounds)
+                  [rounds] "r" (ctx->rounds),
+                  [src] "r" (a)
                 : "cc", "memory");
 
-  asm volatile ("movdqu 0*16(%[src]), %%xmm12\n\t" /* Get block 1.      */
-                "movdqu 1*16(%[src]), %%xmm13\n\t" /* Get block 2.      */
-                "movdqu 2*16(%[src]), %%xmm14\n\t" /* Get block 3.      */
-                "movdqu 3*16(%[src]), %%xmm15\n\t" /* Get block 4.      */
-                "movdqu 4*16(%[src]), %%xmm7\n\t"  /* Get block 5.      */
-                "pxor %%xmm1, %%xmm12\n\t"         /* block1 ^= lastkey */
-                "aesenclast %%xmm12, %%xmm0\n\t"
-                "movdqu 5*16(%[src]), %%xmm12\n\t" /* Get block 6.      */
+  asm volatile ("pxor %%xmm1, %%xmm12\n\t"         /* block1 ^= lastkey */
                 "pxor %%xmm1, %%xmm13\n\t"         /* block2 ^= lastkey */
-                "aesenclast %%xmm13, %%xmm2\n\t"
-                "movdqu 6*16(%[src]), %%xmm13\n\t" /* Get block 7.      */
                 "pxor %%xmm1, %%xmm14\n\t"         /* block3 ^= lastkey */
-                "aesenclast %%xmm14, %%xmm3\n\t"
-                "movdqu 7*16(%[src]), %%xmm14\n\t" /* Get block 8.      */
                 "pxor %%xmm1, %%xmm15\n\t"         /* block4 ^= lastkey */
+                "aesenclast %%xmm12, %%xmm0\n\t"
+                "aesenclast %%xmm13, %%xmm2\n\t"
+                "aesenclast %%xmm14, %%xmm3\n\t"
                 "aesenclast %%xmm15, %%xmm4\n\t"
-                "movdqu %%xmm0, 0*16(%[dst])\n\t"  /* Store block 1     */
-                "pxor %%xmm1,  %%xmm7\n\t"         /* block5 ^= lastkey */
-                "aesenclast %%xmm7, %%xmm8\n\t"
-                "movdqu %%xmm0, 0*16(%[dst])\n\t"  /* Store block 1     */
-                "pxor %%xmm1, %%xmm12\n\t"         /* block6 ^= lastkey */
-                "aesenclast %%xmm12, %%xmm9\n\t"
+                "movdqu 5*16(%[src]), %%xmm12\n\t" /* Get block 6.      */
+                "movdqu 6*16(%[src]), %%xmm13\n\t" /* Get block 7.      */
+                "movdqu 7*16(%[src]), %%xmm14\n\t" /* Get block 8.      */
+                "movdqu %%xmm0, 0*16(%[dst])\n\t"  /* Store block 1.    */
                 "movdqu %%xmm2, 1*16(%[dst])\n\t"  /* Store block 2.    */
-                "pxor %%xmm1, %%xmm13\n\t"         /* block7 ^= lastkey */
-                "aesenclast %%xmm13, %%xmm10\n\t"
                 "movdqu %%xmm3, 2*16(%[dst])\n\t"  /* Store block 3.    */
+                "movdqu %%xmm4, 3*16(%[dst])\n\t"  /* Store block 4.    */
+                "pxor %%xmm1, %%xmm7\n\t"          /* block5 ^= lastkey */
+                "pxor %%xmm1, %%xmm12\n\t"         /* block6 ^= lastkey */
+                "pxor %%xmm1, %%xmm13\n\t"         /* block7 ^= lastkey */
                 "pxor %%xmm1, %%xmm14\n\t"         /* block8 ^= lastkey */
+                "aesenclast %%xmm7, %%xmm8\n\t"
+                "aesenclast %%xmm12, %%xmm9\n\t"
+                "aesenclast %%xmm13, %%xmm10\n\t"
                 "aesenclast %%xmm14, %%xmm11\n\t"
-                "movdqu %%xmm4, 3*16(%[dst])\n\t"  /* Store block 4.    */
                 "movdqu %%xmm8, 4*16(%[dst])\n\t"  /* Store block 8.    */
                 "movdqu %%xmm9, 5*16(%[dst])\n\t"  /* Store block 9.    */
                 "movdqu %%xmm10, 6*16(%[dst])\n\t" /* Store block 10.   */
@@ -1910,7 +1900,9 @@ _gcry_aes_aesni_cfb_dec (RIJNDAEL_context *ctx, unsigned char *iv,
       for ( ;nblocks >= 8; nblocks -= 8)
 	{
 	  asm volatile
-	    ("movdqu %%xmm6,         %%xmm1\n\t" /* load input blocks */
+	    ("movdqa (%[key]), %%xmm0\n\t"
+
+	     "movdqu %%xmm6,         %%xmm1\n\t" /* load input blocks */
 	     "movdqu 0*16(%[inbuf]), %%xmm2\n\t"
 	     "movdqu 1*16(%[inbuf]), %%xmm3\n\t"
 	     "movdqu 2*16(%[inbuf]), %%xmm4\n\t"
@@ -1925,30 +1917,50 @@ _gcry_aes_aesni_cfb_dec (RIJNDAEL_context *ctx, unsigned char *iv,
 	     "movdqa %%xmm3, %%xmm13\n\t"
 	     "movdqa %%xmm4, %%xmm14\n\t"
 	     "movdqa %%xmm8, %%xmm15\n\t"
+
+             "pxor   %%xmm0, %%xmm1\n\t"     /* xmm1 ^= key[0] */
+             "pxor   %%xmm0, %%xmm2\n\t"     /* xmm2 ^= key[0] */
+             "pxor   %%xmm0, %%xmm3\n\t"     /* xmm3 ^= key[0] */
+             "pxor   %%xmm0, %%xmm4\n\t"     /* xmm4 ^= key[0] */
+             "pxor   %%xmm0, %%xmm8\n\t"     /* xmm8 ^= key[0] */
+             "pxor   %%xmm0, %%xmm9\n\t"     /* xmm9 ^= key[0] */
+             "pxor   %%xmm0, %%xmm10\n\t"     /* xmm10 ^= key[0] */
+             "pxor   %%xmm0, %%xmm11\n\t"     /* xmm11 ^= key[0] */
 	     : /* No output */
-	     : [inbuf] "r" (inbuf)
+	     : [inbuf] "r" (inbuf),
+	       [key] "r" (ctx->keyschenc)
 	     : "memory");
 
 	  do_aesni_enc_vec8 (ctx);
 
 	  asm volatile
 	    (
-	     "pxor %%xmm12, %%xmm1\n\t"
+	     "pxor %%xmm0, %%xmm12\n\t"
+	     "pxor %%xmm0, %%xmm13\n\t"
+	     "pxor %%xmm0, %%xmm14\n\t"
+	     "pxor %%xmm0, %%xmm15\n\t"
+	     "aesenclast %%xmm12, %%xmm1\n\t"
+	     "aesenclast %%xmm13, %%xmm2\n\t"
+	     "aesenclast %%xmm14, %%xmm3\n\t"
+	     "aesenclast %%xmm15, %%xmm4\n\t"
+
 	     "movdqu 4*16(%[inbuf]), %%xmm12\n\t"
-	     "pxor %%xmm13, %%xmm2\n\t"
 	     "movdqu 5*16(%[inbuf]), %%xmm13\n\t"
-	     "pxor %%xmm14, %%xmm3\n\t"
 	     "movdqu 6*16(%[inbuf]), %%xmm14\n\t"
-	     "pxor %%xmm15, %%xmm4\n\t"
 	     "movdqu 7*16(%[inbuf]), %%xmm15\n\t"
+	     "pxor %%xmm0, %%xmm12\n\t"
+	     "pxor %%xmm0, %%xmm13\n\t"
+	     "pxor %%xmm0, %%xmm14\n\t"
+	     "pxor %%xmm0, %%xmm15\n\t"
+
+	     "aesenclast %%xmm12, %%xmm8\n\t"
+	     "aesenclast %%xmm13, %%xmm9\n\t"
+	     "aesenclast %%xmm14, %%xmm10\n\t"
+	     "aesenclast %%xmm15, %%xmm11\n\t"
 
-	     "pxor %%xmm12, %%xmm8\n\t"
 	     "movdqu %%xmm1, 0*16(%[outbuf])\n\t"
-	     "pxor %%xmm13, %%xmm9\n\t"
 	     "movdqu %%xmm2, 1*16(%[outbuf])\n\t"
-	     "pxor %%xmm14, %%xmm10\n\t"
 	     "movdqu %%xmm3, 2*16(%[outbuf])\n\t"
-	     "pxor %%xmm15, %%xmm11\n\t"
 	     "movdqu %%xmm4, 3*16(%[outbuf])\n\t"
 
 	     "movdqu %%xmm8, 4*16(%[outbuf])\n\t"
@@ -2070,7 +2082,9 @@ _gcry_aes_aesni_cbc_dec (RIJNDAEL_context *ctx, unsigned char *iv,
       for ( ;nblocks >= 8 ; nblocks -= 8 )
 	{
 	  asm volatile
-	    ("movdqu 0*16(%[inbuf]), %%xmm1\n\t"	/* load input blocks */
+	    ("movdqa (%[key]), %%xmm0\n\t"
+
+	     "movdqu 0*16(%[inbuf]), %%xmm1\n\t"	/* load input blocks */
 	     "movdqu 1*16(%[inbuf]), %%xmm2\n\t"
 	     "movdqu 2*16(%[inbuf]), %%xmm3\n\t"
 	     "movdqu 3*16(%[inbuf]), %%xmm4\n\t"
@@ -2084,31 +2098,50 @@ _gcry_aes_aesni_cbc_dec (RIJNDAEL_context *ctx, unsigned char *iv,
 	     "movdqa %%xmm3, %%xmm14\n\t"
 	     "movdqa %%xmm4, %%xmm15\n\t"
 
+	     "pxor   %%xmm0, %%xmm1\n\t"     /* xmm1 ^= key[0] */
+	     "pxor   %%xmm0, %%xmm2\n\t"     /* xmm2 ^= key[0] */
+	     "pxor   %%xmm0, %%xmm3\n\t"     /* xmm3 ^= key[0] */
+	     "pxor   %%xmm0, %%xmm4\n\t"     /* xmm4 ^= key[0] */
+	     "pxor   %%xmm0, %%xmm8\n\t"     /* xmm8 ^= key[0] */
+	     "pxor   %%xmm0, %%xmm9\n\t"     /* xmm9 ^= key[0] */
+	     "pxor   %%xmm0, %%xmm10\n\t"    /* xmm10 ^= key[0] */
+	     "pxor   %%xmm0, %%xmm11\n\t"    /* xmm11 ^= key[0] */
+
 	     : /* No output */
-	     : [inbuf] "r" (inbuf)
+	     : [inbuf] "r" (inbuf),
+	       [key] "r" (ctx->keyschdec)
 	     : "memory");
 
 	  do_aesni_dec_vec8 (ctx);
 
 	  asm volatile
-	    ("pxor %%xmm5, %%xmm1\n\t"		/* xor IV with output */
+	    (
+	     "pxor %%xmm0, %%xmm5\n\t"			/* xor IV with key */
+	     "pxor %%xmm0, %%xmm12\n\t"			/* xor IV with key */
+	     "pxor %%xmm0, %%xmm13\n\t"			/* xor IV with key */
+	     "pxor %%xmm0, %%xmm14\n\t"			/* xor IV with key */
+	     "pxor %%xmm0, %%xmm15\n\t"			/* xor IV with key */
 
-	     "pxor %%xmm12, %%xmm2\n\t"		/* xor IV with output */
-	     "movdqu 4*16(%[inbuf]), %%xmm12\n\t"
+	     "aesdeclast %%xmm5, %%xmm1\n\t"
+	     "aesdeclast %%xmm12, %%xmm2\n\t"
+	     "aesdeclast %%xmm13, %%xmm3\n\t"
+	     "aesdeclast %%xmm14, %%xmm4\n\t"
 
-	     "pxor %%xmm13, %%xmm3\n\t"		/* xor IV with output */
+	     "movdqu 4*16(%[inbuf]), %%xmm12\n\t"
 	     "movdqu 5*16(%[inbuf]), %%xmm13\n\t"
-
-	     "pxor %%xmm14, %%xmm4\n\t"		/* xor IV with output */
 	     "movdqu 6*16(%[inbuf]), %%xmm14\n\t"
-
-	     "pxor %%xmm15, %%xmm8\n\t"		/* xor IV with output */
 	     "movdqu 7*16(%[inbuf]), %%xmm5\n\t"
-	     "pxor %%xmm12, %%xmm9\n\t"		/* xor IV with output */
+	     "pxor %%xmm0, %%xmm12\n\t"			/* xor IV with key */
+	     "pxor %%xmm0, %%xmm13\n\t"			/* xor IV with key */
+	     "pxor %%xmm0, %%xmm14\n\t"			/* xor IV with key */
+
+	     "aesdeclast %%xmm15, %%xmm8\n\t"
+	     "aesdeclast %%xmm12, %%xmm9\n\t"
+	     "aesdeclast %%xmm13, %%xmm10\n\t"
+	     "aesdeclast %%xmm14, %%xmm11\n\t"
+
 	     "movdqu %%xmm1, 0*16(%[outbuf])\n\t"
-	     "pxor %%xmm13, %%xmm10\n\t"		/* xor IV with output */
 	     "movdqu %%xmm2, 1*16(%[outbuf])\n\t"
-	     "pxor %%xmm14, %%xmm11\n\t"		/* xor IV with output */
 	     "movdqu %%xmm3, 2*16(%[outbuf])\n\t"
 	     "movdqu %%xmm4, 3*16(%[outbuf])\n\t"
 	     "movdqu %%xmm8, 4*16(%[outbuf])\n\t"
@@ -3452,8 +3485,14 @@ _gcry_aes_aesni_ocb_auth (gcry_cipher_hd_t c, const void *abuf_arg,
 			"pxor   %%xmm5,    %%xmm3\n\t"
 
 			"pxor   %%xmm0,    %%xmm5\n\t"
+			"movdqa (%[key]),  %%xmm0\n\t"
 			"pxor   %%xmm5,    %%xmm4\n\t"
 
+			"pxor   %%xmm0, %%xmm1\n\t"     /* xmm1 ^= key[0] */
+			"pxor   %%xmm0, %%xmm2\n\t"     /* xmm2 ^= key[0] */
+			"pxor   %%xmm0, %%xmm3\n\t"     /* xmm3 ^= key[0] */
+			"pxor   %%xmm0, %%xmm4\n\t"     /* xmm4 ^= key[0] */
+
 			"pxor   %%xmm7,    %%xmm8\n\t"
 			"pxor   %%xmm5,    %%xmm8\n\t"
 
@@ -3465,13 +3504,27 @@ _gcry_aes_aesni_ocb_auth (gcry_cipher_hd_t c, const void *abuf_arg,
 
 			"pxor   %%xmm14,   %%xmm5\n\t"
 			"pxor   %%xmm5,    %%xmm11\n\t"
+
+			"pxor   %%xmm0, %%xmm8\n\t"     /* xmm8 ^= key[0] */
+			"pxor   %%xmm0, %%xmm9\n\t"     /* xmm9 ^= key[0] */
+			"pxor   %%xmm0, %%xmm10\n\t"    /* xmm10 ^= key[0] */
+			"pxor   %%xmm0, %%xmm11\n\t"    /* xmm11 ^= key[0] */
 			:
-			:
+			: [key] "r" (ctx->keyschenc)
 			: "memory" );
 
 	  do_aesni_enc_vec8 (ctx);
 
-	  asm volatile ("pxor   %%xmm2,   %%xmm1\n\t"
+	  asm volatile (
+			"aesenclast %%xmm0, %%xmm1\n\t"
+			"aesenclast %%xmm0, %%xmm2\n\t"
+			"aesenclast %%xmm0, %%xmm3\n\t"
+			"aesenclast %%xmm0, %%xmm4\n\t"
+			"aesenclast %%xmm0, %%xmm8\n\t"
+			"aesenclast %%xmm0, %%xmm9\n\t"
+			"aesenclast %%xmm0, %%xmm10\n\t"
+			"aesenclast %%xmm0, %%xmm11\n\t"
+			"pxor   %%xmm2,   %%xmm1\n\t"
 			"pxor   %%xmm3,   %%xmm1\n\t"
 			"pxor   %%xmm4,   %%xmm1\n\t"
 			"pxor   %%xmm8,   %%xmm1\n\t"
-- 
2.25.1



From tianjia.zhang at linux.alibaba.com  Mon Sep 21 15:01:49 2020
From: tianjia.zhang at linux.alibaba.com (Tianjia Zhang)
Date: Mon, 21 Sep 2020 21:01:49 +0800
Subject: [PATCH] tests: Fix typo in comment
Message-ID: <20200921130149.90824-1-tianjia.zhang@linux.alibaba.com>

* tests/basic.c: Fix typo in comment.
--

Signed-off-by: Tianjia Zhang <tianjia.zhang at linux.alibaba.com>
---
 tests/basic.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/basic.c b/tests/basic.c
index fdc5df4b..1d12c4a2 100644
--- a/tests/basic.c
+++ b/tests/basic.c
@@ -13036,7 +13036,7 @@ verify_one_signature (gcry_sexp_t pkey, gcry_sexp_t hash,
 }
 
 
-/* Test the public key sign function using the private ket SKEY. PKEY
+/* Test the public key sign function using the private key SKEY. PKEY
    is used for verification. */
 static void
 check_pubkey_sign (int n, gcry_sexp_t skey, gcry_sexp_t pkey, int algo)
@@ -13149,7 +13149,7 @@ check_pubkey_sign (int n, gcry_sexp_t skey, gcry_sexp_t pkey, int algo)
 }
 
 
-/* Test the public key sign function using the private ket SKEY. PKEY
+/* Test the public key sign function using the private key SKEY. PKEY
    is used for verification.  This variant is only used for ECDSA.  */
 static void
 check_pubkey_sign_ecdsa (int n, gcry_sexp_t skey, gcry_sexp_t pkey)
-- 
2.19.1.3.ge56e4f7



From stefbon at gmail.com  Sun Sep 27 07:22:36 2020
From: stefbon at gmail.com (Stef Bon)
Date: Sun, 27 Sep 2020 07:22:36 +0200
Subject: Generate ECDH shared key - NIST P256
In-Reply-To: <518964648.31228.1592829871223.JavaMail.administrator@n7.nabble.com>
References: <518964648.31228.1592829871223.JavaMail.administrator@n7.nabble.com>
Message-ID: <CANXojcyg_ZSEvcr1VUhhfF8UZnaYBkpfiWUuW1xBgBKt5ryqHQ@mail.gmail.com>

Do you have found a solution yet?

Stef


From jussi.kivilinna at iki.fi  Sun Sep 27 20:08:25 2020
From: jussi.kivilinna at iki.fi (Jussi Kivilinna)
Date: Sun, 27 Sep 2020 21:08:25 +0300
Subject: [PATCH 1/4] rijndael: tidy do_setkey little bit
Message-ID: <20200927180828.744564-1-jussi.kivilinna@iki.fi>

* cipher/rijndael.c (do_setkey): Reduce number of ifdefs by using
function pointer for accelerated key-setup.
--

Signed-off-by: Jussi Kivilinna <jussi.kivilinna at iki.fi>
---
 cipher/rijndael.c | 26 ++++++++------------------
 1 file changed, 8 insertions(+), 18 deletions(-)

diff --git a/cipher/rijndael.c b/cipher/rijndael.c
index 3e9bae55..d81ed809 100644
--- a/cipher/rijndael.c
+++ b/cipher/rijndael.c
@@ -353,6 +353,7 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen,
 {
   static int initialized = 0;
   static const char *selftest_failed = 0;
+  void (*hw_setkey)(RIJNDAEL_context *ctx, const byte *key) = NULL;
   int rounds;
   int i,j, r, t, rconpointer = 0;
   int KC;
@@ -432,6 +433,7 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen,
 #ifdef USE_AESNI
   else if (hwfeatures & HWF_INTEL_AESNI)
     {
+      hw_setkey = _gcry_aes_aesni_do_setkey;
       ctx->encrypt_fn = _gcry_aes_aesni_encrypt;
       ctx->decrypt_fn = _gcry_aes_aesni_decrypt;
       ctx->prefetch_enc_fn = NULL;
@@ -466,6 +468,7 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen,
 #ifdef USE_SSSE3
   else if (hwfeatures & HWF_INTEL_SSSE3)
     {
+      hw_setkey = _gcry_aes_ssse3_do_setkey;
       ctx->encrypt_fn = _gcry_aes_ssse3_encrypt;
       ctx->decrypt_fn = _gcry_aes_ssse3_decrypt;
       ctx->prefetch_enc_fn = NULL;
@@ -486,6 +489,7 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen,
 #ifdef USE_ARM_CE
   else if (hwfeatures & HWF_ARM_AES)
     {
+      hw_setkey = _gcry_aes_armv8_ce_setkey;
       ctx->encrypt_fn = _gcry_aes_armv8_ce_encrypt;
       ctx->decrypt_fn = _gcry_aes_armv8_ce_decrypt;
       ctx->prefetch_enc_fn = NULL;
@@ -507,6 +511,7 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen,
 #ifdef USE_PPC_CRYPTO_WITH_PPC9LE
   else if ((hwfeatures & HWF_PPC_VCRYPTO) && (hwfeatures & HWF_PPC_ARCH_3_00))
     {
+      hw_setkey = _gcry_aes_ppc8_setkey;
       ctx->encrypt_fn = _gcry_aes_ppc9le_encrypt;
       ctx->decrypt_fn = _gcry_aes_ppc9le_decrypt;
       ctx->prefetch_enc_fn = NULL;
@@ -529,6 +534,7 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen,
 #ifdef USE_PPC_CRYPTO
   else if (hwfeatures & HWF_PPC_VCRYPTO)
     {
+      hw_setkey = _gcry_aes_ppc8_setkey;
       ctx->encrypt_fn = _gcry_aes_ppc8_encrypt;
       ctx->decrypt_fn = _gcry_aes_ppc8_decrypt;
       ctx->prefetch_enc_fn = NULL;
@@ -557,26 +563,10 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen,
 
   /* NB: We don't yet support Padlock hardware key generation.  */
 
-  if (0)
+  if (hw_setkey)
     {
-      ;
+      hw_setkey (ctx, key);
     }
-#ifdef USE_AESNI
-  else if (ctx->use_aesni)
-    _gcry_aes_aesni_do_setkey (ctx, key);
-#endif
-#ifdef USE_SSSE3
-  else if (ctx->use_ssse3)
-    _gcry_aes_ssse3_do_setkey (ctx, key);
-#endif
-#ifdef USE_ARM_CE
-  else if (ctx->use_arm_ce)
-    _gcry_aes_armv8_ce_setkey (ctx, key);
-#endif
-#ifdef USE_PPC_CRYPTO
-  else if (ctx->use_ppc_crypto)
-    _gcry_aes_ppc8_setkey (ctx, key);
-#endif
   else
     {
       const byte *sbox = ((const byte *)encT) + 1;
-- 
2.25.1



From jussi.kivilinna at iki.fi  Sun Sep 27 20:08:27 2020
From: jussi.kivilinna at iki.fi (Jussi Kivilinna)
Date: Sun, 27 Sep 2020 21:08:27 +0300
Subject: [PATCH 3/4] rijndael: clean-up generic bulk functions
In-Reply-To: <20200927180828.744564-1-jussi.kivilinna@iki.fi>
References: <20200927180828.744564-1-jussi.kivilinna@iki.fi>
Message-ID: <20200927180828.744564-3-jussi.kivilinna@iki.fi>

* cipher/rijndael.c (_gcry_aes_cfb_enc, _gcry_aes_cbc_enc)
(_gcry_aes_ctr_enc, _gcry_aes_cfb_dec, _gcry_aes_cbc_dec)
(_gcry_aes_ocb_crypt, _gcry_aes_ocb_auth, _gcry_aes_xts_crypt): Remove
calls to hardware accelerated AES bulk functions.
--

Patch removes accelerated implementation selection from generic
bulk functions. These are no longer used/needed as setkey
returns accelerated bulk functions for all callers and all
callers have been updated to used those bulk functions.

Signed-off-by: Jussi Kivilinna <jussi.kivilinna at iki.fi>
---
 cipher/rijndael.c | 542 ++++++++++------------------------------------
 1 file changed, 118 insertions(+), 424 deletions(-)

diff --git a/cipher/rijndael.c b/cipher/rijndael.c
index ac70ba75..bf9c0050 100644
--- a/cipher/rijndael.c
+++ b/cipher/rijndael.c
@@ -974,60 +974,19 @@ _gcry_aes_cfb_enc (void *context, unsigned char *iv,
   unsigned char *outbuf = outbuf_arg;
   const unsigned char *inbuf = inbuf_arg;
   unsigned int burn_depth = 0;
+  rijndael_cryptfn_t encrypt_fn = ctx->encrypt_fn;
 
-  if (0)
-    ;
-#ifdef USE_AESNI
-  else if (ctx->use_aesni)
-    {
-      _gcry_aes_aesni_cfb_enc (ctx, iv, outbuf, inbuf, nblocks);
-      return;
-    }
-#endif /*USE_AESNI*/
-#ifdef USE_SSSE3
-  else if (ctx->use_ssse3)
-    {
-      _gcry_aes_ssse3_cfb_enc (ctx, iv, outbuf, inbuf, nblocks);
-      return;
-    }
-#endif /*USE_SSSE3*/
-#ifdef USE_ARM_CE
-  else if (ctx->use_arm_ce)
-    {
-      _gcry_aes_armv8_ce_cfb_enc (ctx, iv, outbuf, inbuf, nblocks);
-      return;
-    }
-#endif /*USE_ARM_CE*/
-#ifdef USE_PPC_CRYPTO_WITH_PPC9LE
-  else if (ctx->use_ppc9le_crypto)
-    {
-      _gcry_aes_ppc9le_cfb_enc (ctx, iv, outbuf, inbuf, nblocks);
-      return;
-    }
-#endif /*USE_PPC_CRYPTO_WITH_PPC9LE*/
-#ifdef USE_PPC_CRYPTO
-  else if (ctx->use_ppc_crypto)
-    {
-      _gcry_aes_ppc8_cfb_enc (ctx, iv, outbuf, inbuf, nblocks);
-      return;
-    }
-#endif /*USE_PPC_CRYPTO*/
-  else
-    {
-      rijndael_cryptfn_t encrypt_fn = ctx->encrypt_fn;
-
-      if (ctx->prefetch_enc_fn)
-        ctx->prefetch_enc_fn();
+  if (ctx->prefetch_enc_fn)
+    ctx->prefetch_enc_fn();
 
-      for ( ;nblocks; nblocks-- )
-        {
-          /* Encrypt the IV. */
-          burn_depth = encrypt_fn (ctx, iv, iv);
-          /* XOR the input with the IV and store input into IV.  */
-          cipher_block_xor_2dst(outbuf, iv, inbuf, BLOCKSIZE);
-          outbuf += BLOCKSIZE;
-          inbuf  += BLOCKSIZE;
-        }
+  for ( ;nblocks; nblocks-- )
+    {
+      /* Encrypt the IV. */
+      burn_depth = encrypt_fn (ctx, iv, iv);
+      /* XOR the input with the IV and store input into IV.  */
+      cipher_block_xor_2dst(outbuf, iv, inbuf, BLOCKSIZE);
+      outbuf += BLOCKSIZE;
+      inbuf  += BLOCKSIZE;
     }
 
   if (burn_depth)
@@ -1049,69 +1008,28 @@ _gcry_aes_cbc_enc (void *context, unsigned char *iv,
   const unsigned char *inbuf = inbuf_arg;
   unsigned char *last_iv;
   unsigned int burn_depth = 0;
+  rijndael_cryptfn_t encrypt_fn = ctx->encrypt_fn;
 
-  if (0)
-    ;
-#ifdef USE_AESNI
-  else if (ctx->use_aesni)
-    {
-      _gcry_aes_aesni_cbc_enc (ctx, iv, outbuf, inbuf, nblocks, cbc_mac);
-      return;
-    }
-#endif /*USE_AESNI*/
-#ifdef USE_SSSE3
-  else if (ctx->use_ssse3)
-    {
-      _gcry_aes_ssse3_cbc_enc (ctx, iv, outbuf, inbuf, nblocks, cbc_mac);
-      return;
-    }
-#endif /*USE_SSSE3*/
-#ifdef USE_ARM_CE
-  else if (ctx->use_arm_ce)
-    {
-      _gcry_aes_armv8_ce_cbc_enc (ctx, iv, outbuf, inbuf, nblocks, cbc_mac);
-      return;
-    }
-#endif /*USE_ARM_CE*/
-#ifdef USE_PPC_CRYPTO_WITH_PPC9LE
-  else if (ctx->use_ppc9le_crypto)
-    {
-      _gcry_aes_ppc9le_cbc_enc (ctx, iv, outbuf, inbuf, nblocks, cbc_mac);
-      return;
-    }
-#endif /*USE_PPC_CRYPTO_WITH_PPC9LE*/
-#ifdef USE_PPC_CRYPTO
-  else if (ctx->use_ppc_crypto)
-    {
-      _gcry_aes_ppc8_cbc_enc (ctx, iv, outbuf, inbuf, nblocks, cbc_mac);
-      return;
-    }
-#endif /*USE_PPC_CRYPTO*/
-  else
-    {
-      rijndael_cryptfn_t encrypt_fn = ctx->encrypt_fn;
-
-      if (ctx->prefetch_enc_fn)
-        ctx->prefetch_enc_fn();
-
-      last_iv = iv;
+  if (ctx->prefetch_enc_fn)
+    ctx->prefetch_enc_fn();
 
-      for ( ;nblocks; nblocks-- )
-        {
-          cipher_block_xor(outbuf, inbuf, last_iv, BLOCKSIZE);
+  last_iv = iv;
 
-          burn_depth = encrypt_fn (ctx, outbuf, outbuf);
+  for ( ;nblocks; nblocks-- )
+    {
+      cipher_block_xor(outbuf, inbuf, last_iv, BLOCKSIZE);
 
-          last_iv = outbuf;
-          inbuf += BLOCKSIZE;
-          if (!cbc_mac)
-            outbuf += BLOCKSIZE;
-        }
+      burn_depth = encrypt_fn (ctx, outbuf, outbuf);
 
-      if (last_iv != iv)
-        cipher_block_cpy (iv, last_iv, BLOCKSIZE);
+      last_iv = outbuf;
+      inbuf += BLOCKSIZE;
+      if (!cbc_mac)
+	outbuf += BLOCKSIZE;
     }
 
+  if (last_iv != iv)
+    cipher_block_cpy (iv, last_iv, BLOCKSIZE);
+
   if (burn_depth)
     _gcry_burn_stack (burn_depth + 4 * sizeof(void *));
 }
@@ -1131,66 +1049,25 @@ _gcry_aes_ctr_enc (void *context, unsigned char *ctr,
   unsigned char *outbuf = outbuf_arg;
   const unsigned char *inbuf = inbuf_arg;
   unsigned int burn_depth = 0;
+  union { unsigned char x1[16] ATTR_ALIGNED_16; u32 x32[4]; } tmp;
+  rijndael_cryptfn_t encrypt_fn = ctx->encrypt_fn;
 
-  if (0)
-    ;
-#ifdef USE_AESNI
-  else if (ctx->use_aesni)
-    {
-      _gcry_aes_aesni_ctr_enc (ctx, ctr, outbuf, inbuf, nblocks);
-      return;
-    }
-#endif /*USE_AESNI*/
-#ifdef USE_SSSE3
-  else if (ctx->use_ssse3)
-    {
-      _gcry_aes_ssse3_ctr_enc (ctx, ctr, outbuf, inbuf, nblocks);
-      return;
-    }
-#endif /*USE_SSSE3*/
-#ifdef USE_ARM_CE
-  else if (ctx->use_arm_ce)
-    {
-      _gcry_aes_armv8_ce_ctr_enc (ctx, ctr, outbuf, inbuf, nblocks);
-      return;
-    }
-#endif /*USE_ARM_CE*/
-#ifdef USE_PPC_CRYPTO_WITH_PPC9LE
-  else if (ctx->use_ppc9le_crypto)
-    {
-      _gcry_aes_ppc9le_ctr_enc (ctx, ctr, outbuf, inbuf, nblocks);
-      return;
-    }
-#endif /*USE_PPC_CRYPTO_WITH_PPC9LE*/
-#ifdef USE_PPC_CRYPTO
-  else if (ctx->use_ppc_crypto)
+  if (ctx->prefetch_enc_fn)
+    ctx->prefetch_enc_fn();
+
+  for ( ;nblocks; nblocks-- )
     {
-      _gcry_aes_ppc8_ctr_enc (ctx, ctr, outbuf, inbuf, nblocks);
-      return;
+      /* Encrypt the counter. */
+      burn_depth = encrypt_fn (ctx, tmp.x1, ctr);
+      /* XOR the input with the encrypted counter and store in output.  */
+      cipher_block_xor(outbuf, tmp.x1, inbuf, BLOCKSIZE);
+      outbuf += BLOCKSIZE;
+      inbuf  += BLOCKSIZE;
+      /* Increment the counter.  */
+      cipher_block_add(ctr, 1, BLOCKSIZE);
     }
-#endif /*USE_PPC_CRYPTO*/
-  else
-    {
-      union { unsigned char x1[16] ATTR_ALIGNED_16; u32 x32[4]; } tmp;
-      rijndael_cryptfn_t encrypt_fn = ctx->encrypt_fn;
-
-      if (ctx->prefetch_enc_fn)
-        ctx->prefetch_enc_fn();
-
-      for ( ;nblocks; nblocks-- )
-        {
-          /* Encrypt the counter. */
-          burn_depth = encrypt_fn (ctx, tmp.x1, ctr);
-          /* XOR the input with the encrypted counter and store in output.  */
-          cipher_block_xor(outbuf, tmp.x1, inbuf, BLOCKSIZE);
-          outbuf += BLOCKSIZE;
-          inbuf  += BLOCKSIZE;
-          /* Increment the counter.  */
-	  cipher_block_add(ctr, 1, BLOCKSIZE);
-        }
 
-      wipememory(&tmp, sizeof(tmp));
-    }
+  wipememory(&tmp, sizeof(tmp));
 
   if (burn_depth)
     _gcry_burn_stack (burn_depth + 4 * sizeof(void *));
@@ -1388,58 +1265,17 @@ _gcry_aes_cfb_dec (void *context, unsigned char *iv,
   unsigned char *outbuf = outbuf_arg;
   const unsigned char *inbuf = inbuf_arg;
   unsigned int burn_depth = 0;
+  rijndael_cryptfn_t encrypt_fn = ctx->encrypt_fn;
 
-  if (0)
-    ;
-#ifdef USE_AESNI
-  else if (ctx->use_aesni)
-    {
-      _gcry_aes_aesni_cfb_dec (ctx, iv, outbuf, inbuf, nblocks);
-      return;
-    }
-#endif /*USE_AESNI*/
-#ifdef USE_SSSE3
-  else if (ctx->use_ssse3)
-    {
-      _gcry_aes_ssse3_cfb_dec (ctx, iv, outbuf, inbuf, nblocks);
-      return;
-    }
-#endif /*USE_SSSE3*/
-#ifdef USE_ARM_CE
-  else if (ctx->use_arm_ce)
-    {
-      _gcry_aes_armv8_ce_cfb_dec (ctx, iv, outbuf, inbuf, nblocks);
-      return;
-    }
-#endif /*USE_ARM_CE*/
-#ifdef USE_PPC_CRYPTO_WITH_PPC9LE
-  else if (ctx->use_ppc9le_crypto)
-    {
-      _gcry_aes_ppc9le_cfb_dec (ctx, iv, outbuf, inbuf, nblocks);
-      return;
-    }
-#endif /*USE_PPC_CRYPTO_WITH_PPC9LE*/
-#ifdef USE_PPC_CRYPTO
-  else if (ctx->use_ppc_crypto)
-    {
-      _gcry_aes_ppc8_cfb_dec (ctx, iv, outbuf, inbuf, nblocks);
-      return;
-    }
-#endif /*USE_PPC_CRYPTO*/
-  else
-    {
-      rijndael_cryptfn_t encrypt_fn = ctx->encrypt_fn;
-
-      if (ctx->prefetch_enc_fn)
-        ctx->prefetch_enc_fn();
+  if (ctx->prefetch_enc_fn)
+    ctx->prefetch_enc_fn();
 
-      for ( ;nblocks; nblocks-- )
-        {
-          burn_depth = encrypt_fn (ctx, iv, iv);
-          cipher_block_xor_n_copy(outbuf, iv, inbuf, BLOCKSIZE);
-          outbuf += BLOCKSIZE;
-          inbuf  += BLOCKSIZE;
-        }
+  for ( ;nblocks; nblocks-- )
+    {
+      burn_depth = encrypt_fn (ctx, iv, iv);
+      cipher_block_xor_n_copy(outbuf, iv, inbuf, BLOCKSIZE);
+      outbuf += BLOCKSIZE;
+      inbuf  += BLOCKSIZE;
     }
 
   if (burn_depth)
@@ -1460,69 +1296,28 @@ _gcry_aes_cbc_dec (void *context, unsigned char *iv,
   unsigned char *outbuf = outbuf_arg;
   const unsigned char *inbuf = inbuf_arg;
   unsigned int burn_depth = 0;
+  unsigned char savebuf[BLOCKSIZE] ATTR_ALIGNED_16;
+  rijndael_cryptfn_t decrypt_fn = ctx->decrypt_fn;
 
-  if (0)
-    ;
-#ifdef USE_AESNI
-  else if (ctx->use_aesni)
-    {
-      _gcry_aes_aesni_cbc_dec (ctx, iv, outbuf, inbuf, nblocks);
-      return;
-    }
-#endif /*USE_AESNI*/
-#ifdef USE_SSSE3
-  else if (ctx->use_ssse3)
-    {
-      _gcry_aes_ssse3_cbc_dec (ctx, iv, outbuf, inbuf, nblocks);
-      return;
-    }
-#endif /*USE_SSSE3*/
-#ifdef USE_ARM_CE
-  else if (ctx->use_arm_ce)
-    {
-      _gcry_aes_armv8_ce_cbc_dec (ctx, iv, outbuf, inbuf, nblocks);
-      return;
-    }
-#endif /*USE_ARM_CE*/
-#ifdef USE_PPC_CRYPTO_WITH_PPC9LE
-  else if (ctx->use_ppc9le_crypto)
-    {
-      _gcry_aes_ppc9le_cbc_dec (ctx, iv, outbuf, inbuf, nblocks);
-      return;
-    }
-#endif /*USE_PPC_CRYPTO_WITH_PPC9LE*/
-#ifdef USE_PPC_CRYPTO
-  else if (ctx->use_ppc_crypto)
-    {
-      _gcry_aes_ppc8_cbc_dec (ctx, iv, outbuf, inbuf, nblocks);
-      return;
-    }
-#endif /*USE_PPC_CRYPTO*/
-  else
-    {
-      unsigned char savebuf[BLOCKSIZE] ATTR_ALIGNED_16;
-      rijndael_cryptfn_t decrypt_fn = ctx->decrypt_fn;
-
-      check_decryption_preparation (ctx);
-
-      if (ctx->prefetch_dec_fn)
-        ctx->prefetch_dec_fn();
+  check_decryption_preparation (ctx);
 
-      for ( ;nblocks; nblocks-- )
-        {
-          /* INBUF is needed later and it may be identical to OUTBUF, so store
-             the intermediate result to SAVEBUF.  */
+  if (ctx->prefetch_dec_fn)
+    ctx->prefetch_dec_fn();
 
-          burn_depth = decrypt_fn (ctx, savebuf, inbuf);
+  for ( ;nblocks; nblocks-- )
+    {
+      /* INBUF is needed later and it may be identical to OUTBUF, so store
+	  the intermediate result to SAVEBUF.  */
 
-          cipher_block_xor_n_copy_2(outbuf, savebuf, iv, inbuf, BLOCKSIZE);
-          inbuf += BLOCKSIZE;
-          outbuf += BLOCKSIZE;
-        }
+      burn_depth = decrypt_fn (ctx, savebuf, inbuf);
 
-      wipememory(savebuf, sizeof(savebuf));
+      cipher_block_xor_n_copy_2(outbuf, savebuf, iv, inbuf, BLOCKSIZE);
+      inbuf += BLOCKSIZE;
+      outbuf += BLOCKSIZE;
     }
 
+  wipememory(savebuf, sizeof(savebuf));
+
   if (burn_depth)
     _gcry_burn_stack (burn_depth + 4 * sizeof(void *));
 }
@@ -1539,39 +1334,7 @@ _gcry_aes_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg,
   const unsigned char *inbuf = inbuf_arg;
   unsigned int burn_depth = 0;
 
-  if (0)
-    ;
-#ifdef USE_AESNI
-  else if (ctx->use_aesni)
-    {
-      return _gcry_aes_aesni_ocb_crypt (c, outbuf, inbuf, nblocks, encrypt);
-    }
-#endif /*USE_AESNI*/
-#ifdef USE_SSSE3
-  else if (ctx->use_ssse3)
-    {
-      return _gcry_aes_ssse3_ocb_crypt (c, outbuf, inbuf, nblocks, encrypt);
-    }
-#endif /*USE_SSSE3*/
-#ifdef USE_ARM_CE
-  else if (ctx->use_arm_ce)
-    {
-      return _gcry_aes_armv8_ce_ocb_crypt (c, outbuf, inbuf, nblocks, encrypt);
-    }
-#endif /*USE_ARM_CE*/
-#ifdef USE_PPC_CRYPTO_WITH_PPC9LE
-  else if (ctx->use_ppc9le_crypto)
-    {
-      return _gcry_aes_ppc9le_ocb_crypt (c, outbuf, inbuf, nblocks, encrypt);
-    }
-#endif /*USE_PPC_CRYPTO_WITH_PPC9LE*/
-#ifdef USE_PPC_CRYPTO
-  else if (ctx->use_ppc_crypto)
-    {
-      return _gcry_aes_ppc8_ocb_crypt (c, outbuf, inbuf, nblocks, encrypt);
-    }
-#endif /*USE_PPC_CRYPTO*/
-  else if (encrypt)
+  if (encrypt)
     {
       union { unsigned char x1[16] ATTR_ALIGNED_16; u32 x32[4]; } l_tmp;
       rijndael_cryptfn_t encrypt_fn = ctx->encrypt_fn;
@@ -1644,66 +1407,30 @@ _gcry_aes_ocb_auth (gcry_cipher_hd_t c, const void *abuf_arg, size_t nblocks)
   RIJNDAEL_context *ctx = (void *)&c->context.c;
   const unsigned char *abuf = abuf_arg;
   unsigned int burn_depth = 0;
+  union { unsigned char x1[16] ATTR_ALIGNED_16; u32 x32[4]; } l_tmp;
+  rijndael_cryptfn_t encrypt_fn = ctx->encrypt_fn;
 
-  if (0)
-    ;
-#ifdef USE_AESNI
-  else if (ctx->use_aesni)
-    {
-      return _gcry_aes_aesni_ocb_auth (c, abuf, nblocks);
-    }
-#endif /*USE_AESNI*/
-#ifdef USE_SSSE3
-  else if (ctx->use_ssse3)
-    {
-      return _gcry_aes_ssse3_ocb_auth (c, abuf, nblocks);
-    }
-#endif /*USE_SSSE3*/
-#ifdef USE_ARM_CE
-  else if (ctx->use_arm_ce)
-    {
-      return _gcry_aes_armv8_ce_ocb_auth (c, abuf, nblocks);
-    }
-#endif /*USE_ARM_CE*/
-#ifdef USE_PPC_CRYPTO_WITH_PPC9LE
-  else if (ctx->use_ppc9le_crypto)
-    {
-      return _gcry_aes_ppc9le_ocb_auth (c, abuf, nblocks);
-    }
-#endif /*USE_PPC_CRYPTO_WITH_PPC9LE*/
-#ifdef USE_PPC_CRYPTO
-  else if (ctx->use_ppc_crypto)
-    {
-      return _gcry_aes_ppc8_ocb_auth (c, abuf, nblocks);
-    }
-#endif /*USE_PPC_CRYPTO*/
-  else
-    {
-      union { unsigned char x1[16] ATTR_ALIGNED_16; u32 x32[4]; } l_tmp;
-      rijndael_cryptfn_t encrypt_fn = ctx->encrypt_fn;
-
-      if (ctx->prefetch_enc_fn)
-        ctx->prefetch_enc_fn();
-
-      for ( ;nblocks; nblocks-- )
-        {
-          u64 i = ++c->u_mode.ocb.aad_nblocks;
-          const unsigned char *l = ocb_get_l(c, i);
+  if (ctx->prefetch_enc_fn)
+    ctx->prefetch_enc_fn();
 
-          /* Offset_i = Offset_{i-1} xor L_{ntz(i)} */
-          cipher_block_xor_1 (c->u_mode.ocb.aad_offset, l, BLOCKSIZE);
-          /* Sum_i = Sum_{i-1} xor ENCIPHER(K, A_i xor Offset_i)  */
-          cipher_block_xor (l_tmp.x1, c->u_mode.ocb.aad_offset, abuf,
-                            BLOCKSIZE);
-          burn_depth = encrypt_fn (ctx, l_tmp.x1, l_tmp.x1);
-          cipher_block_xor_1 (c->u_mode.ocb.aad_sum, l_tmp.x1, BLOCKSIZE);
+  for ( ;nblocks; nblocks-- )
+    {
+      u64 i = ++c->u_mode.ocb.aad_nblocks;
+      const unsigned char *l = ocb_get_l(c, i);
 
-          abuf += BLOCKSIZE;
-        }
+      /* Offset_i = Offset_{i-1} xor L_{ntz(i)} */
+      cipher_block_xor_1 (c->u_mode.ocb.aad_offset, l, BLOCKSIZE);
+      /* Sum_i = Sum_{i-1} xor ENCIPHER(K, A_i xor Offset_i)  */
+      cipher_block_xor (l_tmp.x1, c->u_mode.ocb.aad_offset, abuf,
+			BLOCKSIZE);
+      burn_depth = encrypt_fn (ctx, l_tmp.x1, l_tmp.x1);
+      cipher_block_xor_1 (c->u_mode.ocb.aad_sum, l_tmp.x1, BLOCKSIZE);
 
-      wipememory(&l_tmp, sizeof(l_tmp));
+      abuf += BLOCKSIZE;
     }
 
+  wipememory(&l_tmp, sizeof(l_tmp));
+
   if (burn_depth)
     _gcry_burn_stack (burn_depth + 4 * sizeof(void *));
 
@@ -1724,89 +1451,56 @@ _gcry_aes_xts_crypt (void *context, unsigned char *tweak,
   rijndael_cryptfn_t crypt_fn;
   u64 tweak_lo, tweak_hi, tweak_next_lo, tweak_next_hi, tmp_lo, tmp_hi, carry;
 
-  if (0)
-    ;
-#ifdef USE_AESNI
-  else if (ctx->use_aesni)
-    {
-      _gcry_aes_aesni_xts_crypt (ctx, tweak, outbuf, inbuf, nblocks, encrypt);
-      return;
-    }
-#endif /*USE_AESNI*/
-#ifdef USE_ARM_CE
-  else if (ctx->use_arm_ce)
-    {
-      _gcry_aes_armv8_ce_xts_crypt (ctx, tweak, outbuf, inbuf, nblocks, encrypt);
-      return;
-    }
-#endif /*USE_ARM_CE*/
-#ifdef USE_PPC_CRYPTO_WITH_PPC9LE
-  else if (ctx->use_ppc9le_crypto)
+  if (encrypt)
     {
-      _gcry_aes_ppc9le_xts_crypt (ctx, tweak, outbuf, inbuf, nblocks, encrypt);
-      return;
-    }
-#endif /*USE_PPC_CRYPTO_WITH_PPC9LE*/
-#ifdef USE_PPC_CRYPTO
-  else if (ctx->use_ppc_crypto)
-    {
-      _gcry_aes_ppc8_xts_crypt (ctx, tweak, outbuf, inbuf, nblocks, encrypt);
-      return;
+      if (ctx->prefetch_enc_fn)
+	ctx->prefetch_enc_fn();
+
+      crypt_fn = ctx->encrypt_fn;
     }
-#endif /*USE_PPC_CRYPTO*/
   else
     {
-      if (encrypt)
-        {
-          if (ctx->prefetch_enc_fn)
-            ctx->prefetch_enc_fn();
-
-          crypt_fn = ctx->encrypt_fn;
-        }
-      else
-        {
-          check_decryption_preparation (ctx);
-
-          if (ctx->prefetch_dec_fn)
-            ctx->prefetch_dec_fn();
+      check_decryption_preparation (ctx);
 
-          crypt_fn = ctx->decrypt_fn;
-        }
+      if (ctx->prefetch_dec_fn)
+	ctx->prefetch_dec_fn();
 
-      tweak_next_lo = buf_get_le64 (tweak + 0);
-      tweak_next_hi = buf_get_le64 (tweak + 8);
+      crypt_fn = ctx->decrypt_fn;
+    }
 
-      while (nblocks)
-	{
-	  tweak_lo = tweak_next_lo;
-	  tweak_hi = tweak_next_hi;
+  tweak_next_lo = buf_get_le64 (tweak + 0);
+  tweak_next_hi = buf_get_le64 (tweak + 8);
 
-	  /* Xor-Encrypt/Decrypt-Xor block. */
-	  tmp_lo = buf_get_le64 (inbuf + 0) ^ tweak_lo;
-	  tmp_hi = buf_get_le64 (inbuf + 8) ^ tweak_hi;
+  while (nblocks)
+    {
+      tweak_lo = tweak_next_lo;
+      tweak_hi = tweak_next_hi;
 
-	  buf_put_le64 (outbuf + 0, tmp_lo);
-	  buf_put_le64 (outbuf + 8, tmp_hi);
+      /* Xor-Encrypt/Decrypt-Xor block. */
+      tmp_lo = buf_get_le64 (inbuf + 0) ^ tweak_lo;
+      tmp_hi = buf_get_le64 (inbuf + 8) ^ tweak_hi;
 
-	  /* Generate next tweak. */
-	  carry = -(tweak_next_hi >> 63) & 0x87;
-	  tweak_next_hi = (tweak_next_hi << 1) + (tweak_next_lo >> 63);
-	  tweak_next_lo = (tweak_next_lo << 1) ^ carry;
+      buf_put_le64 (outbuf + 0, tmp_lo);
+      buf_put_le64 (outbuf + 8, tmp_hi);
 
-	  burn_depth = crypt_fn (ctx, outbuf, outbuf);
+      /* Generate next tweak. */
+      carry = -(tweak_next_hi >> 63) & 0x87;
+      tweak_next_hi = (tweak_next_hi << 1) + (tweak_next_lo >> 63);
+      tweak_next_lo = (tweak_next_lo << 1) ^ carry;
 
-	  buf_put_le64 (outbuf + 0, buf_get_le64 (outbuf + 0) ^ tweak_lo);
-	  buf_put_le64 (outbuf + 8, buf_get_le64 (outbuf + 8) ^ tweak_hi);
+      burn_depth = crypt_fn (ctx, outbuf, outbuf);
 
-	  outbuf += GCRY_XTS_BLOCK_LEN;
-	  inbuf += GCRY_XTS_BLOCK_LEN;
-	  nblocks--;
-	}
+      buf_put_le64 (outbuf + 0, buf_get_le64 (outbuf + 0) ^ tweak_lo);
+      buf_put_le64 (outbuf + 8, buf_get_le64 (outbuf + 8) ^ tweak_hi);
 
-      buf_put_le64 (tweak + 0, tweak_next_lo);
-      buf_put_le64 (tweak + 8, tweak_next_hi);
+      outbuf += GCRY_XTS_BLOCK_LEN;
+      inbuf += GCRY_XTS_BLOCK_LEN;
+      nblocks--;
     }
 
+  buf_put_le64 (tweak + 0, tweak_next_lo);
+  buf_put_le64 (tweak + 8, tweak_next_hi);
+
   if (burn_depth)
     _gcry_burn_stack (burn_depth + 5 * sizeof(void *));
 }
-- 
2.25.1



From jussi.kivilinna at iki.fi  Sun Sep 27 20:08:28 2020
From: jussi.kivilinna at iki.fi (Jussi Kivilinna)
Date: Sun, 27 Sep 2020 21:08:28 +0300
Subject: [PATCH 4/4] rijndael: clean-up prepare_decryption function
In-Reply-To: <20200927180828.744564-1-jussi.kivilinna@iki.fi>
References: <20200927180828.744564-1-jussi.kivilinna@iki.fi>
Message-ID: <20200927180828.744564-4-jussi.kivilinna@iki.fi>

* cipher/rijndael-internal.h (rijndael_prepare_decfn_t): New.
(RIJNDAEL_context_s): New member 'prepare_decryption'.
* cipher/rijndael-padlock.c (_gcry_aes_padlock_prepare_decryption): New.
* cipher/rijndael.c (_gcry_aes_padlock_prepare_decryption): New.
(do_setkey): Setup 'ctx->prepare_decryption' for each acceleration type.
(prepare_decryption): Remove calls to other prepare decryption functions.
(check_decryption_preparation): Call 'ctx->prepare_decryption' instead
of 'prepare_decryption'.
--

Signed-off-by: Jussi Kivilinna <jussi.kivilinna at iki.fi>
---
 cipher/rijndael-internal.h |   2 +
 cipher/rijndael-padlock.c  |   6 ++
 cipher/rijndael.c          | 135 ++++++++++++++-----------------------
 3 files changed, 59 insertions(+), 84 deletions(-)

diff --git a/cipher/rijndael-internal.h b/cipher/rijndael-internal.h
index bdd3bee1..92c47a9d 100644
--- a/cipher/rijndael-internal.h
+++ b/cipher/rijndael-internal.h
@@ -127,6 +127,7 @@ typedef unsigned int (*rijndael_cryptfn_t)(const struct RIJNDAEL_context_s *ctx,
                                            unsigned char *bx,
                                            const unsigned char *ax);
 typedef void (*rijndael_prefetchfn_t)(void);
+typedef void (*rijndael_prepare_decfn_t)(struct RIJNDAEL_context_s *ctx);
 
 /* Our context object.  */
 typedef struct RIJNDAEL_context_s
@@ -181,6 +182,7 @@ typedef struct RIJNDAEL_context_s
   rijndael_cryptfn_t decrypt_fn;
   rijndael_prefetchfn_t prefetch_enc_fn;
   rijndael_prefetchfn_t prefetch_dec_fn;
+  rijndael_prepare_decfn_t prepare_decryption;
 } RIJNDAEL_context ATTR_ALIGNED_16;
 
 /* Macros defining alias for the keyschedules.  */
diff --git a/cipher/rijndael-padlock.c b/cipher/rijndael-padlock.c
index 234751b9..3af214d7 100644
--- a/cipher/rijndael-padlock.c
+++ b/cipher/rijndael-padlock.c
@@ -101,4 +101,10 @@ _gcry_aes_padlock_decrypt (const RIJNDAEL_context *ctx,
   return do_padlock(ctx, bx, ax, 1);
 }
 
+void
+_gcry_aes_padlock_prepare_decryption (RIJNDAEL_context *ctx)
+{
+  /* Padlock does not need decryption subkeys. */
+  (void)ctx;
+}
 #endif /* USE_PADLOCK */
diff --git a/cipher/rijndael.c b/cipher/rijndael.c
index bf9c0050..60d16c68 100644
--- a/cipher/rijndael.c
+++ b/cipher/rijndael.c
@@ -143,6 +143,7 @@ extern unsigned int _gcry_aes_padlock_encrypt (const RIJNDAEL_context *ctx,
 extern unsigned int _gcry_aes_padlock_decrypt (const RIJNDAEL_context *ctx,
                                                unsigned char *bx,
                                                const unsigned char *ax);
+extern void _gcry_aes_padlock_prepare_decryption (RIJNDAEL_context *ctx);
 #endif
 
 #ifdef USE_ARM_ASM
@@ -314,6 +315,7 @@ static void _gcry_aes_xts_crypt (void *context, unsigned char *tweak,
 
 /* Function prototypes.  */
 static const char *selftest(void);
+static void prepare_decryption(RIJNDAEL_context *ctx);
 
 
 
@@ -465,6 +467,7 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen,
       ctx->decrypt_fn = _gcry_aes_aesni_decrypt;
       ctx->prefetch_enc_fn = NULL;
       ctx->prefetch_dec_fn = NULL;
+      ctx->prepare_decryption = _gcry_aes_aesni_prepare_decryption;
       ctx->use_aesni = 1;
       ctx->use_avx = !!(hwfeatures & HWF_INTEL_AVX);
       ctx->use_avx2 = !!(hwfeatures & HWF_INTEL_AVX2);
@@ -487,6 +490,7 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen,
       ctx->decrypt_fn = _gcry_aes_padlock_decrypt;
       ctx->prefetch_enc_fn = NULL;
       ctx->prefetch_dec_fn = NULL;
+      ctx->prepare_decryption = _gcry_aes_padlock_prepare_decryption;
       ctx->use_padlock = 1;
       memcpy (ctx->padlockkey, key, keylen);
     }
@@ -499,6 +503,7 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen,
       ctx->decrypt_fn = _gcry_aes_ssse3_decrypt;
       ctx->prefetch_enc_fn = NULL;
       ctx->prefetch_dec_fn = NULL;
+      ctx->prepare_decryption = _gcry_aes_ssse3_prepare_decryption;
       ctx->use_ssse3 = 1;
 
       /* Setup SSSE3 bulk encryption routines.  */
@@ -519,6 +524,7 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen,
       ctx->decrypt_fn = _gcry_aes_armv8_ce_decrypt;
       ctx->prefetch_enc_fn = NULL;
       ctx->prefetch_dec_fn = NULL;
+      ctx->prepare_decryption = _gcry_aes_armv8_ce_prepare_decryption;
       ctx->use_arm_ce = 1;
 
       /* Setup ARM-CE bulk encryption routines.  */
@@ -540,6 +546,7 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen,
       ctx->decrypt_fn = _gcry_aes_ppc9le_decrypt;
       ctx->prefetch_enc_fn = NULL;
       ctx->prefetch_dec_fn = NULL;
+      ctx->prepare_decryption = _gcry_aes_ppc8_prepare_decryption;
       ctx->use_ppc_crypto = 1; /* same key-setup as USE_PPC_CRYPTO */
       ctx->use_ppc9le_crypto = 1;
 
@@ -562,6 +569,7 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen,
       ctx->decrypt_fn = _gcry_aes_ppc8_decrypt;
       ctx->prefetch_enc_fn = NULL;
       ctx->prefetch_dec_fn = NULL;
+      ctx->prepare_decryption = _gcry_aes_ppc8_prepare_decryption;
       ctx->use_ppc_crypto = 1;
 
       /* Setup PPC8 bulk encryption routines.  */
@@ -581,6 +589,7 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen,
       ctx->decrypt_fn = do_decrypt;
       ctx->prefetch_enc_fn = prefetch_enc;
       ctx->prefetch_dec_fn = prefetch_dec;
+      ctx->prepare_decryption = prepare_decryption;
     }
 
   /* NB: We don't yet support Padlock hardware key generation.  */
@@ -705,94 +714,52 @@ rijndael_setkey (void *context, const byte *key, const unsigned keylen,
 static void
 prepare_decryption( RIJNDAEL_context *ctx )
 {
+  const byte *sbox = ((const byte *)encT) + 1;
   int r;
 
-  if (0)
-    ;
-#ifdef USE_AESNI
-  else if (ctx->use_aesni)
-    {
-      _gcry_aes_aesni_prepare_decryption (ctx);
-    }
-#endif /*USE_AESNI*/
-#ifdef USE_SSSE3
-  else if (ctx->use_ssse3)
-    {
-      _gcry_aes_ssse3_prepare_decryption (ctx);
-    }
-#endif /*USE_SSSE3*/
-#ifdef USE_ARM_CE
-  else if (ctx->use_arm_ce)
-    {
-      _gcry_aes_armv8_ce_prepare_decryption (ctx);
-    }
-#endif /*USE_ARM_CE*/
-#ifdef USE_ARM_CE
-  else if (ctx->use_arm_ce)
-    {
-      _gcry_aes_armv8_ce_prepare_decryption (ctx);
-    }
-#endif /*USE_ARM_CE*/
-#ifdef USE_PPC_CRYPTO
-  else if (ctx->use_ppc_crypto)
-    {
-      _gcry_aes_ppc8_prepare_decryption (ctx);
-    }
-#endif
-#ifdef USE_PADLOCK
-  else if (ctx->use_padlock)
-    {
-      /* Padlock does not need decryption subkeys. */
-    }
-#endif /*USE_PADLOCK*/
-  else
-    {
-      const byte *sbox = ((const byte *)encT) + 1;
+  prefetch_enc();
+  prefetch_dec();
 
-      prefetch_enc();
-      prefetch_dec();
-
-      ctx->keyschdec32[0][0] = ctx->keyschenc32[0][0];
-      ctx->keyschdec32[0][1] = ctx->keyschenc32[0][1];
-      ctx->keyschdec32[0][2] = ctx->keyschenc32[0][2];
-      ctx->keyschdec32[0][3] = ctx->keyschenc32[0][3];
+  ctx->keyschdec32[0][0] = ctx->keyschenc32[0][0];
+  ctx->keyschdec32[0][1] = ctx->keyschenc32[0][1];
+  ctx->keyschdec32[0][2] = ctx->keyschenc32[0][2];
+  ctx->keyschdec32[0][3] = ctx->keyschenc32[0][3];
 
-      for (r = 1; r < ctx->rounds; r++)
-        {
-          u32 *wi = ctx->keyschenc32[r];
-          u32 *wo = ctx->keyschdec32[r];
-          u32 wt;
-
-          wt = wi[0];
-          wo[0] = rol(decT[sbox[(byte)(wt >> 0) * 4]], 8 * 0)
-                 ^ rol(decT[sbox[(byte)(wt >> 8) * 4]], 8 * 1)
-                 ^ rol(decT[sbox[(byte)(wt >> 16) * 4]], 8 * 2)
-                 ^ rol(decT[sbox[(byte)(wt >> 24) * 4]], 8 * 3);
-
-          wt = wi[1];
-          wo[1] = rol(decT[sbox[(byte)(wt >> 0) * 4]], 8 * 0)
-                 ^ rol(decT[sbox[(byte)(wt >> 8) * 4]], 8 * 1)
-                 ^ rol(decT[sbox[(byte)(wt >> 16) * 4]], 8 * 2)
-                 ^ rol(decT[sbox[(byte)(wt >> 24) * 4]], 8 * 3);
-
-          wt = wi[2];
-          wo[2] = rol(decT[sbox[(byte)(wt >> 0) * 4]], 8 * 0)
-                 ^ rol(decT[sbox[(byte)(wt >> 8) * 4]], 8 * 1)
-                 ^ rol(decT[sbox[(byte)(wt >> 16) * 4]], 8 * 2)
-                 ^ rol(decT[sbox[(byte)(wt >> 24) * 4]], 8 * 3);
-
-          wt = wi[3];
-          wo[3] = rol(decT[sbox[(byte)(wt >> 0) * 4]], 8 * 0)
-                 ^ rol(decT[sbox[(byte)(wt >> 8) * 4]], 8 * 1)
-                 ^ rol(decT[sbox[(byte)(wt >> 16) * 4]], 8 * 2)
-                 ^ rol(decT[sbox[(byte)(wt >> 24) * 4]], 8 * 3);
-        }
-
-      ctx->keyschdec32[r][0] = ctx->keyschenc32[r][0];
-      ctx->keyschdec32[r][1] = ctx->keyschenc32[r][1];
-      ctx->keyschdec32[r][2] = ctx->keyschenc32[r][2];
-      ctx->keyschdec32[r][3] = ctx->keyschenc32[r][3];
+  for (r = 1; r < ctx->rounds; r++)
+    {
+      u32 *wi = ctx->keyschenc32[r];
+      u32 *wo = ctx->keyschdec32[r];
+      u32 wt;
+
+      wt = wi[0];
+      wo[0] = rol(decT[sbox[(byte)(wt >> 0) * 4]], 8 * 0)
+	      ^ rol(decT[sbox[(byte)(wt >> 8) * 4]], 8 * 1)
+	      ^ rol(decT[sbox[(byte)(wt >> 16) * 4]], 8 * 2)
+	      ^ rol(decT[sbox[(byte)(wt >> 24) * 4]], 8 * 3);
+
+      wt = wi[1];
+      wo[1] = rol(decT[sbox[(byte)(wt >> 0) * 4]], 8 * 0)
+	      ^ rol(decT[sbox[(byte)(wt >> 8) * 4]], 8 * 1)
+	      ^ rol(decT[sbox[(byte)(wt >> 16) * 4]], 8 * 2)
+	      ^ rol(decT[sbox[(byte)(wt >> 24) * 4]], 8 * 3);
+
+      wt = wi[2];
+      wo[2] = rol(decT[sbox[(byte)(wt >> 0) * 4]], 8 * 0)
+	      ^ rol(decT[sbox[(byte)(wt >> 8) * 4]], 8 * 1)
+	      ^ rol(decT[sbox[(byte)(wt >> 16) * 4]], 8 * 2)
+	      ^ rol(decT[sbox[(byte)(wt >> 24) * 4]], 8 * 3);
+
+      wt = wi[3];
+      wo[3] = rol(decT[sbox[(byte)(wt >> 0) * 4]], 8 * 0)
+	      ^ rol(decT[sbox[(byte)(wt >> 8) * 4]], 8 * 1)
+	      ^ rol(decT[sbox[(byte)(wt >> 16) * 4]], 8 * 2)
+	      ^ rol(decT[sbox[(byte)(wt >> 24) * 4]], 8 * 3);
     }
+
+  ctx->keyschdec32[r][0] = ctx->keyschenc32[r][0];
+  ctx->keyschdec32[r][1] = ctx->keyschenc32[r][1];
+  ctx->keyschdec32[r][2] = ctx->keyschenc32[r][2];
+  ctx->keyschdec32[r][3] = ctx->keyschenc32[r][3];
 }
 
 
@@ -1232,7 +1199,7 @@ check_decryption_preparation (RIJNDAEL_context *ctx)
 {
   if ( !ctx->decryption_prepared )
     {
-      prepare_decryption ( ctx );
+      ctx->prepare_decryption ( ctx );
       ctx->decryption_prepared = 1;
     }
 }
-- 
2.25.1



From jussi.kivilinna at iki.fi  Sun Sep 27 20:08:26 2020
From: jussi.kivilinna at iki.fi (Jussi Kivilinna)
Date: Sun, 27 Sep 2020 21:08:26 +0300
Subject: [PATCH 2/4] cipher: setup bulk functions at each algorithms key setup
In-Reply-To: <20200927180828.744564-1-jussi.kivilinna@iki.fi>
References: <20200927180828.744564-1-jussi.kivilinna@iki.fi>
Message-ID: <20200927180828.744564-2-jussi.kivilinna@iki.fi>

* cipher/cipher-internal.h (cipher_mode_ops_t, cipher_bulk_ops_t): New.
(gcry_cipher_handle): Define members 'mode_ops' and 'bulk' using new
types.
* cipher/cipher.c (_gcry_cipher_open_internal): Remove bulk function
setup.
(cipher_setkey): Pass context bulk function pointer to algorithm setkey
function.
* cipher/cipher-selftest.c (_gcry_selftest_helper_cbc)
(_gcry_selftest_helper_cfb, _gcry_selftest_helper_ctr): Remove bulk
function parameter; Use bulk function returned by setkey function.
* cipher/cipher-selftest.h (_gcry_selftest_helper_cbc)
(_gcry_selftest_helper_cfb, _gcry_selftest_helper_ctr): Remove bulk
function parameter.
* cipher/arcfour.c (arcfour_setkey): Change 'hd' parameter to
'bulk_ops'.
* cipher/blowfish.c (bf_setkey): Change 'hd' parameter to
'bulk_ops'; Setup 'bulk_ops' with bulk acceleration functions.
(_gcry_blowfish_ctr_enc, _gcry_blowfish_cbc_dec)
(_gcry_blowfish_cfb_dec): Make static.
(selftest_ctr, selftest_cbc, selftest_cfb): Do not pass bulk function
to selftest helper.
(selftest): Pass 'bulk_ops' to setkey function.
* cipher/camellia.c (camellia_setkey): Change 'hd' parameter to
'bulk_ops'; Setup 'bulk_ops' with bulk acceleration functions.
(_gcry_camellia_ctr_enc, _gcry_camellia_cbc_dec)
(_gcry_camellia_cfb_dec, _gcry_camellia_ocb_crypt)
(_gcry_camellia_ocb_auth): Make static.
(selftest_ctr, selftest_cbc, selftest_cfb): Do not pass bulk function
to selftest helper.
(selftest): Pass 'bulk_ops' to setkey function.
* cipher/cast5.c (cast_setkey): Change 'hd' parameter to
'bulk_ops'; Setup 'bulk_ops' with bulk acceleration functions.
(_gcry_cast5_ctr_enc, _gcry_cast5_cbc_dec, _gcry_cast5_cfb_dec): Make
static.
(selftest_ctr, selftest_cbc, selftest_cfb): Do not pass bulk function
to selftest helper.
(selftest): Pass 'bulk_ops' to setkey function.
* cipher/chacha20.c (chacha20_setkey): Change 'hd' parameter to
'bulk_ops'.
* cipher/cast5.c (do_tripledes_setkey): Change 'hd' parameter to
'bulk_ops'; Setup 'bulk_ops' with bulk acceleration functions.
(_gcry_3des_ctr_enc, _gcry_3des_cbc_dec, _gcry_3des_cfb_dec): Make
static.
(bulk_selftest_setkey): Change 'hd' parameter to 'bulk_ops'.
(selftest_ctr, selftest_cbc, selftest_cfb): Do not pass bulk function
to selftest helper.
(do_des_setkey): Change 'hd' parameter to 'bulk_ops'.
* cipher/gost28147.c (gost_setkey): Change 'hd' parameter to
'bulk_ops'.
* cipher/idea.c (idea_setkey): Change 'hd' parameter to 'bulk_ops'.
* cipher/rfc2268.c (do_setkey): Change 'hd' parameter to 'bulk_ops'.
* cipher/rijndael.c (do_setkey): Change 'hd' parameter to
'bulk_ops'; Setup 'bulk_ops' with bulk acceleration functions.
(rijndael_setkey): Change 'hd' parameter to 'bulk_ops'.
(_gcry_aes_cfb_enc, _gcry_aes_cfb_dec, _gcry_aes_cbc_enc)
(_gcry_aes_cbc_dec, _gcry_aes_ctr_enc, _gcry_aes_ocb_crypt)
(_gcry_aes_ocb_auth, _gcry_aes_xts_crypt): Make static.
(selftest_basic_128, selftest_basic_192, selftest_basic_256): Pass
'bulk_ops' to setkey function.
(selftest_ctr, selftest_cbc, selftest_cfb): Do not pass bulk function
to selftest helper.
* cipher/salsa20.c (salsa20_setkey): Change 'hd' parameter to
'bulk_ops'.
* cipher/seed.c (seed_setkey): Change 'hd' parameter to 'bulk_ops'.
* cipher/serpent.c (serpent_setkey): Change 'hd' parameter to
'bulk_ops'; Setup 'bulk_ops' with bulk acceleration functions.
(_gcry_serpent_ctr_enc, _gcry_serpent_cbc_dec, _gcry_serpent_cfb_dec)
(_gcry_serpent_ocb_crypt, _gcry_serpent_ocb_auth): Make static.
(selftest_ctr_128, selftest_cbc_128, selftest_cfb_128): Do not pass
bulk function to selftest helper.
* cipher/sm4.c (sm4_setkey): Change 'hd' parameter to 'bulk_ops'; Setup
'bulk_ops' with bulk acceleration functions.
(_gcry_sm4_ctr_enc, _gcry_sm4_cbc_dec, _gcry_sm4_cfb_dec)
(_gcry_sm4_ocb_crypt, _gcry_sm4_ocb_auth): Make static.
(selftest_ctr_128, selftest_cbc_128, selftest_cfb_128): Do not pass
bulk function to selftest helper.
* cipher/twofish.c (twofish_setkey): Change 'hd' parameter to
'bulk_ops'; Setup 'bulk_ops' with bulk acceleration functions.
(_gcry_twofish_ctr_enc, _gcry_twofish_cbc_dec)
(_gcry_twofish_cfb_dec, _gcry_twofish_ocb_crypt)
(_gcry_twofish_ocb_auth): Make static.
(selftest_ctr, selftest_cbc, selftest_cfb): Do not pass bulk function
to selftest helper.
(selftest, main): Pass 'bulk_ops' to setkey function.
* src/cipher-proto.h: Forward declare 'cipher_bulk_ops_t'.
(gcry_cipher_setkey_t): Replace 'hd' with 'bulk_ops'.
* src/cipher.h: Remove bulk acceleration function prototypes for
'aes', 'blowfish', 'cast5', 'camellia', '3des', 'serpent', 'sm4' and
'twofish'.
--

Signed-off-by: Jussi Kivilinna <jussi.kivilinna at iki.fi>
---
 cipher/arcfour.c         |   4 +-
 cipher/blowfish.c        |  33 ++++---
 cipher/camellia-glue.c   |  53 +++++++----
 cipher/cast5.c           |  34 +++----
 cipher/chacha20.c        |   4 +-
 cipher/cipher-internal.h |  93 +++++++++----------
 cipher/cipher-selftest.c |  29 +++---
 cipher/cipher-selftest.h |   5 +-
 cipher/cipher.c          |  88 +-----------------
 cipher/des.c             |  36 ++++----
 cipher/gost28147.c       |   4 +-
 cipher/idea.c            |   4 +-
 cipher/rfc2268.c         |   4 +-
 cipher/rijndael.c        | 188 ++++++++++++++++++++++-----------------
 cipher/salsa20.c         |   4 +-
 cipher/seed.c            |   4 +-
 cipher/serpent.c         |  48 ++++++----
 cipher/sm4.c             |  45 +++++++---
 cipher/twofish.c         |  71 ++++++++++-----
 src/cipher-proto.h       |   4 +-
 src/cipher.h             | 127 --------------------------
 21 files changed, 392 insertions(+), 490 deletions(-)

diff --git a/cipher/arcfour.c b/cipher/arcfour.c
index 72decf08..9e71857c 100644
--- a/cipher/arcfour.c
+++ b/cipher/arcfour.c
@@ -171,11 +171,11 @@ do_arcfour_setkey (void *context, const byte *key, unsigned int keylen)
 
 static gcry_err_code_t
 arcfour_setkey ( void *context, const byte *key, unsigned int keylen,
-                 gcry_cipher_hd_t hd )
+                 cipher_bulk_ops_t *bulk_ops )
 {
   ARCFOUR_context *ctx = (ARCFOUR_context *) context;
   gcry_err_code_t rc = do_arcfour_setkey (ctx, key, keylen );
-  (void)hd;
+  (void)bulk_ops;
   return rc;
 }
 
diff --git a/cipher/blowfish.c b/cipher/blowfish.c
index a1d81d31..7b001306 100644
--- a/cipher/blowfish.c
+++ b/cipher/blowfish.c
@@ -69,7 +69,7 @@ typedef struct {
 } BLOWFISH_context;
 
 static gcry_err_code_t bf_setkey (void *c, const byte *key, unsigned keylen,
-                                  gcry_cipher_hd_t hd);
+                                  cipher_bulk_ops_t *bulk_ops);
 static unsigned int encrypt_block (void *bc, byte *outbuf, const byte *inbuf);
 static unsigned int decrypt_block (void *bc, byte *outbuf, const byte *inbuf);
 
@@ -629,7 +629,7 @@ decrypt_block (void *context, byte *outbuf, const byte *inbuf)
 /* Bulk encryption of complete blocks in CTR mode.  This function is only
    intended for the bulk encryption feature of cipher.c.  CTR is expected to be
    of size BLOWFISH_BLOCKSIZE. */
-void
+static void
 _gcry_blowfish_ctr_enc(void *context, unsigned char *ctr, void *outbuf_arg,
 		       const void *inbuf_arg, size_t nblocks)
 {
@@ -710,7 +710,7 @@ _gcry_blowfish_ctr_enc(void *context, unsigned char *ctr, void *outbuf_arg,
 
 /* Bulk decryption of complete blocks in CBC mode.  This function is only
    intended for the bulk encryption feature of cipher.c. */
-void
+static void
 _gcry_blowfish_cbc_dec(void *context, unsigned char *iv, void *outbuf_arg,
 		       const void *inbuf_arg, size_t nblocks)
 {
@@ -787,7 +787,7 @@ _gcry_blowfish_cbc_dec(void *context, unsigned char *iv, void *outbuf_arg,
 
 /* Bulk decryption of complete blocks in CFB mode.  This function is only
    intended for the bulk encryption feature of cipher.c. */
-void
+static void
 _gcry_blowfish_cfb_dec(void *context, unsigned char *iv, void *outbuf_arg,
 		       const void *inbuf_arg, size_t nblocks)
 {
@@ -866,8 +866,7 @@ selftest_ctr (void)
   const int context_size = sizeof(BLOWFISH_context);
 
   return _gcry_selftest_helper_ctr("BLOWFISH", &bf_setkey,
-           &encrypt_block, &_gcry_blowfish_ctr_enc, nblocks, blocksize,
-	   context_size);
+           &encrypt_block, nblocks, blocksize, context_size);
 }
 
 
@@ -881,8 +880,7 @@ selftest_cbc (void)
   const int context_size = sizeof(BLOWFISH_context);
 
   return _gcry_selftest_helper_cbc("BLOWFISH", &bf_setkey,
-           &encrypt_block, &_gcry_blowfish_cbc_dec, nblocks, blocksize,
-	   context_size);
+           &encrypt_block, nblocks, blocksize, context_size);
 }
 
 
@@ -896,8 +894,7 @@ selftest_cfb (void)
   const int context_size = sizeof(BLOWFISH_context);
 
   return _gcry_selftest_helper_cfb("BLOWFISH", &bf_setkey,
-           &encrypt_block, &_gcry_blowfish_cfb_dec, nblocks, blocksize,
-	   context_size);
+           &encrypt_block, nblocks, blocksize, context_size);
 }
 
 
@@ -905,6 +902,7 @@ static const char*
 selftest(void)
 {
   BLOWFISH_context c;
+  cipher_bulk_ops_t bulk_ops;
   byte plain[] = "BLOWFISH";
   byte buffer[8];
   static const byte plain3[] =
@@ -916,7 +914,8 @@ selftest(void)
   const char *r;
 
   bf_setkey( (void *) &c,
-             (const unsigned char*)"abcdefghijklmnopqrstuvwxyz", 26, NULL );
+             (const unsigned char*)"abcdefghijklmnopqrstuvwxyz", 26,
+             &bulk_ops );
   encrypt_block( (void *) &c, buffer, plain );
   if( memcmp( buffer, "\x32\x4E\xD0\xFE\xF4\x13\xA2\x03", 8 ) )
     return "Blowfish selftest failed (1).";
@@ -924,7 +923,7 @@ selftest(void)
   if( memcmp( buffer, plain, 8 ) )
     return "Blowfish selftest failed (2).";
 
-  bf_setkey( (void *) &c, key3, 8, NULL );
+  bf_setkey( (void *) &c, key3, 8, &bulk_ops );
   encrypt_block( (void *) &c, buffer, plain3 );
   if( memcmp( buffer, cipher3, 8 ) )
     return "Blowfish selftest failed (3).";
@@ -1119,11 +1118,17 @@ do_bf_setkey (BLOWFISH_context *c, const byte *key, unsigned keylen)
 
 static gcry_err_code_t
 bf_setkey (void *context, const byte *key, unsigned keylen,
-           gcry_cipher_hd_t hd)
+           cipher_bulk_ops_t *bulk_ops)
 {
   BLOWFISH_context *c = (BLOWFISH_context *) context;
   gcry_err_code_t rc = do_bf_setkey (c, key, keylen);
-  (void)hd;
+
+  /* Setup bulk encryption routines.  */
+  memset (bulk_ops, 0, sizeof(*bulk_ops));
+  bulk_ops->cfb_dec = _gcry_blowfish_cfb_dec;
+  bulk_ops->cbc_dec = _gcry_blowfish_cbc_dec;
+  bulk_ops->ctr_enc = _gcry_blowfish_ctr_enc;
+
   return rc;
 }
 
diff --git a/cipher/camellia-glue.c b/cipher/camellia-glue.c
index 4b0989ea..6577b651 100644
--- a/cipher/camellia-glue.c
+++ b/cipher/camellia-glue.c
@@ -203,9 +203,24 @@ extern void _gcry_camellia_aesni_avx2_ocb_auth(CAMELLIA_context *ctx,
 
 static const char *selftest(void);
 
+static void _gcry_camellia_ctr_enc (void *context, unsigned char *ctr,
+				    void *outbuf_arg, const void *inbuf_arg,
+				    size_t nblocks);
+static void _gcry_camellia_cbc_dec (void *context, unsigned char *iv,
+				    void *outbuf_arg, const void *inbuf_arg,
+				    size_t nblocks);
+static void _gcry_camellia_cfb_dec (void *context, unsigned char *iv,
+				    void *outbuf_arg, const void *inbuf_arg,
+				    size_t nblocks);
+static size_t _gcry_camellia_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg,
+					const void *inbuf_arg, size_t nblocks,
+					int encrypt);
+static size_t _gcry_camellia_ocb_auth (gcry_cipher_hd_t c, const void *abuf_arg,
+				       size_t nblocks);
+
 static gcry_err_code_t
 camellia_setkey(void *c, const byte *key, unsigned keylen,
-                gcry_cipher_hd_t hd)
+                cipher_bulk_ops_t *bulk_ops)
 {
   CAMELLIA_context *ctx=c;
   static int initialized=0;
@@ -214,8 +229,6 @@ camellia_setkey(void *c, const byte *key, unsigned keylen,
   unsigned int hwf = _gcry_get_hw_features ();
 #endif
 
-  (void)hd;
-
   if(keylen!=16 && keylen!=24 && keylen!=32)
     return GPG_ERR_INV_KEYLEN;
 
@@ -239,6 +252,14 @@ camellia_setkey(void *c, const byte *key, unsigned keylen,
 
   ctx->keybitlength=keylen*8;
 
+  /* Setup bulk encryption routines.  */
+  memset (bulk_ops, 0, sizeof(*bulk_ops));
+  bulk_ops->cbc_dec = _gcry_camellia_cbc_dec;
+  bulk_ops->cfb_dec = _gcry_camellia_cfb_dec;
+  bulk_ops->ctr_enc = _gcry_camellia_ctr_enc;
+  bulk_ops->ocb_crypt = _gcry_camellia_ocb_crypt;
+  bulk_ops->ocb_auth  = _gcry_camellia_ocb_auth;
+
   if (0)
     { }
 #ifdef USE_AESNI_AVX
@@ -353,7 +374,7 @@ camellia_decrypt(void *c, byte *outbuf, const byte *inbuf)
 /* Bulk encryption of complete blocks in CTR mode.  This function is only
    intended for the bulk encryption feature of cipher.c.  CTR is expected to be
    of size CAMELLIA_BLOCK_SIZE. */
-void
+static void
 _gcry_camellia_ctr_enc(void *context, unsigned char *ctr,
                        void *outbuf_arg, const void *inbuf_arg,
                        size_t nblocks)
@@ -442,7 +463,7 @@ _gcry_camellia_ctr_enc(void *context, unsigned char *ctr,
 
 /* Bulk decryption of complete blocks in CBC mode.  This function is only
    intended for the bulk encryption feature of cipher.c. */
-void
+static void
 _gcry_camellia_cbc_dec(void *context, unsigned char *iv,
                        void *outbuf_arg, const void *inbuf_arg,
                        size_t nblocks)
@@ -529,7 +550,7 @@ _gcry_camellia_cbc_dec(void *context, unsigned char *iv,
 
 /* Bulk decryption of complete blocks in CFB mode.  This function is only
    intended for the bulk encryption feature of cipher.c. */
-void
+static void
 _gcry_camellia_cfb_dec(void *context, unsigned char *iv,
                        void *outbuf_arg, const void *inbuf_arg,
                        size_t nblocks)
@@ -609,7 +630,7 @@ _gcry_camellia_cfb_dec(void *context, unsigned char *iv,
 }
 
 /* Bulk encryption/decryption of complete blocks in OCB mode. */
-size_t
+static size_t
 _gcry_camellia_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg,
 			  const void *inbuf_arg, size_t nblocks, int encrypt)
 {
@@ -762,7 +783,7 @@ _gcry_camellia_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg,
 }
 
 /* Bulk authentication of complete blocks in OCB mode. */
-size_t
+static size_t
 _gcry_camellia_ocb_auth (gcry_cipher_hd_t c, const void *abuf_arg,
 			 size_t nblocks)
 {
@@ -912,8 +933,7 @@ selftest_ctr_128 (void)
   const int context_size = sizeof(CAMELLIA_context);
 
   return _gcry_selftest_helper_ctr("CAMELLIA", &camellia_setkey,
-           &camellia_encrypt, &_gcry_camellia_ctr_enc, nblocks, blocksize,
-	   context_size);
+           &camellia_encrypt, nblocks, blocksize, context_size);
 }
 
 /* Run the self-tests for CAMELLIA-CBC-128, tests bulk CBC decryption.
@@ -926,8 +946,7 @@ selftest_cbc_128 (void)
   const int context_size = sizeof(CAMELLIA_context);
 
   return _gcry_selftest_helper_cbc("CAMELLIA", &camellia_setkey,
-           &camellia_encrypt, &_gcry_camellia_cbc_dec, nblocks, blocksize,
-	   context_size);
+           &camellia_encrypt, nblocks, blocksize, context_size);
 }
 
 /* Run the self-tests for CAMELLIA-CFB-128, tests bulk CFB decryption.
@@ -940,8 +959,7 @@ selftest_cfb_128 (void)
   const int context_size = sizeof(CAMELLIA_context);
 
   return _gcry_selftest_helper_cfb("CAMELLIA", &camellia_setkey,
-           &camellia_encrypt, &_gcry_camellia_cfb_dec, nblocks, blocksize,
-	   context_size);
+           &camellia_encrypt, nblocks, blocksize, context_size);
 }
 
 static const char *
@@ -949,6 +967,7 @@ selftest(void)
 {
   CAMELLIA_context ctx;
   byte scratch[16];
+  cipher_bulk_ops_t bulk_ops;
   const char *r;
 
   /* These test vectors are from RFC-3713 */
@@ -989,7 +1008,7 @@ selftest(void)
       0x20,0xef,0x7c,0x91,0x9e,0x3a,0x75,0x09
     };
 
-  camellia_setkey(&ctx,key_128,sizeof(key_128),NULL);
+  camellia_setkey(&ctx,key_128,sizeof(key_128),&bulk_ops);
   camellia_encrypt(&ctx,scratch,plaintext);
   if(memcmp(scratch,ciphertext_128,sizeof(ciphertext_128))!=0)
     return "CAMELLIA-128 test encryption failed.";
@@ -997,7 +1016,7 @@ selftest(void)
   if(memcmp(scratch,plaintext,sizeof(plaintext))!=0)
     return "CAMELLIA-128 test decryption failed.";
 
-  camellia_setkey(&ctx,key_192,sizeof(key_192),NULL);
+  camellia_setkey(&ctx,key_192,sizeof(key_192),&bulk_ops);
   camellia_encrypt(&ctx,scratch,plaintext);
   if(memcmp(scratch,ciphertext_192,sizeof(ciphertext_192))!=0)
     return "CAMELLIA-192 test encryption failed.";
@@ -1005,7 +1024,7 @@ selftest(void)
   if(memcmp(scratch,plaintext,sizeof(plaintext))!=0)
     return "CAMELLIA-192 test decryption failed.";
 
-  camellia_setkey(&ctx,key_256,sizeof(key_256),NULL);
+  camellia_setkey(&ctx,key_256,sizeof(key_256),&bulk_ops);
   camellia_encrypt(&ctx,scratch,plaintext);
   if(memcmp(scratch,ciphertext_256,sizeof(ciphertext_256))!=0)
     return "CAMELLIA-256 test encryption failed.";
diff --git a/cipher/cast5.c b/cipher/cast5.c
index 7219e3ea..837ea0fe 100644
--- a/cipher/cast5.c
+++ b/cipher/cast5.c
@@ -74,7 +74,7 @@ typedef struct {
 } CAST5_context;
 
 static gcry_err_code_t cast_setkey (void *c, const byte *key, unsigned keylen,
-                                    gcry_cipher_hd_t hd);
+                                    cipher_bulk_ops_t *bulk_ops);
 static unsigned int encrypt_block (void *c, byte *outbuf, const byte *inbuf);
 static unsigned int decrypt_block (void *c, byte *outbuf, const byte *inbuf);
 
@@ -765,7 +765,7 @@ do_decrypt_block_3 (CAST5_context *c, byte *outbuf, const byte *inbuf )
 /* Bulk encryption of complete blocks in CTR mode.  This function is only
    intended for the bulk encryption feature of cipher.c.  CTR is expected to be
    of size CAST5_BLOCKSIZE. */
-void
+static void
 _gcry_cast5_ctr_enc(void *context, unsigned char *ctr, void *outbuf_arg,
 		    const void *inbuf_arg, size_t nblocks)
 {
@@ -846,7 +846,7 @@ _gcry_cast5_ctr_enc(void *context, unsigned char *ctr, void *outbuf_arg,
 
 /* Bulk decryption of complete blocks in CBC mode.  This function is only
    intended for the bulk encryption feature of cipher.c. */
-void
+static void
 _gcry_cast5_cbc_dec(void *context, unsigned char *iv, void *outbuf_arg,
 		    const void *inbuf_arg, size_t nblocks)
 {
@@ -922,7 +922,7 @@ _gcry_cast5_cbc_dec(void *context, unsigned char *iv, void *outbuf_arg,
 
 /* Bulk decryption of complete blocks in CFB mode.  This function is only
    intended for the bulk encryption feature of cipher.c. */
-void
+static void
 _gcry_cast5_cfb_dec(void *context, unsigned char *iv, void *outbuf_arg,
 		    const void *inbuf_arg, size_t nblocks)
 {
@@ -1001,8 +1001,7 @@ selftest_ctr (void)
   const int context_size = sizeof(CAST5_context);
 
   return _gcry_selftest_helper_ctr("CAST5", &cast_setkey,
-           &encrypt_block, &_gcry_cast5_ctr_enc, nblocks, blocksize,
-	   context_size);
+           &encrypt_block, nblocks, blocksize, context_size);
 }
 
 
@@ -1016,8 +1015,7 @@ selftest_cbc (void)
   const int context_size = sizeof(CAST5_context);
 
   return _gcry_selftest_helper_cbc("CAST5", &cast_setkey,
-           &encrypt_block, &_gcry_cast5_cbc_dec, nblocks, blocksize,
-	   context_size);
+           &encrypt_block, nblocks, blocksize, context_size);
 }
 
 
@@ -1031,8 +1029,7 @@ selftest_cfb (void)
   const int context_size = sizeof(CAST5_context);
 
   return _gcry_selftest_helper_cfb("CAST5", &cast_setkey,
-           &encrypt_block, &_gcry_cast5_cfb_dec, nblocks, blocksize,
-	   context_size);
+           &encrypt_block, nblocks, blocksize, context_size);
 }
 
 
@@ -1040,6 +1037,7 @@ static const char*
 selftest(void)
 {
     CAST5_context c;
+    cipher_bulk_ops_t bulk_ops;
     static const byte key[16] =
                     { 0x01, 0x23, 0x45, 0x67, 0x12, 0x34, 0x56, 0x78,
 		      0x23, 0x45, 0x67, 0x89, 0x34, 0x56, 0x78, 0x9A  };
@@ -1050,7 +1048,7 @@ selftest(void)
     byte buffer[8];
     const char *r;
 
-    cast_setkey( &c, key, 16, NULL );
+    cast_setkey( &c, key, 16, &bulk_ops );
     encrypt_block( &c, buffer, plain );
     if( memcmp( buffer, cipher, 8 ) )
 	return "1";
@@ -1071,10 +1069,10 @@ selftest(void)
 			0x80,0xAC,0x05,0xB8,0xE8,0x3D,0x69,0x6E };
 
 	for(i=0; i < 1000000; i++ ) {
-	    cast_setkey( &c, b0, 16, NULL );
+	    cast_setkey( &c, b0, 16, &bulk_ops );
 	    encrypt_block( &c, a0, a0 );
 	    encrypt_block( &c, a0+8, a0+8 );
-	    cast_setkey( &c, a0, 16, NULL );
+	    cast_setkey( &c, a0, 16, &bulk_ops );
 	    encrypt_block( &c, b0, b0 );
 	    encrypt_block( &c, b0+8, b0+8 );
 	}
@@ -1217,11 +1215,17 @@ do_cast_setkey( CAST5_context *c, const byte *key, unsigned keylen )
 
 static gcry_err_code_t
 cast_setkey (void *context, const byte *key, unsigned keylen,
-             gcry_cipher_hd_t hd )
+             cipher_bulk_ops_t *bulk_ops)
 {
   CAST5_context *c = (CAST5_context *) context;
   gcry_err_code_t rc = do_cast_setkey (c, key, keylen);
-  (void)hd;
+
+  /* Setup bulk encryption routines.  */
+  memset (bulk_ops, 0, sizeof(*bulk_ops));
+  bulk_ops->cfb_dec = _gcry_cast5_cfb_dec;
+  bulk_ops->cbc_dec = _gcry_cast5_cbc_dec;
+  bulk_ops->ctr_enc = _gcry_cast5_ctr_enc;
+
   return rc;
 }
 
diff --git a/cipher/chacha20.c b/cipher/chacha20.c
index 9d95723b..c5967b6f 100644
--- a/cipher/chacha20.c
+++ b/cipher/chacha20.c
@@ -452,11 +452,11 @@ chacha20_do_setkey (CHACHA20_context_t *ctx,
 
 static gcry_err_code_t
 chacha20_setkey (void *context, const byte *key, unsigned int keylen,
-                 gcry_cipher_hd_t hd)
+                 cipher_bulk_ops_t *bulk_ops)
 {
   CHACHA20_context_t *ctx = (CHACHA20_context_t *) context;
   gcry_err_code_t rc = chacha20_do_setkey (ctx, key, keylen);
-  (void)hd;
+  (void)bulk_ops;
   _gcry_burn_stack (4 + sizeof (void *) + 4 * sizeof (void *));
   return rc;
 }
diff --git a/cipher/cipher-internal.h b/cipher/cipher-internal.h
index 262ca902..a2d0e5c7 100644
--- a/cipher/cipher-internal.h
+++ b/cipher/cipher-internal.h
@@ -100,6 +100,51 @@ typedef unsigned int (*ghash_fn_t) (gcry_cipher_hd_t c, byte *result,
                                     const byte *buf, size_t nblocks);
 
 
+/* A structure with function pointers for mode operations. */
+typedef struct cipher_mode_ops
+{
+  gcry_err_code_t (*encrypt)(gcry_cipher_hd_t c, unsigned char *outbuf,
+			     size_t outbuflen, const unsigned char *inbuf,
+			     size_t inbuflen);
+  gcry_err_code_t (*decrypt)(gcry_cipher_hd_t c, unsigned char *outbuf,
+			     size_t outbuflen, const unsigned char *inbuf,
+			     size_t inbuflen);
+  gcry_err_code_t (*setiv)(gcry_cipher_hd_t c, const unsigned char *iv,
+			   size_t ivlen);
+
+  gcry_err_code_t (*authenticate)(gcry_cipher_hd_t c,
+				  const unsigned char *abuf, size_t abuflen);
+  gcry_err_code_t (*get_tag)(gcry_cipher_hd_t c, unsigned char *outtag,
+			     size_t taglen);
+  gcry_err_code_t (*check_tag)(gcry_cipher_hd_t c, const unsigned char *intag,
+			       size_t taglen);
+} cipher_mode_ops_t;
+
+
+/* A structure with function pointers for bulk operations.  The cipher
+   algorithm setkey function initializes them when bulk operations are
+   available and the actual encryption routines use them if they are
+   not NULL.  */
+typedef struct cipher_bulk_ops
+{
+  void (*cfb_enc)(void *context, unsigned char *iv, void *outbuf_arg,
+		  const void *inbuf_arg, size_t nblocks);
+  void (*cfb_dec)(void *context, unsigned char *iv, void *outbuf_arg,
+		  const void *inbuf_arg, size_t nblocks);
+  void (*cbc_enc)(void *context, unsigned char *iv, void *outbuf_arg,
+		  const void *inbuf_arg, size_t nblocks, int cbc_mac);
+  void (*cbc_dec)(void *context, unsigned char *iv, void *outbuf_arg,
+		  const void *inbuf_arg, size_t nblocks);
+  void (*ctr_enc)(void *context, unsigned char *iv, void *outbuf_arg,
+		  const void *inbuf_arg, size_t nblocks);
+  size_t (*ocb_crypt)(gcry_cipher_hd_t c, void *outbuf_arg,
+		      const void *inbuf_arg, size_t nblocks, int encrypt);
+  size_t (*ocb_auth)(gcry_cipher_hd_t c, const void *abuf_arg, size_t nblocks);
+  void (*xts_crypt)(void *context, unsigned char *tweak, void *outbuf_arg,
+		    const void *inbuf_arg, size_t nblocks, int encrypt);
+} cipher_bulk_ops_t;
+
+
 /* A VIA processor with the Padlock engine as well as the Intel AES_NI
    instructions require an alignment of most data on a 16 byte
    boundary.  Because we trick out the compiler while allocating the
@@ -150,54 +195,12 @@ struct gcry_cipher_handle
   int algo;
 
   /* A structure with function pointers for mode operations. */
-  struct {
-    gcry_err_code_t (*encrypt)(gcry_cipher_hd_t c,
-                               unsigned char *outbuf, size_t outbuflen,
-                               const unsigned char *inbuf, size_t inbuflen);
-    gcry_err_code_t (*decrypt)(gcry_cipher_hd_t c,
-                               unsigned char *outbuf, size_t outbuflen,
-                               const unsigned char *inbuf, size_t inbuflen);
-    gcry_err_code_t (*setiv)(gcry_cipher_hd_t c, const unsigned char *iv,
-                             size_t ivlen);
-
-    gcry_err_code_t (*authenticate)(gcry_cipher_hd_t c,
-                                    const unsigned char *abuf, size_t abuflen);
-    gcry_err_code_t (*get_tag)(gcry_cipher_hd_t c, unsigned char *outtag,
-                               size_t taglen);
-    gcry_err_code_t (*check_tag)(gcry_cipher_hd_t c, const unsigned char *intag,
-                                 size_t taglen);
-  } mode_ops;
+  cipher_mode_ops_t mode_ops;
 
   /* A structure with function pointers for bulk operations.  Due to
      limitations of the module system (we don't want to change the
-     API) we need to keep these function pointers here.  The cipher
-     open function initializes them and the actual encryption routines
-     use them if they are not NULL.  */
-  struct {
-    void (*cfb_enc)(void *context, unsigned char *iv,
-                    void *outbuf_arg, const void *inbuf_arg,
-                    size_t nblocks);
-    void (*cfb_dec)(void *context, unsigned char *iv,
-                    void *outbuf_arg, const void *inbuf_arg,
-                    size_t nblocks);
-    void (*cbc_enc)(void *context, unsigned char *iv,
-                    void *outbuf_arg, const void *inbuf_arg,
-                    size_t nblocks, int cbc_mac);
-    void (*cbc_dec)(void *context, unsigned char *iv,
-                    void *outbuf_arg, const void *inbuf_arg,
-                    size_t nblocks);
-    void (*ctr_enc)(void *context, unsigned char *iv,
-                    void *outbuf_arg, const void *inbuf_arg,
-                    size_t nblocks);
-    size_t (*ocb_crypt)(gcry_cipher_hd_t c, void *outbuf_arg,
-			const void *inbuf_arg, size_t nblocks, int encrypt);
-    size_t (*ocb_auth)(gcry_cipher_hd_t c, const void *abuf_arg,
-		       size_t nblocks);
-    void (*xts_crypt)(void *context, unsigned char *tweak,
-		      void *outbuf_arg, const void *inbuf_arg,
-		      size_t nblocks, int encrypt);
-  } bulk;
-
+     API) we need to keep these function pointers here.  */
+  cipher_bulk_ops_t bulk;
 
   int mode;
   unsigned int flags;
diff --git a/cipher/cipher-selftest.c b/cipher/cipher-selftest.c
index eb3614ad..d7f38a42 100644
--- a/cipher/cipher-selftest.c
+++ b/cipher/cipher-selftest.c
@@ -1,5 +1,5 @@
 /* cipher-selftest.c - Helper functions for bulk encryption selftests.
- * Copyright (C) 2013 Jussi Kivilinna <jussi.kivilinna at iki.fi>
+ * Copyright (C) 2013,2020 Jussi Kivilinna <jussi.kivilinna at iki.fi>
  *
  * This file is part of Libgcrypt.
  *
@@ -27,6 +27,7 @@
 #include "cipher.h"
 #include "bufhelp.h"
 #include "cipher-selftest.h"
+#include "cipher-internal.h"
 
 #ifdef HAVE_STDINT_H
 # include <stdint.h> /* uintptr_t */
@@ -72,10 +73,10 @@ _gcry_cipher_selftest_alloc_ctx (const int context_size, unsigned char **r_mem)
 const char *
 _gcry_selftest_helper_cbc (const char *cipher, gcry_cipher_setkey_t setkey_func,
 			   gcry_cipher_encrypt_t encrypt_one,
-			   gcry_cipher_bulk_cbc_dec_t bulk_cbc_dec,
 			   const int nblocks, const int blocksize,
 			   const int context_size)
 {
+  cipher_bulk_ops_t bulk_ops = { 0, };
   int i, offs;
   unsigned char *ctx, *plaintext, *plaintext2, *ciphertext, *iv, *iv2, *mem;
   unsigned int ctx_aligned_size, memsize;
@@ -105,7 +106,7 @@ _gcry_selftest_helper_cbc (const char *cipher, gcry_cipher_setkey_t setkey_func,
   ciphertext = plaintext2 + nblocks * blocksize;
 
   /* Initialize ctx */
-  if (setkey_func (ctx, key, sizeof(key), NULL) != GPG_ERR_NO_ERROR)
+  if (setkey_func (ctx, key, sizeof(key), &bulk_ops) != GPG_ERR_NO_ERROR)
    {
      xfree(mem);
      return "setkey failed";
@@ -123,7 +124,7 @@ _gcry_selftest_helper_cbc (const char *cipher, gcry_cipher_setkey_t setkey_func,
   memcpy (iv, ciphertext, blocksize);
 
   /* CBC decrypt.  */
-  bulk_cbc_dec (ctx, iv2, plaintext2, ciphertext, 1);
+  bulk_ops.cbc_dec (ctx, iv2, plaintext2, ciphertext, 1);
   if (memcmp (plaintext2, plaintext, blocksize))
     {
       xfree (mem);
@@ -163,7 +164,7 @@ _gcry_selftest_helper_cbc (const char *cipher, gcry_cipher_setkey_t setkey_func,
     }
 
   /* Decrypt using bulk CBC and compare result.  */
-  bulk_cbc_dec (ctx, iv2, plaintext2, ciphertext, nblocks);
+  bulk_ops.cbc_dec (ctx, iv2, plaintext2, ciphertext, nblocks);
 
   if (memcmp (plaintext2, plaintext, nblocks * blocksize))
     {
@@ -195,10 +196,10 @@ _gcry_selftest_helper_cbc (const char *cipher, gcry_cipher_setkey_t setkey_func,
 const char *
 _gcry_selftest_helper_cfb (const char *cipher, gcry_cipher_setkey_t setkey_func,
 			   gcry_cipher_encrypt_t encrypt_one,
-			   gcry_cipher_bulk_cfb_dec_t bulk_cfb_dec,
 			   const int nblocks, const int blocksize,
 			   const int context_size)
 {
+  cipher_bulk_ops_t bulk_ops = { 0, };
   int i, offs;
   unsigned char *ctx, *plaintext, *plaintext2, *ciphertext, *iv, *iv2, *mem;
   unsigned int ctx_aligned_size, memsize;
@@ -228,7 +229,7 @@ _gcry_selftest_helper_cfb (const char *cipher, gcry_cipher_setkey_t setkey_func,
   ciphertext = plaintext2 + nblocks * blocksize;
 
   /* Initialize ctx */
-  if (setkey_func (ctx, key, sizeof(key), NULL) != GPG_ERR_NO_ERROR)
+  if (setkey_func (ctx, key, sizeof(key), &bulk_ops) != GPG_ERR_NO_ERROR)
    {
      xfree(mem);
      return "setkey failed";
@@ -245,7 +246,7 @@ _gcry_selftest_helper_cfb (const char *cipher, gcry_cipher_setkey_t setkey_func,
   buf_xor_2dst (iv, ciphertext, plaintext, blocksize);
 
   /* CFB decrypt.  */
-  bulk_cfb_dec (ctx, iv2, plaintext2, ciphertext, 1);
+  bulk_ops.cfb_dec (ctx, iv2, plaintext2, ciphertext, 1);
   if (memcmp(plaintext2, plaintext, blocksize))
     {
       xfree(mem);
@@ -284,7 +285,7 @@ _gcry_selftest_helper_cfb (const char *cipher, gcry_cipher_setkey_t setkey_func,
     }
 
   /* Decrypt using bulk CBC and compare result.  */
-  bulk_cfb_dec (ctx, iv2, plaintext2, ciphertext, nblocks);
+  bulk_ops.cfb_dec (ctx, iv2, plaintext2, ciphertext, nblocks);
 
   if (memcmp(plaintext2, plaintext, nblocks * blocksize))
     {
@@ -316,10 +317,10 @@ _gcry_selftest_helper_cfb (const char *cipher, gcry_cipher_setkey_t setkey_func,
 const char *
 _gcry_selftest_helper_ctr (const char *cipher, gcry_cipher_setkey_t setkey_func,
 			   gcry_cipher_encrypt_t encrypt_one,
-			   gcry_cipher_bulk_ctr_enc_t bulk_ctr_enc,
 			   const int nblocks, const int blocksize,
 			   const int context_size)
 {
+  cipher_bulk_ops_t bulk_ops = { 0, };
   int i, j, offs, diff;
   unsigned char *ctx, *plaintext, *plaintext2, *ciphertext, *ciphertext2,
                 *iv, *iv2, *mem;
@@ -351,7 +352,7 @@ _gcry_selftest_helper_ctr (const char *cipher, gcry_cipher_setkey_t setkey_func,
   ciphertext2 = ciphertext + nblocks * blocksize;
 
   /* Initialize ctx */
-  if (setkey_func (ctx, key, sizeof(key), NULL) != GPG_ERR_NO_ERROR)
+  if (setkey_func (ctx, key, sizeof(key), &bulk_ops) != GPG_ERR_NO_ERROR)
    {
      xfree(mem);
      return "setkey failed";
@@ -374,7 +375,7 @@ _gcry_selftest_helper_ctr (const char *cipher, gcry_cipher_setkey_t setkey_func,
     }
 
   memset (iv2, 0xff, blocksize);
-  bulk_ctr_enc (ctx, iv2, plaintext2, ciphertext, 1);
+  bulk_ops.ctr_enc (ctx, iv2, plaintext2, ciphertext, 1);
 
   if (memcmp (plaintext2, plaintext, blocksize))
     {
@@ -429,7 +430,7 @@ _gcry_selftest_helper_ctr (const char *cipher, gcry_cipher_setkey_t setkey_func,
         }
     }
 
-  bulk_ctr_enc (ctx, iv2, ciphertext2, plaintext2, nblocks);
+  bulk_ops.ctr_enc (ctx, iv2, ciphertext2, plaintext2, nblocks);
 
   if (memcmp (ciphertext2, ciphertext, blocksize * nblocks))
     {
@@ -482,7 +483,7 @@ _gcry_selftest_helper_ctr (const char *cipher, gcry_cipher_setkey_t setkey_func,
     iv2[0] = iv2[1] = 0;
     iv2[2] = 0x07;
 
-    bulk_ctr_enc (ctx, iv2, plaintext2, ciphertext, nblocks);
+    bulk_ops.ctr_enc (ctx, iv2, plaintext2, ciphertext, nblocks);
 
     if (memcmp (plaintext2, plaintext, blocksize * nblocks))
       {
diff --git a/cipher/cipher-selftest.h b/cipher/cipher-selftest.h
index a435080f..c3090ad1 100644
--- a/cipher/cipher-selftest.h
+++ b/cipher/cipher-selftest.h
@@ -1,5 +1,5 @@
 /* cipher-selftest.h - Helper functions for bulk encryption selftests.
- * Copyright (C) 2013 Jussi Kivilinna <jussi.kivilinna at iki.fi>
+ * Copyright (C) 2013,2020 Jussi Kivilinna <jussi.kivilinna at iki.fi>
  *
  * This file is part of Libgcrypt.
  *
@@ -49,7 +49,6 @@ void *_gcry_cipher_selftest_alloc_ctx (const int context_size,
 const char *
 _gcry_selftest_helper_cbc (const char *cipher, gcry_cipher_setkey_t setkey,
 			   gcry_cipher_encrypt_t encrypt_one,
-			   gcry_cipher_bulk_cbc_dec_t bulk_cbc_dec,
 			   const int nblocks, const int blocksize,
 			   const int context_size);
 
@@ -57,7 +56,6 @@ _gcry_selftest_helper_cbc (const char *cipher, gcry_cipher_setkey_t setkey,
 const char *
 _gcry_selftest_helper_cfb (const char *cipher, gcry_cipher_setkey_t setkey,
 			   gcry_cipher_encrypt_t encrypt_one,
-			   gcry_cipher_bulk_cfb_dec_t bulk_cfb_dec,
 			   const int nblocks, const int blocksize,
 			   const int context_size);
 
@@ -65,7 +63,6 @@ _gcry_selftest_helper_cfb (const char *cipher, gcry_cipher_setkey_t setkey,
 const char *
 _gcry_selftest_helper_ctr (const char *cipher, gcry_cipher_setkey_t setkey,
 			   gcry_cipher_encrypt_t encrypt_one,
-			   gcry_cipher_bulk_ctr_enc_t bulk_ctr_enc,
 			   const int nblocks, const int blocksize,
 			   const int context_size);
 
diff --git a/cipher/cipher.c b/cipher/cipher.c
index c77c9682..1039dff7 100644
--- a/cipher/cipher.c
+++ b/cipher/cipher.c
@@ -647,90 +647,6 @@ _gcry_cipher_open_internal (gcry_cipher_hd_t *handle,
 	  h->mode = mode;
 	  h->flags = flags;
 
-          /* Setup bulk encryption routines.  */
-          switch (algo)
-            {
-#ifdef USE_AES
-            case GCRY_CIPHER_AES128:
-            case GCRY_CIPHER_AES192:
-            case GCRY_CIPHER_AES256:
-              h->bulk.cfb_enc = _gcry_aes_cfb_enc;
-              h->bulk.cfb_dec = _gcry_aes_cfb_dec;
-              h->bulk.cbc_enc = _gcry_aes_cbc_enc;
-              h->bulk.cbc_dec = _gcry_aes_cbc_dec;
-              h->bulk.ctr_enc = _gcry_aes_ctr_enc;
-              h->bulk.ocb_crypt = _gcry_aes_ocb_crypt;
-              h->bulk.ocb_auth  = _gcry_aes_ocb_auth;
-              h->bulk.xts_crypt = _gcry_aes_xts_crypt;
-              break;
-#endif /*USE_AES*/
-#ifdef USE_BLOWFISH
-	    case GCRY_CIPHER_BLOWFISH:
-              h->bulk.cfb_dec = _gcry_blowfish_cfb_dec;
-              h->bulk.cbc_dec = _gcry_blowfish_cbc_dec;
-              h->bulk.ctr_enc = _gcry_blowfish_ctr_enc;
-              break;
-#endif /*USE_BLOWFISH*/
-#ifdef USE_CAST5
-	    case GCRY_CIPHER_CAST5:
-              h->bulk.cfb_dec = _gcry_cast5_cfb_dec;
-              h->bulk.cbc_dec = _gcry_cast5_cbc_dec;
-              h->bulk.ctr_enc = _gcry_cast5_ctr_enc;
-              break;
-#endif /*USE_CAMELLIA*/
-#ifdef USE_CAMELLIA
-	    case GCRY_CIPHER_CAMELLIA128:
-	    case GCRY_CIPHER_CAMELLIA192:
-	    case GCRY_CIPHER_CAMELLIA256:
-              h->bulk.cbc_dec = _gcry_camellia_cbc_dec;
-              h->bulk.cfb_dec = _gcry_camellia_cfb_dec;
-              h->bulk.ctr_enc = _gcry_camellia_ctr_enc;
-              h->bulk.ocb_crypt = _gcry_camellia_ocb_crypt;
-              h->bulk.ocb_auth  = _gcry_camellia_ocb_auth;
-              break;
-#endif /*USE_CAMELLIA*/
-#ifdef USE_DES
-            case GCRY_CIPHER_3DES:
-              h->bulk.cbc_dec =  _gcry_3des_cbc_dec;
-              h->bulk.cfb_dec =  _gcry_3des_cfb_dec;
-              h->bulk.ctr_enc =  _gcry_3des_ctr_enc;
-              break;
-#endif /*USE_DES*/
-#ifdef USE_SERPENT
-	    case GCRY_CIPHER_SERPENT128:
-	    case GCRY_CIPHER_SERPENT192:
-	    case GCRY_CIPHER_SERPENT256:
-              h->bulk.cbc_dec = _gcry_serpent_cbc_dec;
-              h->bulk.cfb_dec = _gcry_serpent_cfb_dec;
-              h->bulk.ctr_enc = _gcry_serpent_ctr_enc;
-              h->bulk.ocb_crypt = _gcry_serpent_ocb_crypt;
-              h->bulk.ocb_auth  = _gcry_serpent_ocb_auth;
-              break;
-#endif /*USE_SERPENT*/
-#ifdef USE_SM4
-	    case GCRY_CIPHER_SM4:
-              h->bulk.cbc_dec = _gcry_sm4_cbc_dec;
-              h->bulk.cfb_dec = _gcry_sm4_cfb_dec;
-              h->bulk.ctr_enc = _gcry_sm4_ctr_enc;
-              h->bulk.ocb_crypt = _gcry_sm4_ocb_crypt;
-              h->bulk.ocb_auth  = _gcry_sm4_ocb_auth;
-              break;
-#endif /*USE_SM4*/
-#ifdef USE_TWOFISH
-	    case GCRY_CIPHER_TWOFISH:
-	    case GCRY_CIPHER_TWOFISH128:
-              h->bulk.cbc_dec = _gcry_twofish_cbc_dec;
-              h->bulk.cfb_dec = _gcry_twofish_cfb_dec;
-              h->bulk.ctr_enc = _gcry_twofish_ctr_enc;
-              h->bulk.ocb_crypt = _gcry_twofish_ocb_crypt;
-              h->bulk.ocb_auth  = _gcry_twofish_ocb_auth;
-              break;
-#endif /*USE_TWOFISH*/
-
-            default:
-              break;
-            }
-
           /* Setup mode routines. */
           _gcry_cipher_setup_mode_ops(h, mode);
 
@@ -816,7 +732,7 @@ cipher_setkey (gcry_cipher_hd_t c, byte *key, size_t keylen)
 	}
     }
 
-  rc = c->spec->setkey (&c->context.c, key, keylen, c);
+  rc = c->spec->setkey (&c->context.c, key, keylen, &c->bulk);
   if (!rc || (c->marks.allow_weak_key && rc == GPG_ERR_WEAK_KEY))
     {
       /* Duplicate initial context.  */
@@ -850,7 +766,7 @@ cipher_setkey (gcry_cipher_hd_t c, byte *key, size_t keylen)
 	case GCRY_CIPHER_MODE_XTS:
 	  /* Setup tweak cipher with second part of XTS key. */
 	  rc = c->spec->setkey (c->u_mode.xts.tweak_context, key + keylen,
-				keylen, c);
+				keylen, &c->bulk);
 	  if (!rc || (c->marks.allow_weak_key && rc == GPG_ERR_WEAK_KEY))
 	    {
 	      /* Duplicate initial tweak context.  */
diff --git a/cipher/des.c b/cipher/des.c
index e4d10caa..1580ea4e 100644
--- a/cipher/des.c
+++ b/cipher/des.c
@@ -199,7 +199,7 @@ static unsigned int do_tripledes_decrypt(void *context, byte *outbuf,
 					 const byte *inbuf );
 static gcry_err_code_t do_tripledes_setkey(void *context, const byte *key,
                                            unsigned keylen,
-                                           gcry_cipher_hd_t hd);
+                                           cipher_bulk_ops_t *bulk_ops);
 
 static int initialized;
 
@@ -872,7 +872,7 @@ tripledes_ecb_crypt (struct _tripledes_ctx *ctx, const byte * from,
 /* Bulk encryption of complete blocks in CTR mode.  This function is only
    intended for the bulk encryption feature of cipher.c.  CTR is expected to be
    of size DES_BLOCKSIZE. */
-void
+static void
 _gcry_3des_ctr_enc(void *context, unsigned char *ctr, void *outbuf_arg,
                    const void *inbuf_arg, size_t nblocks)
 {
@@ -922,7 +922,7 @@ _gcry_3des_ctr_enc(void *context, unsigned char *ctr, void *outbuf_arg,
 
 /* Bulk decryption of complete blocks in CBC mode.  This function is only
    intended for the bulk encryption feature of cipher.c. */
-void
+static void
 _gcry_3des_cbc_dec(void *context, unsigned char *iv, void *outbuf_arg,
                    const void *inbuf_arg, size_t nblocks)
 {
@@ -971,7 +971,7 @@ _gcry_3des_cbc_dec(void *context, unsigned char *iv, void *outbuf_arg,
 
 /* Bulk decryption of complete blocks in CFB mode.  This function is only
    intended for the bulk encryption feature of cipher.c. */
-void
+static void
 _gcry_3des_cfb_dec(void *context, unsigned char *iv, void *outbuf_arg,
 		   const void *inbuf_arg, size_t nblocks)
 {
@@ -1050,7 +1050,7 @@ is_weak_key ( const byte *key )
 /* Alternative setkey for selftests; need larger key than default. */
 static gcry_err_code_t
 bulk_selftest_setkey (void *context, const byte *__key, unsigned __keylen,
-                      gcry_cipher_hd_t hd)
+                      cipher_bulk_ops_t *bulk_ops)
 {
   static const unsigned char key[24] ATTR_ALIGNED_16 = {
       0x66,0x9A,0x00,0x7F,0xC7,0x6A,0x45,0x9F,
@@ -1058,11 +1058,10 @@ bulk_selftest_setkey (void *context, const byte *__key, unsigned __keylen,
       0x18,0x2A,0x39,0x47,0x5E,0x6F,0x75,0x82
     };
 
-  (void)hd;
   (void)__key;
   (void)__keylen;
 
-  return do_tripledes_setkey(context, key, sizeof(key), NULL);
+  return do_tripledes_setkey(context, key, sizeof(key), bulk_ops);
 }
 
 
@@ -1076,8 +1075,7 @@ selftest_ctr (void)
   const int context_size = sizeof(struct _tripledes_ctx);
 
   return _gcry_selftest_helper_ctr("3DES", &bulk_selftest_setkey,
-           &do_tripledes_encrypt, &_gcry_3des_ctr_enc, nblocks, blocksize,
-           context_size);
+           &do_tripledes_encrypt, nblocks, blocksize, context_size);
 }
 
 
@@ -1091,8 +1089,7 @@ selftest_cbc (void)
   const int context_size = sizeof(struct _tripledes_ctx);
 
   return _gcry_selftest_helper_cbc("3DES", &bulk_selftest_setkey,
-           &do_tripledes_encrypt, &_gcry_3des_cbc_dec, nblocks, blocksize,
-           context_size);
+           &do_tripledes_encrypt, nblocks, blocksize, context_size);
 }
 
 
@@ -1106,8 +1103,7 @@ selftest_cfb (void)
   const int context_size = sizeof(struct _tripledes_ctx);
 
   return _gcry_selftest_helper_cfb("3DES", &bulk_selftest_setkey,
-           &do_tripledes_encrypt, &_gcry_3des_cfb_dec, nblocks, blocksize,
-           context_size);
+           &do_tripledes_encrypt, nblocks, blocksize, context_size);
 }
 
 
@@ -1315,15 +1311,19 @@ selftest (void)
 
 static gcry_err_code_t
 do_tripledes_setkey ( void *context, const byte *key, unsigned keylen,
-                      gcry_cipher_hd_t hd )
+                      cipher_bulk_ops_t *bulk_ops )
 {
   struct _tripledes_ctx *ctx = (struct _tripledes_ctx *) context;
 
-  (void)hd;
-
   if( keylen != 24 )
     return GPG_ERR_INV_KEYLEN;
 
+  /* Setup bulk encryption routines.  */
+  memset (bulk_ops, 0, sizeof(*bulk_ops));
+  bulk_ops->cbc_dec =  _gcry_3des_cbc_dec;
+  bulk_ops->cfb_dec =  _gcry_3des_cfb_dec;
+  bulk_ops->ctr_enc =  _gcry_3des_ctr_enc;
+
   tripledes_set3keys ( ctx, key, key+8, key+16);
 
   if (ctx->flags.no_weak_key)
@@ -1382,11 +1382,11 @@ do_tripledes_decrypt( void *context, byte *outbuf, const byte *inbuf )
 
 static gcry_err_code_t
 do_des_setkey (void *context, const byte *key, unsigned keylen,
-               gcry_cipher_hd_t hd)
+               cipher_bulk_ops_t *bulk_ops)
 {
   struct _des_ctx *ctx = (struct _des_ctx *) context;
 
-  (void)hd;
+  (void)bulk_ops;
 
   if (keylen != 8)
     return GPG_ERR_INV_KEYLEN;
diff --git a/cipher/gost28147.c b/cipher/gost28147.c
index 24385915..df16c3c6 100644
--- a/cipher/gost28147.c
+++ b/cipher/gost28147.c
@@ -48,12 +48,12 @@ gost_do_set_sbox (GOST28147_context *ctx, unsigned int index)
 
 static gcry_err_code_t
 gost_setkey (void *c, const byte *key, unsigned keylen,
-             gcry_cipher_hd_t hd)
+             cipher_bulk_ops_t *bulk_ops)
 {
   int i;
   GOST28147_context *ctx = c;
 
-  (void)hd;
+  (void)bulk_ops;
 
   if (keylen != 256 / 8)
     return GPG_ERR_INV_KEYLEN;
diff --git a/cipher/idea.c b/cipher/idea.c
index abfe6755..a503c08f 100644
--- a/cipher/idea.c
+++ b/cipher/idea.c
@@ -259,11 +259,11 @@ do_setkey( IDEA_context *c, const byte *key, unsigned int keylen )
 
 static gcry_err_code_t
 idea_setkey (void *context, const byte *key, unsigned int keylen,
-             gcry_cipher_hd_t hd)
+             cipher_bulk_ops_t *bulk_ops)
 {
     IDEA_context *ctx = context;
     int rc = do_setkey (ctx, key, keylen);
-    (void)hd;
+    (void)bulk_ops;
     _gcry_burn_stack (23+6*sizeof(void*));
     return rc;
 }
diff --git a/cipher/rfc2268.c b/cipher/rfc2268.c
index 09149462..348c4d41 100644
--- a/cipher/rfc2268.c
+++ b/cipher/rfc2268.c
@@ -263,9 +263,9 @@ setkey_core (void *context, const unsigned char *key, unsigned int keylen, int w
 
 static gpg_err_code_t
 do_setkey (void *context, const unsigned char *key, unsigned int keylen,
-           gcry_cipher_hd_t hd)
+           cipher_bulk_ops_t *bulk_ops)
 {
-  (void)hd;
+  (void)bulk_ops;
   return setkey_core (context, key, keylen, 1);
 }
 
diff --git a/cipher/rijndael.c b/cipher/rijndael.c
index d81ed809..ac70ba75 100644
--- a/cipher/rijndael.c
+++ b/cipher/rijndael.c
@@ -281,6 +281,29 @@ static unsigned int do_encrypt (const RIJNDAEL_context *ctx, unsigned char *bx,
 static unsigned int do_decrypt (const RIJNDAEL_context *ctx, unsigned char *bx,
                                 const unsigned char *ax);
 
+static void _gcry_aes_cfb_enc (void *context, unsigned char *iv,
+			       void *outbuf, const void *inbuf,
+			       size_t nblocks);
+static void _gcry_aes_cfb_dec (void *context, unsigned char *iv,
+			       void *outbuf_arg, const void *inbuf_arg,
+			       size_t nblocks);
+static void _gcry_aes_cbc_enc (void *context, unsigned char *iv,
+			       void *outbuf_arg, const void *inbuf_arg,
+			       size_t nblocks, int cbc_mac);
+static void _gcry_aes_cbc_dec (void *context, unsigned char *iv,
+			       void *outbuf_arg, const void *inbuf_arg,
+			       size_t nblocks);
+static void _gcry_aes_ctr_enc (void *context, unsigned char *ctr,
+			       void *outbuf_arg, const void *inbuf_arg,
+			       size_t nblocks);
+static size_t _gcry_aes_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg,
+				   const void *inbuf_arg, size_t nblocks,
+				   int encrypt);
+static size_t _gcry_aes_ocb_auth (gcry_cipher_hd_t c, const void *abuf_arg,
+				  size_t nblocks);
+static void _gcry_aes_xts_crypt (void *context, unsigned char *tweak,
+				 void *outbuf_arg, const void *inbuf_arg,
+				 size_t nblocks, int encrypt);
 
 
 /* All the numbers.  */
@@ -349,7 +372,7 @@ static void prefetch_dec(void)
 /* Perform the key setup.  */
 static gcry_err_code_t
 do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen,
-           gcry_cipher_hd_t hd)
+           cipher_bulk_ops_t *bulk_ops)
 {
   static int initialized = 0;
   static const char *selftest_failed = 0;
@@ -357,12 +380,7 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen,
   int rounds;
   int i,j, r, t, rconpointer = 0;
   int KC;
-#if defined(USE_AESNI) || defined(USE_PADLOCK) || defined(USE_SSSE3) \
-    || defined(USE_ARM_CE) || defined(USE_PPC_CRYPTO)
   unsigned int hwfeatures;
-#endif
-
-  (void)hd;
 
   /* The on-the-fly self tests are only run in non-fips mode. In fips
      mode explicit self-tests are required.  Actually the on-the-fly
@@ -400,11 +418,7 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen,
     return GPG_ERR_INV_KEYLEN;
 
   ctx->rounds = rounds;
-
-#if defined(USE_AESNI) || defined(USE_PADLOCK) || defined(USE_SSSE3) \
-    || defined(USE_ARM_CE) || defined(USE_PPC_CRYPTO)
   hwfeatures = _gcry_get_hw_features ();
-#endif
 
   ctx->decryption_prepared = 0;
 #ifdef USE_PADLOCK
@@ -426,6 +440,19 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen,
   ctx->use_ppc9le_crypto = 0;
 #endif
 
+  /* Setup default bulk encryption routines.  */
+  memset (bulk_ops, 0, sizeof(*bulk_ops));
+  bulk_ops->cfb_enc = _gcry_aes_cfb_enc;
+  bulk_ops->cfb_dec = _gcry_aes_cfb_dec;
+  bulk_ops->cbc_enc = _gcry_aes_cbc_enc;
+  bulk_ops->cbc_dec = _gcry_aes_cbc_dec;
+  bulk_ops->ctr_enc = _gcry_aes_ctr_enc;
+  bulk_ops->ocb_crypt = _gcry_aes_ocb_crypt;
+  bulk_ops->ocb_auth  = _gcry_aes_ocb_auth;
+  bulk_ops->xts_crypt = _gcry_aes_xts_crypt;
+
+  (void)hwfeatures;
+
   if (0)
     {
       ;
@@ -441,17 +468,16 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen,
       ctx->use_aesni = 1;
       ctx->use_avx = !!(hwfeatures & HWF_INTEL_AVX);
       ctx->use_avx2 = !!(hwfeatures & HWF_INTEL_AVX2);
-      if (hd)
-        {
-          hd->bulk.cfb_enc = _gcry_aes_aesni_cfb_enc;
-          hd->bulk.cfb_dec = _gcry_aes_aesni_cfb_dec;
-          hd->bulk.cbc_enc = _gcry_aes_aesni_cbc_enc;
-          hd->bulk.cbc_dec = _gcry_aes_aesni_cbc_dec;
-          hd->bulk.ctr_enc = _gcry_aes_aesni_ctr_enc;
-          hd->bulk.ocb_crypt = _gcry_aes_aesni_ocb_crypt;
-          hd->bulk.ocb_auth = _gcry_aes_aesni_ocb_auth;
-          hd->bulk.xts_crypt = _gcry_aes_aesni_xts_crypt;
-        }
+
+      /* Setup AES-NI bulk encryption routines.  */
+      bulk_ops->cfb_enc = _gcry_aes_aesni_cfb_enc;
+      bulk_ops->cfb_dec = _gcry_aes_aesni_cfb_dec;
+      bulk_ops->cbc_enc = _gcry_aes_aesni_cbc_enc;
+      bulk_ops->cbc_dec = _gcry_aes_aesni_cbc_dec;
+      bulk_ops->ctr_enc = _gcry_aes_aesni_ctr_enc;
+      bulk_ops->ocb_crypt = _gcry_aes_aesni_ocb_crypt;
+      bulk_ops->ocb_auth = _gcry_aes_aesni_ocb_auth;
+      bulk_ops->xts_crypt = _gcry_aes_aesni_xts_crypt;
     }
 #endif
 #ifdef USE_PADLOCK
@@ -474,16 +500,15 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen,
       ctx->prefetch_enc_fn = NULL;
       ctx->prefetch_dec_fn = NULL;
       ctx->use_ssse3 = 1;
-      if (hd)
-        {
-          hd->bulk.cfb_enc = _gcry_aes_ssse3_cfb_enc;
-          hd->bulk.cfb_dec = _gcry_aes_ssse3_cfb_dec;
-          hd->bulk.cbc_enc = _gcry_aes_ssse3_cbc_enc;
-          hd->bulk.cbc_dec = _gcry_aes_ssse3_cbc_dec;
-          hd->bulk.ctr_enc = _gcry_aes_ssse3_ctr_enc;
-          hd->bulk.ocb_crypt = _gcry_aes_ssse3_ocb_crypt;
-          hd->bulk.ocb_auth = _gcry_aes_ssse3_ocb_auth;
-        }
+
+      /* Setup SSSE3 bulk encryption routines.  */
+      bulk_ops->cfb_enc = _gcry_aes_ssse3_cfb_enc;
+      bulk_ops->cfb_dec = _gcry_aes_ssse3_cfb_dec;
+      bulk_ops->cbc_enc = _gcry_aes_ssse3_cbc_enc;
+      bulk_ops->cbc_dec = _gcry_aes_ssse3_cbc_dec;
+      bulk_ops->ctr_enc = _gcry_aes_ssse3_ctr_enc;
+      bulk_ops->ocb_crypt = _gcry_aes_ssse3_ocb_crypt;
+      bulk_ops->ocb_auth = _gcry_aes_ssse3_ocb_auth;
     }
 #endif
 #ifdef USE_ARM_CE
@@ -495,17 +520,16 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen,
       ctx->prefetch_enc_fn = NULL;
       ctx->prefetch_dec_fn = NULL;
       ctx->use_arm_ce = 1;
-      if (hd)
-        {
-          hd->bulk.cfb_enc = _gcry_aes_armv8_ce_cfb_enc;
-          hd->bulk.cfb_dec = _gcry_aes_armv8_ce_cfb_dec;
-          hd->bulk.cbc_enc = _gcry_aes_armv8_ce_cbc_enc;
-          hd->bulk.cbc_dec = _gcry_aes_armv8_ce_cbc_dec;
-          hd->bulk.ctr_enc = _gcry_aes_armv8_ce_ctr_enc;
-          hd->bulk.ocb_crypt = _gcry_aes_armv8_ce_ocb_crypt;
-          hd->bulk.ocb_auth = _gcry_aes_armv8_ce_ocb_auth;
-          hd->bulk.xts_crypt = _gcry_aes_armv8_ce_xts_crypt;
-        }
+
+      /* Setup ARM-CE bulk encryption routines.  */
+      bulk_ops->cfb_enc = _gcry_aes_armv8_ce_cfb_enc;
+      bulk_ops->cfb_dec = _gcry_aes_armv8_ce_cfb_dec;
+      bulk_ops->cbc_enc = _gcry_aes_armv8_ce_cbc_enc;
+      bulk_ops->cbc_dec = _gcry_aes_armv8_ce_cbc_dec;
+      bulk_ops->ctr_enc = _gcry_aes_armv8_ce_ctr_enc;
+      bulk_ops->ocb_crypt = _gcry_aes_armv8_ce_ocb_crypt;
+      bulk_ops->ocb_auth = _gcry_aes_armv8_ce_ocb_auth;
+      bulk_ops->xts_crypt = _gcry_aes_armv8_ce_xts_crypt;
     }
 #endif
 #ifdef USE_PPC_CRYPTO_WITH_PPC9LE
@@ -518,17 +542,16 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen,
       ctx->prefetch_dec_fn = NULL;
       ctx->use_ppc_crypto = 1; /* same key-setup as USE_PPC_CRYPTO */
       ctx->use_ppc9le_crypto = 1;
-      if (hd)
-        {
-          hd->bulk.cfb_enc = _gcry_aes_ppc9le_cfb_enc;
-          hd->bulk.cfb_dec = _gcry_aes_ppc9le_cfb_dec;
-          hd->bulk.cbc_enc = _gcry_aes_ppc9le_cbc_enc;
-          hd->bulk.cbc_dec = _gcry_aes_ppc9le_cbc_dec;
-          hd->bulk.ctr_enc = _gcry_aes_ppc9le_ctr_enc;
-          hd->bulk.ocb_crypt = _gcry_aes_ppc9le_ocb_crypt;
-          hd->bulk.ocb_auth = _gcry_aes_ppc9le_ocb_auth;
-          hd->bulk.xts_crypt = _gcry_aes_ppc9le_xts_crypt;
-        }
+
+      /* Setup PPC9LE bulk encryption routines.  */
+      bulk_ops->cfb_enc = _gcry_aes_ppc9le_cfb_enc;
+      bulk_ops->cfb_dec = _gcry_aes_ppc9le_cfb_dec;
+      bulk_ops->cbc_enc = _gcry_aes_ppc9le_cbc_enc;
+      bulk_ops->cbc_dec = _gcry_aes_ppc9le_cbc_dec;
+      bulk_ops->ctr_enc = _gcry_aes_ppc9le_ctr_enc;
+      bulk_ops->ocb_crypt = _gcry_aes_ppc9le_ocb_crypt;
+      bulk_ops->ocb_auth = _gcry_aes_ppc9le_ocb_auth;
+      bulk_ops->xts_crypt = _gcry_aes_ppc9le_xts_crypt;
     }
 #endif
 #ifdef USE_PPC_CRYPTO
@@ -540,17 +563,16 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen,
       ctx->prefetch_enc_fn = NULL;
       ctx->prefetch_dec_fn = NULL;
       ctx->use_ppc_crypto = 1;
-      if (hd)
-        {
-          hd->bulk.cfb_enc = _gcry_aes_ppc8_cfb_enc;
-          hd->bulk.cfb_dec = _gcry_aes_ppc8_cfb_dec;
-          hd->bulk.cbc_enc = _gcry_aes_ppc8_cbc_enc;
-          hd->bulk.cbc_dec = _gcry_aes_ppc8_cbc_dec;
-          hd->bulk.ctr_enc = _gcry_aes_ppc8_ctr_enc;
-          hd->bulk.ocb_crypt = _gcry_aes_ppc8_ocb_crypt;
-          hd->bulk.ocb_auth = _gcry_aes_ppc8_ocb_auth;
-          hd->bulk.xts_crypt = _gcry_aes_ppc8_xts_crypt;
-        }
+
+      /* Setup PPC8 bulk encryption routines.  */
+      bulk_ops->cfb_enc = _gcry_aes_ppc8_cfb_enc;
+      bulk_ops->cfb_dec = _gcry_aes_ppc8_cfb_dec;
+      bulk_ops->cbc_enc = _gcry_aes_ppc8_cbc_enc;
+      bulk_ops->cbc_dec = _gcry_aes_ppc8_cbc_dec;
+      bulk_ops->ctr_enc = _gcry_aes_ppc8_ctr_enc;
+      bulk_ops->ocb_crypt = _gcry_aes_ppc8_ocb_crypt;
+      bulk_ops->ocb_auth = _gcry_aes_ppc8_ocb_auth;
+      bulk_ops->xts_crypt = _gcry_aes_ppc8_xts_crypt;
     }
 #endif
   else
@@ -672,10 +694,10 @@ do_setkey (RIJNDAEL_context *ctx, const byte *key, const unsigned keylen,
 
 static gcry_err_code_t
 rijndael_setkey (void *context, const byte *key, const unsigned keylen,
-                 gcry_cipher_hd_t hd)
+                 cipher_bulk_ops_t *bulk_ops)
 {
   RIJNDAEL_context *ctx = context;
-  return do_setkey (ctx, key, keylen, hd);
+  return do_setkey (ctx, key, keylen, bulk_ops);
 }
 
 
@@ -943,7 +965,7 @@ rijndael_encrypt (void *context, byte *b, const byte *a)
    make sure that IV is aligned on an unsigned long boundary.  This
    function is only intended for the bulk encryption feature of
    cipher.c. */
-void
+static void
 _gcry_aes_cfb_enc (void *context, unsigned char *iv,
                    void *outbuf_arg, const void *inbuf_arg,
                    size_t nblocks)
@@ -1017,7 +1039,7 @@ _gcry_aes_cfb_enc (void *context, unsigned char *iv,
    make sure that IV is aligned on an unsigned long boundary.  This
    function is only intended for the bulk encryption feature of
    cipher.c. */
-void
+static void
 _gcry_aes_cbc_enc (void *context, unsigned char *iv,
                    void *outbuf_arg, const void *inbuf_arg,
                    size_t nblocks, int cbc_mac)
@@ -1100,7 +1122,7 @@ _gcry_aes_cbc_enc (void *context, unsigned char *iv,
    minimum alignment is for an u32.  This function is only intended
    for the bulk encryption feature of cipher.c.  CTR is expected to be
    of size BLOCKSIZE. */
-void
+static void
 _gcry_aes_ctr_enc (void *context, unsigned char *ctr,
                    void *outbuf_arg, const void *inbuf_arg,
                    size_t nblocks)
@@ -1357,7 +1379,7 @@ rijndael_decrypt (void *context, byte *b, const byte *a)
    make sure that IV is aligned on an unsigned long boundary.  This
    function is only intended for the bulk encryption feature of
    cipher.c. */
-void
+static void
 _gcry_aes_cfb_dec (void *context, unsigned char *iv,
                    void *outbuf_arg, const void *inbuf_arg,
                    size_t nblocks)
@@ -1429,7 +1451,7 @@ _gcry_aes_cfb_dec (void *context, unsigned char *iv,
    make sure that IV is aligned on an unsigned long boundary.  This
    function is only intended for the bulk encryption feature of
    cipher.c. */
-void
+static void
 _gcry_aes_cbc_dec (void *context, unsigned char *iv,
                    void *outbuf_arg, const void *inbuf_arg,
                    size_t nblocks)
@@ -1508,7 +1530,7 @@ _gcry_aes_cbc_dec (void *context, unsigned char *iv,
 
 
 /* Bulk encryption/decryption of complete blocks in OCB mode. */
-size_t
+static size_t
 _gcry_aes_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg,
                      const void *inbuf_arg, size_t nblocks, int encrypt)
 {
@@ -1616,7 +1638,7 @@ _gcry_aes_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg,
 
 
 /* Bulk authentication of complete blocks in OCB mode. */
-size_t
+static size_t
 _gcry_aes_ocb_auth (gcry_cipher_hd_t c, const void *abuf_arg, size_t nblocks)
 {
   RIJNDAEL_context *ctx = (void *)&c->context.c;
@@ -1690,7 +1712,7 @@ _gcry_aes_ocb_auth (gcry_cipher_hd_t c, const void *abuf_arg, size_t nblocks)
 
 
 /* Bulk encryption/decryption of complete blocks in XTS mode. */
-void
+static void
 _gcry_aes_xts_crypt (void *context, unsigned char *tweak,
 		     void *outbuf_arg, const void *inbuf_arg,
 		     size_t nblocks, int encrypt)
@@ -1797,6 +1819,7 @@ selftest_basic_128 (void)
   RIJNDAEL_context *ctx;
   unsigned char *ctxmem;
   unsigned char scratch[16];
+  cipher_bulk_ops_t bulk_ops;
 
   /* The test vectors are from the AES supplied ones; more or less
      randomly taken from ecb_tbl.txt (I=42,81,14) */
@@ -1844,7 +1867,7 @@ selftest_basic_128 (void)
   if (!ctx)
     return "failed to allocate memory";
 
-  rijndael_setkey (ctx, key_128, sizeof (key_128), NULL);
+  rijndael_setkey (ctx, key_128, sizeof (key_128), &bulk_ops);
   rijndael_encrypt (ctx, scratch, plaintext_128);
   if (memcmp (scratch, ciphertext_128, sizeof (ciphertext_128)))
     {
@@ -1866,6 +1889,7 @@ selftest_basic_192 (void)
   RIJNDAEL_context *ctx;
   unsigned char *ctxmem;
   unsigned char scratch[16];
+  cipher_bulk_ops_t bulk_ops;
 
   static unsigned char plaintext_192[16] =
     {
@@ -1887,7 +1911,7 @@ selftest_basic_192 (void)
   ctx = _gcry_cipher_selftest_alloc_ctx (sizeof *ctx, &ctxmem);
   if (!ctx)
     return "failed to allocate memory";
-  rijndael_setkey (ctx, key_192, sizeof(key_192), NULL);
+  rijndael_setkey (ctx, key_192, sizeof(key_192), &bulk_ops);
   rijndael_encrypt (ctx, scratch, plaintext_192);
   if (memcmp (scratch, ciphertext_192, sizeof (ciphertext_192)))
     {
@@ -1910,6 +1934,7 @@ selftest_basic_256 (void)
   RIJNDAEL_context *ctx;
   unsigned char *ctxmem;
   unsigned char scratch[16];
+  cipher_bulk_ops_t bulk_ops;
 
   static unsigned char plaintext_256[16] =
     {
@@ -1932,7 +1957,7 @@ selftest_basic_256 (void)
   ctx = _gcry_cipher_selftest_alloc_ctx (sizeof *ctx, &ctxmem);
   if (!ctx)
     return "failed to allocate memory";
-  rijndael_setkey (ctx, key_256, sizeof(key_256), NULL);
+  rijndael_setkey (ctx, key_256, sizeof(key_256), &bulk_ops);
   rijndael_encrypt (ctx, scratch, plaintext_256);
   if (memcmp (scratch, ciphertext_256, sizeof (ciphertext_256)))
     {
@@ -1958,8 +1983,7 @@ selftest_ctr_128 (void)
   const int context_size = sizeof(RIJNDAEL_context);
 
   return _gcry_selftest_helper_ctr("AES", &rijndael_setkey,
-           &rijndael_encrypt, &_gcry_aes_ctr_enc, nblocks, blocksize,
-	   context_size);
+           &rijndael_encrypt, nblocks, blocksize, context_size);
 }
 
 
@@ -1973,8 +1997,7 @@ selftest_cbc_128 (void)
   const int context_size = sizeof(RIJNDAEL_context);
 
   return _gcry_selftest_helper_cbc("AES", &rijndael_setkey,
-           &rijndael_encrypt, &_gcry_aes_cbc_dec, nblocks, blocksize,
-	   context_size);
+           &rijndael_encrypt, nblocks, blocksize, context_size);
 }
 
 
@@ -1988,8 +2011,7 @@ selftest_cfb_128 (void)
   const int context_size = sizeof(RIJNDAEL_context);
 
   return _gcry_selftest_helper_cfb("AES", &rijndael_setkey,
-           &rijndael_encrypt, &_gcry_aes_cfb_dec, nblocks, blocksize,
-	   context_size);
+           &rijndael_encrypt, nblocks, blocksize, context_size);
 }
 
 
diff --git a/cipher/salsa20.c b/cipher/salsa20.c
index 5c5e2b54..58e2dc6f 100644
--- a/cipher/salsa20.c
+++ b/cipher/salsa20.c
@@ -367,11 +367,11 @@ salsa20_do_setkey (SALSA20_context_t *ctx,
 
 static gcry_err_code_t
 salsa20_setkey (void *context, const byte *key, unsigned int keylen,
-                gcry_cipher_hd_t hd)
+                cipher_bulk_ops_t *bulk_ops)
 {
   SALSA20_context_t *ctx = (SALSA20_context_t *)context;
   gcry_err_code_t rc = salsa20_do_setkey (ctx, key, keylen);
-  (void)hd;
+  (void)bulk_ops;
   _gcry_burn_stack (4 + sizeof (void *) + 4 * sizeof (void *));
   return rc;
 }
diff --git a/cipher/seed.c b/cipher/seed.c
index e36d3cf9..c8ae4a01 100644
--- a/cipher/seed.c
+++ b/cipher/seed.c
@@ -310,11 +310,11 @@ do_setkey (SEED_context *ctx, const byte *key, const unsigned keylen)
 
 static gcry_err_code_t
 seed_setkey (void *context, const byte *key, const unsigned keylen,
-             gcry_cipher_hd_t hd)
+             cipher_bulk_ops_t *bulk_ops)
 {
   SEED_context *ctx = context;
   int rc = do_setkey (ctx, key, keylen);
-  (void)hd;
+  (void)bulk_ops;
   _gcry_burn_stack (4*6 + sizeof(void*)*2 + sizeof(int)*2);
   return rc;
 }
diff --git a/cipher/serpent.c b/cipher/serpent.c
index 71d843d0..3c5eed2c 100644
--- a/cipher/serpent.c
+++ b/cipher/serpent.c
@@ -222,9 +222,24 @@ extern void _gcry_serpent_neon_ocb_auth(serpent_context_t *ctx,
 #endif
 
 
-/* A prototype.  */
+/* Prototypes.  */
 static const char *serpent_test (void);
 
+static void _gcry_serpent_ctr_enc (void *context, unsigned char *ctr,
+				   void *outbuf_arg, const void *inbuf_arg,
+				   size_t nblocks);
+static void _gcry_serpent_cbc_dec (void *context, unsigned char *iv,
+				   void *outbuf_arg, const void *inbuf_arg,
+				   size_t nblocks);
+static void _gcry_serpent_cfb_dec (void *context, unsigned char *iv,
+				   void *outbuf_arg, const void *inbuf_arg,
+				   size_t nblocks);
+static size_t _gcry_serpent_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg,
+				       const void *inbuf_arg, size_t nblocks,
+				       int encrypt);
+static size_t _gcry_serpent_ocb_auth (gcry_cipher_hd_t c, const void *abuf_arg,
+				      size_t nblocks);
+
 
 /*
  * These are the S-Boxes of Serpent from following research paper.
@@ -749,15 +764,13 @@ serpent_setkey_internal (serpent_context_t *context,
 static gcry_err_code_t
 serpent_setkey (void *ctx,
 		const byte *key, unsigned int key_length,
-                gcry_cipher_hd_t hd)
+                cipher_bulk_ops_t *bulk_ops)
 {
   serpent_context_t *context = ctx;
   static const char *serpent_test_ret;
   static int serpent_init_done;
   gcry_err_code_t ret = GPG_ERR_NO_ERROR;
 
-  (void)hd;
-
   if (! serpent_init_done)
     {
       /* Execute a self-test the first time, Serpent is used.  */
@@ -767,6 +780,14 @@ serpent_setkey (void *ctx,
 	log_error ("Serpent test failure: %s\n", serpent_test_ret);
     }
 
+  /* Setup bulk encryption routines.  */
+  memset (bulk_ops, 0, sizeof(*bulk_ops));
+  bulk_ops->cbc_dec = _gcry_serpent_cbc_dec;
+  bulk_ops->cfb_dec = _gcry_serpent_cfb_dec;
+  bulk_ops->ctr_enc = _gcry_serpent_ctr_enc;
+  bulk_ops->ocb_crypt = _gcry_serpent_ocb_crypt;
+  bulk_ops->ocb_auth  = _gcry_serpent_ocb_auth;
+
   if (serpent_test_ret)
     ret = GPG_ERR_SELFTEST_FAILED;
   else
@@ -902,7 +923,7 @@ serpent_decrypt (void *ctx, byte *buffer_out, const byte *buffer_in)
 /* Bulk encryption of complete blocks in CTR mode.  This function is only
    intended for the bulk encryption feature of cipher.c.  CTR is expected to be
    of size sizeof(serpent_block_t). */
-void
+static void
 _gcry_serpent_ctr_enc(void *context, unsigned char *ctr,
                       void *outbuf_arg, const void *inbuf_arg,
                       size_t nblocks)
@@ -1014,7 +1035,7 @@ _gcry_serpent_ctr_enc(void *context, unsigned char *ctr,
 
 /* Bulk decryption of complete blocks in CBC mode.  This function is only
    intended for the bulk encryption feature of cipher.c. */
-void
+static void
 _gcry_serpent_cbc_dec(void *context, unsigned char *iv,
                       void *outbuf_arg, const void *inbuf_arg,
                       size_t nblocks)
@@ -1123,7 +1144,7 @@ _gcry_serpent_cbc_dec(void *context, unsigned char *iv,
 
 /* Bulk decryption of complete blocks in CFB mode.  This function is only
    intended for the bulk encryption feature of cipher.c. */
-void
+static void
 _gcry_serpent_cfb_dec(void *context, unsigned char *iv,
                       void *outbuf_arg, const void *inbuf_arg,
                       size_t nblocks)
@@ -1225,7 +1246,7 @@ _gcry_serpent_cfb_dec(void *context, unsigned char *iv,
 }
 
 /* Bulk encryption/decryption of complete blocks in OCB mode. */
-size_t
+static size_t
 _gcry_serpent_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg,
 			const void *inbuf_arg, size_t nblocks, int encrypt)
 {
@@ -1412,7 +1433,7 @@ _gcry_serpent_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg,
 }
 
 /* Bulk authentication of complete blocks in OCB mode. */
-size_t
+static size_t
 _gcry_serpent_ocb_auth (gcry_cipher_hd_t c, const void *abuf_arg,
 			size_t nblocks)
 {
@@ -1592,8 +1613,7 @@ selftest_ctr_128 (void)
   const int context_size = sizeof(serpent_context_t);
 
   return _gcry_selftest_helper_ctr("SERPENT", &serpent_setkey,
-           &serpent_encrypt, &_gcry_serpent_ctr_enc, nblocks, blocksize,
-	   context_size);
+           &serpent_encrypt, nblocks, blocksize, context_size);
 }
 
 
@@ -1607,8 +1627,7 @@ selftest_cbc_128 (void)
   const int context_size = sizeof(serpent_context_t);
 
   return _gcry_selftest_helper_cbc("SERPENT", &serpent_setkey,
-           &serpent_encrypt, &_gcry_serpent_cbc_dec, nblocks, blocksize,
-	   context_size);
+           &serpent_encrypt, nblocks, blocksize, context_size);
 }
 
 
@@ -1622,8 +1641,7 @@ selftest_cfb_128 (void)
   const int context_size = sizeof(serpent_context_t);
 
   return _gcry_selftest_helper_cfb("SERPENT", &serpent_setkey,
-           &serpent_encrypt, &_gcry_serpent_cfb_dec, nblocks, blocksize,
-	   context_size);
+           &serpent_encrypt, nblocks, blocksize, context_size);
 }
 
 
diff --git a/cipher/sm4.c b/cipher/sm4.c
index 75f2e412..c8dd0406 100644
--- a/cipher/sm4.c
+++ b/cipher/sm4.c
@@ -69,6 +69,21 @@
 
 static const char *sm4_selftest (void);
 
+static void _gcry_sm4_ctr_enc (void *context, unsigned char *ctr,
+			       void *outbuf_arg, const void *inbuf_arg,
+			       size_t nblocks);
+static void _gcry_sm4_cbc_dec (void *context, unsigned char *iv,
+			       void *outbuf_arg, const void *inbuf_arg,
+			       size_t nblocks);
+static void _gcry_sm4_cfb_dec (void *context, unsigned char *iv,
+			       void *outbuf_arg, const void *inbuf_arg,
+			       size_t nblocks);
+static size_t _gcry_sm4_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg,
+				   const void *inbuf_arg, size_t nblocks,
+				   int encrypt);
+static size_t _gcry_sm4_ocb_auth (gcry_cipher_hd_t c, const void *abuf_arg,
+				  size_t nblocks);
+
 typedef struct
 {
   u32 rkey_enc[32];
@@ -329,14 +344,13 @@ sm4_expand_key (SM4_context *ctx, const byte *key)
 
 static gcry_err_code_t
 sm4_setkey (void *context, const byte *key, const unsigned keylen,
-            gcry_cipher_hd_t hd)
+            cipher_bulk_ops_t *bulk_ops)
 {
   SM4_context *ctx = context;
   static int init = 0;
   static const char *selftest_failed = NULL;
   unsigned int hwf = _gcry_get_hw_features ();
 
-  (void)hd;
   (void)hwf;
 
   if (!init)
@@ -359,6 +373,14 @@ sm4_setkey (void *context, const byte *key, const unsigned keylen,
   ctx->use_aesni_avx2 = (hwf & HWF_INTEL_AESNI) && (hwf & HWF_INTEL_AVX2);
 #endif
 
+  /* Setup bulk encryption routines.  */
+  memset (bulk_ops, 0, sizeof(*bulk_ops));
+  bulk_ops->cbc_dec = _gcry_sm4_cbc_dec;
+  bulk_ops->cfb_dec = _gcry_sm4_cfb_dec;
+  bulk_ops->ctr_enc = _gcry_sm4_ctr_enc;
+  bulk_ops->ocb_crypt = _gcry_sm4_ocb_crypt;
+  bulk_ops->ocb_auth  = _gcry_sm4_ocb_auth;
+
   sm4_expand_key (ctx, key);
   return 0;
 }
@@ -491,7 +513,7 @@ sm4_crypt_blocks (const u32 *rk, byte *out, const byte *in,
 /* Bulk encryption of complete blocks in CTR mode.  This function is only
    intended for the bulk encryption feature of cipher.c.  CTR is expected to be
    of size 16. */
-void
+static void
 _gcry_sm4_ctr_enc(void *context, unsigned char *ctr,
                   void *outbuf_arg, const void *inbuf_arg,
                   size_t nblocks)
@@ -592,7 +614,7 @@ _gcry_sm4_ctr_enc(void *context, unsigned char *ctr,
 
 /* Bulk decryption of complete blocks in CBC mode.  This function is only
    intended for the bulk encryption feature of cipher.c. */
-void
+static void
 _gcry_sm4_cbc_dec(void *context, unsigned char *iv,
                   void *outbuf_arg, const void *inbuf_arg,
                   size_t nblocks)
@@ -686,7 +708,7 @@ _gcry_sm4_cbc_dec(void *context, unsigned char *iv,
 
 /* Bulk decryption of complete blocks in CFB mode.  This function is only
    intended for the bulk encryption feature of cipher.c. */
-void
+static void
 _gcry_sm4_cfb_dec(void *context, unsigned char *iv,
                   void *outbuf_arg, const void *inbuf_arg,
                   size_t nblocks)
@@ -783,7 +805,7 @@ _gcry_sm4_cfb_dec(void *context, unsigned char *iv,
 }
 
 /* Bulk encryption/decryption of complete blocks in OCB mode. */
-size_t
+static size_t
 _gcry_sm4_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg,
 		     const void *inbuf_arg, size_t nblocks, int encrypt)
 {
@@ -955,7 +977,7 @@ _gcry_sm4_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg,
 }
 
 /* Bulk authentication of complete blocks in OCB mode. */
-size_t
+static size_t
 _gcry_sm4_ocb_auth (gcry_cipher_hd_t c, const void *abuf_arg, size_t nblocks)
 {
   SM4_context *ctx = (void *)&c->context.c;
@@ -1112,8 +1134,7 @@ selftest_ctr_128 (void)
   const int context_size = sizeof(SM4_context);
 
   return _gcry_selftest_helper_ctr("SM4", &sm4_setkey,
-           &sm4_encrypt, &_gcry_sm4_ctr_enc, nblocks, blocksize,
-	   context_size);
+           &sm4_encrypt, nblocks, blocksize, context_size);
 }
 
 /* Run the self-tests for SM4-CBC, tests bulk CBC decryption.
@@ -1126,8 +1147,7 @@ selftest_cbc_128 (void)
   const int context_size = sizeof(SM4_context);
 
   return _gcry_selftest_helper_cbc("SM4", &sm4_setkey,
-           &sm4_encrypt, &_gcry_sm4_cbc_dec, nblocks, blocksize,
-	   context_size);
+           &sm4_encrypt, nblocks, blocksize, context_size);
 }
 
 /* Run the self-tests for SM4-CFB, tests bulk CFB decryption.
@@ -1140,8 +1160,7 @@ selftest_cfb_128 (void)
   const int context_size = sizeof(SM4_context);
 
   return _gcry_selftest_helper_cfb("SM4", &sm4_setkey,
-           &sm4_encrypt, &_gcry_sm4_cfb_dec, nblocks, blocksize,
-	   context_size);
+           &sm4_encrypt, nblocks, blocksize, context_size);
 }
 
 static const char *
diff --git a/cipher/twofish.c b/cipher/twofish.c
index 417d7378..d19e0790 100644
--- a/cipher/twofish.c
+++ b/cipher/twofish.c
@@ -85,6 +85,24 @@
 /* Prototype for the self-test function. */
 static const char *selftest(void);
 
+
+/* Prototypes for the bulk functions. */
+static void _gcry_twofish_ctr_enc (void *context, unsigned char *ctr,
+				   void *outbuf_arg, const void *inbuf_arg,
+				   size_t nblocks);
+static void _gcry_twofish_cbc_dec (void *context, unsigned char *iv,
+				   void *outbuf_arg, const void *inbuf_arg,
+				   size_t nblocks);
+static void _gcry_twofish_cfb_dec (void *context, unsigned char *iv,
+				   void *outbuf_arg, const void *inbuf_arg,
+				   size_t nblocks);
+static size_t _gcry_twofish_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg,
+				       const void *inbuf_arg, size_t nblocks,
+				       int encrypt);
+static size_t _gcry_twofish_ocb_auth (gcry_cipher_hd_t c, const void *abuf_arg,
+				      size_t nblocks);
+
+
 /* Structure for an expanded Twofish key.  s contains the key-dependent
  * S-boxes composed with the MDS matrix; w contains the eight "whitening"
  * subkeys, K[0] through K[7].	k holds the remaining, "round" subkeys.  Note
@@ -735,14 +753,12 @@ do_twofish_setkey (TWOFISH_context *ctx, const byte *key, const unsigned keylen)
 
 static gcry_err_code_t
 twofish_setkey (void *context, const byte *key, unsigned int keylen,
-                gcry_cipher_hd_t hd)
+                cipher_bulk_ops_t *bulk_ops)
 {
   TWOFISH_context *ctx = context;
   unsigned int hwfeatures = _gcry_get_hw_features ();
   int rc;
 
-  (void)hd;
-
   rc = do_twofish_setkey (ctx, key, keylen);
 
 #ifdef USE_AVX2
@@ -753,6 +769,14 @@ twofish_setkey (void *context, const byte *key, unsigned int keylen,
     }
 #endif
 
+  /* Setup bulk encryption routines.  */
+  memset (bulk_ops, 0, sizeof(*bulk_ops));
+  bulk_ops->cbc_dec = _gcry_twofish_cbc_dec;
+  bulk_ops->cfb_dec = _gcry_twofish_cfb_dec;
+  bulk_ops->ctr_enc = _gcry_twofish_ctr_enc;
+  bulk_ops->ocb_crypt = _gcry_twofish_ocb_crypt;
+  bulk_ops->ocb_auth  = _gcry_twofish_ocb_auth;
+
   (void)hwfeatures;
 
   _gcry_burn_stack (23+6*sizeof(void*));
@@ -1096,7 +1120,7 @@ twofish_decrypt (void *context, byte *out, const byte *in)
 /* Bulk encryption of complete blocks in CTR mode.  This function is only
    intended for the bulk encryption feature of cipher.c.  CTR is expected to be
    of size TWOFISH_BLOCKSIZE. */
-void
+static void
 _gcry_twofish_ctr_enc(void *context, unsigned char *ctr, void *outbuf_arg,
 		      const void *inbuf_arg, size_t nblocks)
 {
@@ -1174,7 +1198,7 @@ _gcry_twofish_ctr_enc(void *context, unsigned char *ctr, void *outbuf_arg,
 
 /* Bulk decryption of complete blocks in CBC mode.  This function is only
    intended for the bulk encryption feature of cipher.c. */
-void
+static void
 _gcry_twofish_cbc_dec(void *context, unsigned char *iv, void *outbuf_arg,
 		      const void *inbuf_arg, size_t nblocks)
 {
@@ -1249,7 +1273,7 @@ _gcry_twofish_cbc_dec(void *context, unsigned char *iv, void *outbuf_arg,
 
 /* Bulk decryption of complete blocks in CFB mode.  This function is only
    intended for the bulk encryption feature of cipher.c. */
-void
+static void
 _gcry_twofish_cfb_dec(void *context, unsigned char *iv, void *outbuf_arg,
 		    const void *inbuf_arg, size_t nblocks)
 {
@@ -1318,7 +1342,7 @@ _gcry_twofish_cfb_dec(void *context, unsigned char *iv, void *outbuf_arg,
 }
 
 /* Bulk encryption/decryption of complete blocks in OCB mode. */
-size_t
+static size_t
 _gcry_twofish_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg,
 			const void *inbuf_arg, size_t nblocks, int encrypt)
 {
@@ -1432,7 +1456,7 @@ _gcry_twofish_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg,
 }
 
 /* Bulk authentication of complete blocks in OCB mode. */
-size_t
+static size_t
 _gcry_twofish_ocb_auth (gcry_cipher_hd_t c, const void *abuf_arg,
 			size_t nblocks)
 {
@@ -1546,8 +1570,7 @@ selftest_ctr (void)
   const int context_size = sizeof(TWOFISH_context);
 
   return _gcry_selftest_helper_ctr("TWOFISH", &twofish_setkey,
-           &twofish_encrypt, &_gcry_twofish_ctr_enc, nblocks, blocksize,
-	   context_size);
+           &twofish_encrypt, nblocks, blocksize, context_size);
 }
 
 /* Run the self-tests for TWOFISH-CBC, tests bulk CBC decryption.
@@ -1560,8 +1583,7 @@ selftest_cbc (void)
   const int context_size = sizeof(TWOFISH_context);
 
   return _gcry_selftest_helper_cbc("TWOFISH", &twofish_setkey,
-           &twofish_encrypt, &_gcry_twofish_cbc_dec, nblocks, blocksize,
-	   context_size);
+           &twofish_encrypt, nblocks, blocksize, context_size);
 }
 
 /* Run the self-tests for TWOFISH-CFB, tests bulk CBC decryption.
@@ -1574,8 +1596,7 @@ selftest_cfb (void)
   const int context_size = sizeof(TWOFISH_context);
 
   return _gcry_selftest_helper_cfb("TWOFISH", &twofish_setkey,
-           &twofish_encrypt, &_gcry_twofish_cfb_dec, nblocks, blocksize,
-	   context_size);
+           &twofish_encrypt, nblocks, blocksize, context_size);
 }
 
 
@@ -1585,7 +1606,8 @@ static const char*
 selftest (void)
 {
   TWOFISH_context ctx; /* Expanded key. */
-  byte scratch[16];	/* Encryption/decryption result buffer. */
+  byte scratch[16];    /* Encryption/decryption result buffer. */
+  cipher_bulk_ops_t bulk_ops;
   const char *r;
 
   /* Test vectors for single encryption/decryption.  Note that I am using
@@ -1620,7 +1642,7 @@ selftest (void)
     0x05, 0x93, 0x1C, 0xB6, 0xD4, 0x08, 0xE7, 0xFA
   };
 
-  twofish_setkey (&ctx, key, sizeof(key), NULL);
+  twofish_setkey (&ctx, key, sizeof(key), &bulk_ops);
   twofish_encrypt (&ctx, scratch, plaintext);
   if (memcmp (scratch, ciphertext, sizeof (ciphertext)))
     return "Twofish-128 test encryption failed.";
@@ -1628,7 +1650,7 @@ selftest (void)
   if (memcmp (scratch, plaintext, sizeof (plaintext)))
     return "Twofish-128 test decryption failed.";
 
-  twofish_setkey (&ctx, key_256, sizeof(key_256), NULL);
+  twofish_setkey (&ctx, key_256, sizeof(key_256), &bulk_ops);
   twofish_encrypt (&ctx, scratch, plaintext_256);
   if (memcmp (scratch, ciphertext_256, sizeof (ciphertext_256)))
     return "Twofish-256 test encryption failed.";
@@ -1663,7 +1685,8 @@ int
 main()
 {
   TWOFISH_context ctx;     /* Expanded key. */
-  int i, j;		    /* Loop counters. */
+  int i, j;                /* Loop counters. */
+  cipher_bulk_ops_t bulk_ops;
 
   const char *encrypt_msg; /* Message to print regarding encryption test;
                             * the printf is done outside the loop to avoid
@@ -1710,13 +1733,13 @@ main()
   /* Encryption test. */
   for (i = 0; i < 125; i++)
     {
-      twofish_setkey (&ctx, buffer[0], sizeof (buffer[0]), NULL);
+      twofish_setkey (&ctx, buffer[0], sizeof (buffer[0]), &bulk_ops);
       for (j = 0; j < 1000; j++)
         twofish_encrypt (&ctx, buffer[2], buffer[2]);
-      twofish_setkey (&ctx, buffer[1], sizeof (buffer[1]), NULL);
+      twofish_setkey (&ctx, buffer[1], sizeof (buffer[1]), &bulk_ops);
       for (j = 0; j < 1000; j++)
         twofish_encrypt (&ctx, buffer[3], buffer[3]);
-      twofish_setkey (&ctx, buffer[2], sizeof (buffer[2])*2, NULL);
+      twofish_setkey (&ctx, buffer[2], sizeof (buffer[2])*2, &bulk_ops);
       for (j = 0; j < 1000; j++) {
         twofish_encrypt (&ctx, buffer[0], buffer[0]);
         twofish_encrypt (&ctx, buffer[1], buffer[1]);
@@ -1728,15 +1751,15 @@ main()
   /* Decryption test. */
   for (i = 0; i < 125; i++)
     {
-      twofish_setkey (&ctx, buffer[2], sizeof (buffer[2])*2, NULL);
+      twofish_setkey (&ctx, buffer[2], sizeof (buffer[2])*2, &bulk_ops);
       for (j = 0; j < 1000; j++) {
         twofish_decrypt (&ctx, buffer[0], buffer[0]);
         twofish_decrypt (&ctx, buffer[1], buffer[1]);
       }
-      twofish_setkey (&ctx, buffer[1], sizeof (buffer[1]), NULL);
+      twofish_setkey (&ctx, buffer[1], sizeof (buffer[1]), &bulk_ops);
       for (j = 0; j < 1000; j++)
         twofish_decrypt (&ctx, buffer[3], buffer[3]);
-      twofish_setkey (&ctx, buffer[0], sizeof (buffer[0]), NULL);
+      twofish_setkey (&ctx, buffer[0], sizeof (buffer[0]), &bulk_ops);
       for (j = 0; j < 1000; j++)
         twofish_decrypt (&ctx, buffer[2], buffer[2]);
     }
diff --git a/src/cipher-proto.h b/src/cipher-proto.h
index 97eb0d9a..d87559ed 100644
--- a/src/cipher-proto.h
+++ b/src/cipher-proto.h
@@ -129,11 +129,13 @@ typedef struct gcry_pk_spec
  *
  */
 
+typedef struct cipher_bulk_ops cipher_bulk_ops_t;
+
 /* Type for the cipher_setkey function.  */
 typedef gcry_err_code_t (*gcry_cipher_setkey_t) (void *c,
 						 const unsigned char *key,
 						 unsigned keylen,
-						 gcry_cipher_hd_t hd);
+						 cipher_bulk_ops_t *bulk_ops);
 
 /* Type for the cipher_encrypt function.  */
 typedef unsigned int (*gcry_cipher_encrypt_t) (void *c,
diff --git a/src/cipher.h b/src/cipher.h
index f6b2151a..908e3b78 100644
--- a/src/cipher.h
+++ b/src/cipher.h
@@ -147,133 +147,6 @@ gcry_err_code_t _gcry_blake2_init_with_key(void *ctx, unsigned int flags,
 					   const unsigned char *key,
 					   size_t keylen, int algo);
 
-/*-- rijndael.c --*/
-void _gcry_aes_cfb_enc (void *context, unsigned char *iv,
-                        void *outbuf, const void *inbuf,
-                        size_t nblocks);
-void _gcry_aes_cfb_dec (void *context, unsigned char *iv,
-                        void *outbuf_arg, const void *inbuf_arg,
-                        size_t nblocks);
-void _gcry_aes_cbc_enc (void *context, unsigned char *iv,
-                        void *outbuf_arg, const void *inbuf_arg,
-                        size_t nblocks, int cbc_mac);
-void _gcry_aes_cbc_dec (void *context, unsigned char *iv,
-                        void *outbuf_arg, const void *inbuf_arg,
-                        size_t nblocks);
-void _gcry_aes_ctr_enc (void *context, unsigned char *ctr,
-                        void *outbuf_arg, const void *inbuf_arg,
-                        size_t nblocks);
-size_t _gcry_aes_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg,
-			    const void *inbuf_arg, size_t nblocks, int encrypt);
-size_t _gcry_aes_ocb_auth (gcry_cipher_hd_t c, const void *abuf_arg,
-			   size_t nblocks);
-void _gcry_aes_xts_crypt (void *context, unsigned char *tweak,
-			  void *outbuf_arg, const void *inbuf_arg,
-			  size_t nblocks, int encrypt);
-
-/*-- blowfish.c --*/
-void _gcry_blowfish_cfb_dec (void *context, unsigned char *iv,
-			     void *outbuf_arg, const void *inbuf_arg,
-			     size_t nblocks);
-
-void _gcry_blowfish_cbc_dec (void *context, unsigned char *iv,
-			     void *outbuf_arg, const void *inbuf_arg,
-			     size_t nblocks);
-
-void _gcry_blowfish_ctr_enc (void *context, unsigned char *ctr,
-			     void *outbuf_arg, const void *inbuf_arg,
-			     size_t nblocks);
-
-/*-- cast5.c --*/
-void _gcry_cast5_cfb_dec (void *context, unsigned char *iv,
-			  void *outbuf_arg, const void *inbuf_arg,
-			  size_t nblocks);
-
-void _gcry_cast5_cbc_dec (void *context, unsigned char *iv,
-			  void *outbuf_arg, const void *inbuf_arg,
-			  size_t nblocks);
-
-void _gcry_cast5_ctr_enc (void *context, unsigned char *ctr,
-			  void *outbuf_arg, const void *inbuf_arg,
-			  size_t nblocks);
-
-/*-- camellia-glue.c --*/
-void _gcry_camellia_ctr_enc (void *context, unsigned char *ctr,
-                             void *outbuf_arg, const void *inbuf_arg,
-                             size_t nblocks);
-void _gcry_camellia_cbc_dec (void *context, unsigned char *iv,
-                             void *outbuf_arg, const void *inbuf_arg,
-                             size_t nblocks);
-void _gcry_camellia_cfb_dec (void *context, unsigned char *iv,
-                             void *outbuf_arg, const void *inbuf_arg,
-                             size_t nblocks);
-size_t _gcry_camellia_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg,
-				 const void *inbuf_arg, size_t nblocks,
-				 int encrypt);
-size_t _gcry_camellia_ocb_auth (gcry_cipher_hd_t c, const void *abuf_arg,
-				size_t nblocks);
-
-/*-- des.c --*/
-void _gcry_3des_ctr_enc (void *context, unsigned char *ctr,
-                         void *outbuf_arg, const void *inbuf_arg,
-                         size_t nblocks);
-
-void _gcry_3des_cbc_dec (void *context, unsigned char *iv,
-                         void *outbuf_arg, const void *inbuf_arg,
-                         size_t nblocks);
-
-void _gcry_3des_cfb_dec (void *context, unsigned char *iv,
-                         void *outbuf_arg, const void *inbuf_arg,
-                         size_t nblocks);
-
-/*-- serpent.c --*/
-void _gcry_serpent_ctr_enc (void *context, unsigned char *ctr,
-                            void *outbuf_arg, const void *inbuf_arg,
-                            size_t nblocks);
-void _gcry_serpent_cbc_dec (void *context, unsigned char *iv,
-                            void *outbuf_arg, const void *inbuf_arg,
-                            size_t nblocks);
-void _gcry_serpent_cfb_dec (void *context, unsigned char *iv,
-                            void *outbuf_arg, const void *inbuf_arg,
-                            size_t nblocks);
-size_t _gcry_serpent_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg,
-				const void *inbuf_arg, size_t nblocks,
-				int encrypt);
-size_t _gcry_serpent_ocb_auth (gcry_cipher_hd_t c, const void *abuf_arg,
-			       size_t nblocks);
-
-/*-- sm4.c --*/
-void _gcry_sm4_ctr_enc (void *context, unsigned char *ctr,
-			void *outbuf_arg, const void *inbuf_arg,
-			size_t nblocks);
-void _gcry_sm4_cbc_dec (void *context, unsigned char *iv,
-                        void *outbuf_arg, const void *inbuf_arg,
-			size_t nblocks);
-void _gcry_sm4_cfb_dec (void *context, unsigned char *iv,
-			void *outbuf_arg, const void *inbuf_arg,
-			size_t nblocks);
-size_t _gcry_sm4_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg,
-			    const void *inbuf_arg, size_t nblocks,
-			    int encrypt);
-size_t _gcry_sm4_ocb_auth (gcry_cipher_hd_t c, const void *abuf_arg,
-			   size_t nblocks);
-
-/*-- twofish.c --*/
-void _gcry_twofish_ctr_enc (void *context, unsigned char *ctr,
-                            void *outbuf_arg, const void *inbuf_arg,
-                            size_t nblocks);
-void _gcry_twofish_cbc_dec (void *context, unsigned char *iv,
-                            void *outbuf_arg, const void *inbuf_arg,
-                            size_t nblocks);
-void _gcry_twofish_cfb_dec (void *context, unsigned char *iv,
-                            void *outbuf_arg, const void *inbuf_arg,
-                            size_t nblocks);
-size_t _gcry_twofish_ocb_crypt (gcry_cipher_hd_t c, void *outbuf_arg,
-				const void *inbuf_arg, size_t nblocks,
-				int encrypt);
-size_t _gcry_twofish_ocb_auth (gcry_cipher_hd_t c, const void *abuf_arg,
-			       size_t nblocks);
-
 /*-- dsa.c --*/
 void _gcry_register_pk_dsa_progress (gcry_handler_progress_t cbc, void *cb_data);
 
-- 
2.25.1