[PATCH v2] Disallow XOF algorithms for gcry_md_hash_buffers

[Peter Wu]

* cipher/md.c (_gcry_md_hash_buffer): Skip calculation for XOFs.
(_gcry_md_hash_buffers): Fail when XOFs are selected.
* doc/gcrypt.texi: Explicitly document above behavior for XOFs.
* tests/benchmark.c: Skip benchmarking hash functions without a fixed
output length.
--

Caught by UndefinedBehaviorSanitizer while running tests/benchmarks
where gcry_md_hash_buffer(GCRY_MD_SHAKE128) would result in
memcpy(digest, NULL, 0).

Signed-off-by: Peter Wu <peter at lekensteyn.nl>
---
 v2: just shortcircuit in gcry_md_hash_buffer for XOFs and reject the algo in
     gcry_md_hash_buffers. Adjust documentation and tests accordingly.

tests/benchmarks md SHAKE128 now shows "0ms" for the last column, but at least
the UB warning is gone.
---
 cipher/md.c       | 5 +++++
 doc/gcrypt.texi   | 4 +++-
 tests/benchmark.c | 2 ++
 3 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/cipher/md.c b/cipher/md.c
index 0414dcb..d1d0812 100644
--- a/cipher/md.c
+++ b/cipher/md.c
@@ -948,6 +948,8 @@ _gcry_md_hash_buffer (int algo, void *digest,
     _gcry_sha1_hash_buffer (digest, buffer, length);
   else if (algo == GCRY_MD_RMD160 && !fips_mode () )
     _gcry_rmd160_hash_buffer (digest, buffer, length);
+  else if (md_digest_length (algo) == 0)
+    ; /* Nothing to do for hashes without a fixed output length. */
   else
     {
       /* For the others we do not have a fast function, so we use the
@@ -1007,6 +1009,9 @@ _gcry_md_hash_buffers (int algo, unsigned int flags, void *digest,
   if (hmac && iovcnt < 1)
     return GPG_ERR_INV_ARG;
 
+  if (md_digest_length (algo) == 0)
+    return GPG_ERR_INV_ARG;
+
   if (algo == GCRY_MD_SHA1 && !hmac)
     _gcry_sha1_hash_buffers (digest, iov, iovcnt);
   else
diff --git a/doc/gcrypt.texi b/doc/gcrypt.texi
index 3265a70..9481be9 100644
--- a/doc/gcrypt.texi
+++ b/doc/gcrypt.texi
@@ -3378,6 +3378,7 @@ described by @var{iov} and @var{iovcnt}.  @var{digest} must be
 allocated by the caller, large enough to hold the message digest
 yielded by the the specified algorithm @var{algo}.  This required size
 may be obtained by using the function @code{gcry_md_get_algo_dlen}.
+This function cannot be used for extendable-output functions.
 
 @var{iov} is an array of buffer descriptions with @var{iovcnt} items.
 The caller should zero out the structures in this array and for each
@@ -3402,7 +3403,8 @@ immediately returns the message digest of the @var{length} bytes at
 @var{buffer}.  @var{digest} must be allocated by the caller, large
 enough to hold the message digest yielded by the the specified algorithm
 @var{algo}.  This required size may be obtained by using the function
- at code{gcry_md_get_algo_dlen}.
+ at code{gcry_md_get_algo_dlen}.  This function has no effect for
+extendable-output functions.
 
 Note that in contrast to @code{gcry_md_hash_buffers} this function
 will abort the process if an unavailable algorithm is used.
diff --git a/tests/benchmark.c b/tests/benchmark.c
index 53b83b1..5f94cc8 100644
--- a/tests/benchmark.c
+++ b/tests/benchmark.c
@@ -499,6 +499,8 @@ md_bench ( const char *algoname )
       for (i=1; i < 400; i++)
         if (in_fips_mode && i == GCRY_MD_MD5)
           ; /* Don't use MD5 in fips mode.  */
+        else if (gcry_md_get_algo_dlen (i) == 0)
+          ; /* Skip hash functions without a fixed output length. */
         else if ( !gcry_md_test_algo (i) )
           md_bench (gcry_md_algo_name (i));
       return;
-- 
2.7.4