gcry_mpi_invm succeeds if the inverse does not exist

[Guido Vranken]

I'm running my crypto library fuzzer on Google OSS-Fuzz and it keeps
finding bugs in gcry_mpi_invm.

jussi kivilinna receives these reports. Would you mind taking a look?

On Tue, Apr 14, 2020 at 8:32 AM NIIBE Yutaka <gniibe at fsij.org> wrote:

> Hello,
>
> Thank you for your report with a test case.  It helps a lot.
>
> Guido Vranken wrote:
> > This is not in accordance with the documentation:
> >
> > Function: int gcry_mpi_invm (gcry_mpi_t x, gcry_mpi_t a, gcry_mpi_t m)
> > Set x to the multiplicative inverse of a \bmod m. Return true if the
> > inverse exists.
>
> Indeed.  It seems that API was changed to return the result
> (success/failure) in libgcrypt 1.3.2 in 2007.  Before that, the API was
> void (no return value).  Since the initial change, it never returns
> correct value.
>
> I'll fix for 1.9.
> --
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.gnupg.org/pipermail/gcrypt-devel/attachments/20200508/85e78542/attachment.html>