wrong results and hang in gcry_mpi_invm()

[Hanno Böck]

Hi,

I want to report two issues with the function gcry_mpi_invm().

In many cases where there exists no inverse the function will not
detect that and produce a wrong result. A very simple example is
calculating the inverse of 2 mod 4.
gcry_mpi_inv will calculate the inverse to be 1, which is obviously
wrong. It will set the return value to 1, while it should set it to 0
if no inverse exists.

A second issue is that when one sets the modulus to zero the function
will hang forever. This may be seen as okay, because this is an invalid
input, but I think it would still be better to return an "inverse
doesn't exist" / 0 return code.

I have attached simple example files for both issues.

-- 
Hanno Böck
http://hboeck.de/

mail/jabber: hanno at hboeck.de
GPG: BBB51E42
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gcrypt-invm-hang.c
Type: text/x-c++src
Size: 351 bytes
Desc: not available
URL: </pipermail/attachments/20151207/5d36d757/attachment.c>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gcrypt-invm-wrongresult.c
Type: text/x-c++src
Size: 351 bytes
Desc: not available
URL: </pipermail/attachments/20151207/5d36d757/attachment-0001.c>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: </pipermail/attachments/20151207/5d36d757/attachment.sig>