gpgsm/cms: int_rsa_verify:wrong signature length

[Andreas Fenkart]

Hi,

I'm using GnuPG to sign 'swupdate' update images. They are verified on the
target using openssl:

  gpgsm -o sw-description.sig -sb sw-description

swupdate links against the openssl, but the equivalent cmd line is:

  openssl cms -verify -in sw-description.sig -inform DER -content
sw-description \
                -CAfile mycert.cert.pem -binary

This works for quite a while with no problem, just suddenly I see this error
when verifying

  140250102395072:error:04091077:rsa routines:int_rsa_verify:wrong
signature length:../crypto/rsa/rsa_sign.c:132:
  140250102395072:error:2E09809E:CMS
routines:CMS_SignerInfo_verify:verification
failure:../crypto/cms/cms_sd.c:738:

gpgsm verfication though works.

Comparing the asn1parse output of working/non-working signatures, I can see
that the non-working signature block is 1-byte shorter then the working version

$ diff /tmp/works /tmp/broken
76,77c76,77
<   929:d=3  hl=4 l= 561 cons: SET
<   933:d=4  hl=4 l= 557 cons: SEQUENCE
---
>   929:d=3  hl=4 l= 559 cons: SET
>   933:d=4  hl=4 l= 555 cons: SEQUENCE
105c105
<  1234:d=5  hl=4 l= 256 prim: OCTET STRING      [HEX
DUMP]:EF4378D3F5AFDE79974518722EA1B12F7DE7C71239B03676378A9B2AC7F5BB76AF91DFE05B004AA0C68CAD8CE54021C61CD49943BA513888D29A6B5AD41942FC57BCBFE1A1B68607EC875434119A28DC6C3463CE4C9B0C948E89C93CB18B7FFDBCDAB6467501E15CD5B603CAA8693DBF27B70B624F15E2BE0A1BD02EB26E619D2EC6A8A939BAB6CA169ABB6A0A76319AD9208728AE566312B63281D03B77B0A38A46FB63FFF5741D4484B14CBF46D092B42D3878AA333CA6CDF6A2412B4DA4DB2AD2DE401506D9B7C24B6CEC18160BC1CBF30426217C27C4CCFDB3B444BBE9C2C9A95D12925A77FA6E6795DE4FFE223C943F15823BC642483F0244A7551052AE
<  1494:d=3  hl=2 l=   0 prim: EOC
<  1496:d=2  hl=2 l=   0 prim: EOC
<  1498:d=1  hl=2 l=   0 prim: EOC
---
>  1234:d=5  hl=3 l= 255 prim: OCTET STRING      [HEX DUMP]:AE0DAC25E87228FF787FD632BC056BC26601E5077537435AD5F653F52B1D13C5194E1B2E35DBD09C059EE55092575CBB7F9AD5B23F5599CC7009BB494FD96CD2C1D4CD5BC0EE674713ADCA41322BCEF25EC1EDB25EAE308E668E30298C88660917775AA76CCF1E60FA8941181B6F1E7D96600A8B9116ABFDDA7A3AFB4D17B15CC8B761EDE09D6E3B6A257B2F1E129A1BE1F192B8E684B872ACA72BDEC7F482D568ABF482EBB97F5A24D0D14281AD8AF169204A395071C96BF2F4BB780EBD3ABD640D373CA8F0EB7AE46CA37E8A8D5E2DB575354DF79F46E004B700E0FC50F06AD8670FD6855B62B7F1279C2A0640794485A824760ED681C4BF9E2742A0698D
>  1492:d=3  hl=2 l=   0 prim: EOC
>  1494:d=2  hl=2 l=   0 prim: EOC
>  1496:d=1  hl=2 l=   0 prim: EOC

This is my gpgsm version, I did not upgrade to the latest version yet

  $ gpgsm --version
  gpgsm (GnuPG) 2.1.7-unknown
  libgcrypt 1.6.3
  libksba 1.3.3-unknown
  Copyright (C) 2015 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.

I also sent an email to openssl mailing list:
https://mta.openssl.org/pipermail/openssl-users/2018-April/007913.html

The suggestion was that it might be lacking zero-padding. Might this be
the problem here? I will update the gpgsm version to latest of course. But
since I observed the problem only once I wonder if this is a known issue and
has been fixed in upstream version.

kind regards,
Andi