Non-deterministic behavior using GnuPG and a smart-card

[NIIBE Yutaka]

Hello,

"Dr. Basil Becker" <basil at basilbecker.de> writes:
> Authentication and signatures work like a charme. I'm only having
> problems concerning the decryption of mails I received.
[...]
> Some messages, however, fail to decrypt:
> bb at melmac:~$ gpg2 -vv --output /dev/null -d /tmp/message-fail.txt
> gpg: armor: BEGIN PGP MESSAGE
> gpg: armor header: Version: GnuPG v2
> # off=0 ctb=85 tag=1 hlen=3 plen=400
> :pubkey enc packet: version 3, algo 1, keyid DBC1D85BA9D1D189
> 	data: [3104 bits]
> gpg: public key is 0xDBC1D85BA9D1D189
> gpg: using subkey 0xDBC1D85BA9D1D189 instead of primary key
> 0x8501968486DF0281
> # off=403 ctb=d2 tag=18 hlen=2 plen=0 partial new-ctb
> :encrypted data packet:
> 	length: unknown
> 	mdc_method: 2
> gpg: using subkey 0xDBC1D85BA9D1D189 instead of primary key
> 0x8501968486DF0281
> gpg: encrypted with 3104-bit RSA key, ID 0xDBC1D85BA9D1D189, created
> 2017-01-10
>       "Dr. Basil Becker <basil at basilbecker.de>"
> gpg: public key decryption failed: Hardware problem
> gpg: decryption failed: No secret key
[...]
> The only difference I see, is that the pubkey data is 3103 bits vs 3104
> bits. Unfortunately, I have no idea, whether this is a meaningful
> difference and if this

I think that it is deterministic; The cause is that the RSA keysize is
not the one in the set of: 1024, 1536, 2048, 3072, 4096.  When data to
be decrypted is padded, scdaemon can't decrypt, I suppose.

I am not sure the exact reason why scdaemon only supports limited set of
keysize for encryption.   But we have this handling of padding in the
current code:

    https://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=blob;f=scd/app-openpgp.c;h=71c9e1b83003af07b0984688ba1ec5e9013b877c;hb=refs/heads/master#l4334

       /* We might encounter a couple of leading zeroes in the
          cryptogram.  Due to internal use of MPIs these leading zeroes
          are stripped.  However the OpenPGP card expects exactly 128
          bytes for the cryptogram (for a 1k key).  Thus we need to fix
          it up.  We do this for up to 16 leading zero bytes; a
          cryptogram with more than this is with a very high
          probability anyway broken.  If a signed conversion was used
          we may also encounter one leading zero followed by the correct
          length.  We fix that as well.  */
       if (indatalen >= (128-16) && indatalen < 128)      /* 1024 bit key.  */
         fixuplen = 128 - indatalen;
       else if (indatalen >= (192-16) && indatalen < 192) /* 1536 bit key.  */
         fixuplen = 192 - indatalen;
       else if (indatalen >= (256-16) && indatalen < 256) /* 2048 bit key.  */
         fixuplen = 256 - indatalen;
       else if (indatalen >= (384-16) && indatalen < 384) /* 3072 bit key.  */
         fixuplen = 384 - indatalen;
       else if (indatalen >= (512-16) && indatalen < 512) /* 4096 bit key.  */
         fixuplen = 512 - indatalen;
       else if (!*(const char *)indata && (indatalen == 129
                                           || indatalen == 193
                                           || indatalen == 257
                                           || indatalen == 385
                                           || indatalen == 513))
         fixuplen = -1;
       else
         fixuplen = 0;

Perhaps, it was due to support all existing OpenPGP card
implementations, I mean, somehow historical, and it was easier to list
up specific keysizes.

This should be fixed.
--