status of S/MIME and interoperability?

[Daiki Ueno]

Daiki Ueno <ueno at gnu.org> writes:

> Ed Finnerty <edfinnerty at gmx.com> writes:
>
>>> I've actually logged the problem here and even attached a script that
>>> easily tests for the issue and can be modified to produce trivial test
>>> output for comparison purposes.
>>> 
>>> Nobody seemed to really care. So if you do S/MIME with GPG, and use
>>> Thunderbird, every once in a while you'll get an undecryptable message
>>> that you have to save to disk and try to deal with in some way with GPG.
>>
>> Of course, I forgot to add the link:
>> https://bugzilla.mozilla.org/show_bug.cgi?id=990958
>
> Although I am not sure if this is related to the original issue of this
> thread, I have the same impression that this is an NSS bug.

Actually, reading the CMS specification more closely, my initial
impression seems wrong, and this looks really like an issue in gpgsm
(actually in libksba, see below).

> In my test, it happens only when the encrypted session key is shorter
> than 256 bytes.  However, I don't see any problem in the CMS data which
> gpgsm produced.

When rsaEncryption is used, the length of the encrypted session key
should never change, as long as the same key is used.

The rationale is:

- The section 4.2 of RFC3370 states:

   The RSA key transport algorithm is the RSA encryption scheme defined
   in RFC 2313 [PKCS#1], block type is 02, where the message to be
   encrypted is the content-encryption key.

  ...

   Note that the same RSA encryption scheme is also defined in RFC 2437
   [NEWPKCS#1].  Within RFC 2437, this RSA encryption scheme is called
   RSAES-PKCS1-v1_5.
  
- In the section 7.2.1 of RFC2437, the step 4 of RSAES-PKCS1-v1_5 calls
  I2OPS as:

   4. Convert the ciphertext representative c to a ciphertext C of
   length k octets: C = I2OSP (c, k)

  where k is the length in octets of the modulus n

- I2OSP(x,l) adds leading zeros if the length of x in octets is shorter
  than l, as noted in the step 2 of I2OSP
  
In libksba/src/cms.c, ksba_cms_set_enc_val removes the first leading
zero:

  1834    if (n > 1 && !*s)
  1835      { /* We might have a leading zero due to the way we encode
  1836           MPIs - this zero should not go into the OCTECT STRING.  */
  1837        s++;
  1838        n--;
  1839      }

I think this should be bypassed, at least when rsaEncryption is used.
With the attached patch, I could no longer reproduce the issue with the
provided test_nss.sh script (with ~1000 iterations).

Regards,
-- 
Daiki Ueno
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Preserve-leading-zero-resulting-from-PKCS-1-scheme.patch
Type: text/x-patch
Size: 3064 bytes
Desc: not available
URL: </pipermail/attachments/20160218/576555c8/attachment.bin>