status of S/MIME and interoperability?

[Ed Finnerty]

On 02/18/2016 05:33 AM, Daiki Ueno wrote:
> Daiki Ueno <ueno at gnu.org> writes:
> 
>> Ed Finnerty <edfinnerty at gmx.com> writes:
>>
>>>> I've actually logged the problem here and even attached a script that
>>>> easily tests for the issue and can be modified to produce trivial test
>>>> output for comparison purposes.
>>>>
>>>> Nobody seemed to really care. So if you do S/MIME with GPG, and use
>>>> Thunderbird, every once in a while you'll get an undecryptable message
>>>> that you have to save to disk and try to deal with in some way with GPG.
>>>
>>> Of course, I forgot to add the link:
>>> https://bugzilla.mozilla.org/show_bug.cgi?id=990958
>>
>> Although I am not sure if this is related to the original issue of this
>> thread, I have the same impression that this is an NSS bug.
> 
> Actually, reading the CMS specification more closely, my initial
> impression seems wrong, and this looks really like an issue in gpgsm
> (actually in libksba, see below).
> 
>> In my test, it happens only when the encrypted session key is shorter
>> than 256 bytes.  However, I don't see any problem in the CMS data which
>> gpgsm produced.
> 
> When rsaEncryption is used, the length of the encrypted session key
> should never change, as long as the same key is used.
> 
> The rationale is:
> 
> - The section 4.2 of RFC3370 states:
> 
>    The RSA key transport algorithm is the RSA encryption scheme defined
>    in RFC 2313 [PKCS#1], block type is 02, where the message to be
>    encrypted is the content-encryption key.
> 
>   ...
> 
>    Note that the same RSA encryption scheme is also defined in RFC 2437
>    [NEWPKCS#1].  Within RFC 2437, this RSA encryption scheme is called
>    RSAES-PKCS1-v1_5.
>   
> - In the section 7.2.1 of RFC2437, the step 4 of RSAES-PKCS1-v1_5 calls
>   I2OPS as:
> 
>    4. Convert the ciphertext representative c to a ciphertext C of
>    length k octets: C = I2OSP (c, k)
> 
>   where k is the length in octets of the modulus n
> 
> - I2OSP(x,l) adds leading zeros if the length of x in octets is shorter
>   than l, as noted in the step 2 of I2OSP
>   
> In libksba/src/cms.c, ksba_cms_set_enc_val removes the first leading
> zero:
> 
>   1834    if (n > 1 && !*s)
>   1835      { /* We might have a leading zero due to the way we encode
>   1836           MPIs - this zero should not go into the OCTECT STRING.  */
>   1837        s++;
>   1838        n--;
>   1839      }
> 
> I think this should be bypassed, at least when rsaEncryption is used.
> With the attached patch, I could no longer reproduce the issue with the
> provided test_nss.sh script (with ~1000 iterations).

Thanks! Hopefully this patch will make it into GPG.