[gnutls-help] OAEP supported?

[Nikos Mavrogiannopoulos]

On Sat, Oct 3, 2015 at 10:55 AM, Frediano Ziglio <freddy77 at gmail.com> wrote:
> Hi,
>   I'm going mad trying to encrypt/decrypt some data using RSA with OAEP padding.
> The detail. I'm trying to implement the part of a database server
> (Sybase) to support
> sending encrypted password instead of plain one. The server send a
> public rsa key in
> this format:
> -----BEGIN RSA PUBLIC KEY-----
> MIGJAoGBANImhdQpGBmwkYveMfMPe0LmWj9vQOuqTdp0SOdQuNRtLEvr53axmV7jTx4ZCLTWnsAX
> i2ap524TEMnJdhzRUQeRIk3U7LLfwklq4YOhxgXolqPbDWhjS4YIohmEX2Ldto0pHqRrs8bIWu6H
> TvYYQhdiTAFJhQSvq5VoriL0MD+PAgMBAAE=
> -----END RSA PUBLIC KEY-----

Hello,
 This format isn't supported by gnutls. Only the SubjectPublicKeyInfo
format is (i.e., the "BEGIN PUBLIC KEY").

It is specified in: http://tools.ietf.org/html/rfc7468#section-13

> and some random data.
> It uses RSA with OAEP (the message is the random data plus the password).
>
> I would like to implement with GnuTLS but I found some problems:
> - gnutls_pubkey_import_x509_raw does not recognizing the format above
> as the only prefix it uses
>   is "PUBLIC KEY", not "RSA PUBLIC KEY". I'm actually working around using
>   gnutls_pem_base64_decode and using directly nettle_rsa_key_from_der_iterator;

We could extend the gnutls_pubkey_import() function to cope with these
public keys, if that's a widely used format. That could be using
header detection as in gnutls_x509_privkey_import(). If you have some
initial patch, I'd be happy to take a look at it.

> - nettle does not support OAEP so the workaround was to implement a
> rsa_encrypt_oaep  instead of the rsa_encrypt_tr.
> Now... this solution uses really few GnuTLS but a lot of lower level
> functions (nettle and gmp
> directly). OAEP is not that new in the RSA world so I think I'm doing
> something wrong with
> GnuTLS. Is there a way to this stuff with GnuTLS? I don't understand
> the relationship between
> GnuTLS and p11-kit modules. Looking at the code p11-kit module
> supports OAEP but is not clear how to use this module do to RSA encryption.

Indeed, RSASSA-PSS and OAEP are missing from nettle (and thus from
gnutls). So far TLS didn't use it so it was ignored. Moreover, because
there are no definitions for it, there is also no way to use it via
gnutls' interface for PKCS #11. I believe the fastest way to use it,
is via extending nettle's low level functions and bypassing gnutls for
that. If you send your nettle's extensions upstream, I would extend
the gnutls' API to use it.

regards,
Nikos