From kloecker at kde.org  Fri Aug  1 09:48:37 2025
From: kloecker at kde.org (Ingo =?UTF-8?B?S2zDtmNrZXI=?=)
Date: Fri, 01 Aug 2025 09:48:37 +0200
Subject: 8bit mime support? (linked to thunderbird issue)
In-Reply-To: <d85ef4f1-0fe7-42b9-8be0-a6e88168f7d2@dolce-energy.com>
References: <d0e9c13f-8f1b-41fe-ac47-7531b0357669@dolce-energy.com>
 <26123D2E-D6A5-412A-BC9F-5E14FDCF170C@andrewg.com>
 <d85ef4f1-0fe7-42b9-8be0-a6e88168f7d2@dolce-energy.com>
Message-ID: <5706176.rdbgypaU67@daneel>

On Donnerstag, 31. Juli 2025 23:28:51 Mitteleurop?ische Sommerzeit JL wrote:
> well it does compress, however you can't compress it on the imap server,

Pardon my ignorance, but why can't the IMAP server store your emails 
compressed? Maybe you should complain to the operator of the IMAP server if, 
in your opinion, they are wasting storage space. Or complain to its authors if 
you are operate the server yourself.

Given the insane growth of email headers it would actually make sense that the 
IMAP server used compression. Taking your message (as it is stored on my IMAP 
server) as example:
Size of the header: 12,315 bytes
Size of the body: 8,556 bytes.

By the way, why is the body of your message base64 encoded? Was is re-encoded 
in flight or did your Thunderbird send your unsigned plain text message base64 
encoded?

Regards,
Ingo
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 228 bytes
Desc: This is a digitally signed message part.
URL: <https://lists.gnupg.org/pipermail/gnupg-devel/attachments/20250801/057101b5/attachment.sig>

From noloader at gmail.com  Fri Aug  1 12:40:43 2025
From: noloader at gmail.com (Jeffrey Walton)
Date: Fri, 1 Aug 2025 06:40:43 -0400
Subject: 8bit mime support? (linked to thunderbird issue)
In-Reply-To: <5706176.rdbgypaU67@daneel>
References: <d0e9c13f-8f1b-41fe-ac47-7531b0357669@dolce-energy.com>
 <26123D2E-D6A5-412A-BC9F-5E14FDCF170C@andrewg.com>
 <d85ef4f1-0fe7-42b9-8be0-a6e88168f7d2@dolce-energy.com>
 <5706176.rdbgypaU67@daneel>
Message-ID: <CAH8yC8=1Q-1VFBBP7cSoy6+EE+0QeenU55d-0U-bVoLzRj2w3A@mail.gmail.com>

On Fri, Aug 1, 2025 at 3:49?AM Ingo Kl?cker <kloecker at kde.org> wrote:

> On Donnerstag, 31. Juli 2025 23:28:51 Mitteleurop?ische Sommerzeit JL
> wrote:
> > well it does compress, however you can't compress it on the imap server,
>
> Pardon my ignorance, but why can't the IMAP server store your emails
> compressed? Maybe you should complain to the operator of the IMAP server
> if,
> in your opinion, they are wasting storage space. Or complain to its
> authors if
> you are operate the server yourself.
>

I would hazard a guess that it relates to searching and sorting. IMAP
supports the SEARCH command. The SEARCH command allows searching on the
server based on keywords/free text, date ranges, and header fields.


> Given the insane growth of email headers it would actually make sense that
> the
> IMAP server used compression. Taking your message (as it is stored on my
> IMAP
> server) as example:
> Size of the header: 12,315 bytes
> Size of the body: 8,556 bytes.
>
> By the way, why is the body of your message base64 encoded? Was is
> re-encoded
> in flight or did your Thunderbird send your unsigned plain text message
> base64
> encoded?
>

Jeff
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.gnupg.org/pipermail/gnupg-devel/attachments/20250801/671d55f2/attachment-0001.html>

From steffen at sdaoden.eu  Fri Aug  1 17:34:01 2025
From: steffen at sdaoden.eu (Steffen Nurpmeso)
Date: Fri, 01 Aug 2025 17:34:01 +0200
Subject: 8bit mime support? (linked to thunderbird issue)
In-Reply-To: <d85ef4f1-0fe7-42b9-8be0-a6e88168f7d2@dolce-energy.com>
References: <d0e9c13f-8f1b-41fe-ac47-7531b0357669@dolce-energy.com>
 <87ms8lsuau.fsf@jacob.g10code.de>
 <67e83ba3-46b3-4268-8235-68ae9a8b947b@dolce-energy.com>
 <26123D2E-D6A5-412A-BC9F-5E14FDCF170C@andrewg.com>
 <d85ef4f1-0fe7-42b9-8be0-a6e88168f7d2@dolce-energy.com>
Message-ID: <20250801153401.ikqqW6Hr@steffen%sdaoden.eu>

JL wrote in
 <d85ef4f1-0fe7-42b9-8be0-a6e88168f7d2 at dolce-energy.com>:

It also depends you know.
Ie normally MUAs then "classify" whatever they create a MIME Part
for, and this classification may use some ratio of XY characters
to choose the one or the other transfer encoding.  See also
libmagic(1); my MUA differs to that for example (last i looked) as
we also wave through \v/\x0B/VT and \f=\x0C=NP, and we ignore ESC,
ie, may use quoted-printable not base64.

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
|
|During summer's humble, here's David Leonard's grumble
|
|The black bear,          The black bear,
|blithely holds his own   holds himself at leisure
|beating it, up and down  tossing over his ups and downs with pleasure
|
|Farewell, dear collar bear


From devm23k73ju29h3r at dolce-energy.com  Fri Aug  1 23:07:08 2025
From: devm23k73ju29h3r at dolce-energy.com (JL)
Date: Fri, 1 Aug 2025 23:07:08 +0200
Subject: 8bit mime support? (linked to thunderbird issue)
In-Reply-To: <CAH8yC8=1Q-1VFBBP7cSoy6+EE+0QeenU55d-0U-bVoLzRj2w3A@mail.gmail.com>
References: <d0e9c13f-8f1b-41fe-ac47-7531b0357669@dolce-energy.com>
 <26123D2E-D6A5-412A-BC9F-5E14FDCF170C@andrewg.com>
 <d85ef4f1-0fe7-42b9-8be0-a6e88168f7d2@dolce-energy.com>
 <5706176.rdbgypaU67@daneel>
 <CAH8yC8=1Q-1VFBBP7cSoy6+EE+0QeenU55d-0U-bVoLzRj2w3A@mail.gmail.com>
Message-ID: <3777908a-f8c1-4a4b-8874-774ef0fdfee3@dolce-energy.com>


Le 01/08/2025 ? 12:40, Jeffrey Walton via Gnupg-devel a ?crit?:
>
>
> On Fri, Aug 1, 2025 at 3:49?AM Ingo Kl?cker <kloecker at kde.org> wrote:
>
>     On Donnerstag, 31. Juli 2025 23:28:51 Mitteleurop?ische Sommerzeit
>     JL wrote:
>     > well it does compress, however you can't compress it on the imap
>     server,
>
>     Pardon my ignorance, but why can't the IMAP server store your emails
>     compressed? Maybe you should complain to the operator of the IMAP
>     server if,
>     in your opinion, they are wasting storage space. Or complain to
>     its authors if
>     you are operate the server yourself.
>
>
> I would hazard a guess that it relates to searching and sorting. IMAP 
> supports the SEARCH command. The SEARCH command allows searching on 
> the server based on keywords/free text, date ranges, and header fields.

this is server side... doesn't apply to local storage, you might want to 
keep things on server, I don't. Even if it's usefull, a server is a good 
target for hackers, even google, apple, facebook, got hacked, they often 
assure that "no personal emails were gatered" we just have their words, 
no one beside them know the truth... since emails on servers are not 
cyphered, all the important data is freely accessible.

Beside, there is mail quota... so you can't store everything on imap

>     Given the insane growth of email headers it would actually make
>     sense that the
>     IMAP server used compression. Taking your message (as it is stored
>     on my IMAP
>     server) as example:
>     Size of the header: 12,315 bytes
>     Size of the body: 8,556 bytes.
>
>     By the way, why is the body of your message base64 encoded? Was is
>     re-encoded
>     in flight or did your Thunderbird send your unsigned plain text
>     message base64
>     encoded?
>
Because it's the safe fallback, and rather than make the things evolve, 
all email client prefer the safe way : the one that will works even on 
30 years ago server/client.

8bit mime isn't perfect because it didn't defined way to handle the crlf 
in binary or because it didn't enforced that the buffer has to be 
filled. Smtp defined a way to prevent end of transmission if the message 
contains <cr><lf>.<cr><lf> but I didn't saw some escape mecanism for 8 
bit mime... which means that it's up to the client to handle this case 
(even if in binary data, it should occurs not so often) don?t know if 
there is some rfc that define way to escape <cr><lf> so that they could 
be dropped by client (for example <cr><lf> in data recoded into 
\<cr>\<lf> so that <cr><lf> can be dropped) anyway this is only a smtp 
issue, or the server should handle rfc3030 "BDAT" command for example 
that is cleaner...

beside the smtp consideration, on IMAP side, the data can be accessed in 
binary, and mime has a "Content-Transfer-Encoding" binary... so from the 
client point of view, the received message could be "decoded" on the fly 
and stored with decoded value...

some server might transform the data (because one server on the way 
don't handle BDAT for exemple) but the end server could re-code it as 
binary... instead of keeping it base64...? The problem is that client 
often "write the data to file" and then "send the file through smtp" so 
not knowing if the smtp server can handle the BDAT... so the safe way, 
if you don't know the server, is to encode everything in base64...

so the problem is multiple :

  * user email client : don't handle well binary.... and don?t encode
    data "on the fly" when talking with the smtp server....
  * SMTP : a server might encode to base64 because the other server
    don't handle BDAT transmission, if there are relay server, they
    should reverse it back to binary.... and the final server should
    re-format the data...

best regards
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.gnupg.org/pipermail/gnupg-devel/attachments/20250801/be02af8e/attachment.html>

From devm23k73ju29h3r at dolce-energy.com  Sat Aug  2 00:17:04 2025
From: devm23k73ju29h3r at dolce-energy.com (JL)
Date: Sat, 2 Aug 2025 00:17:04 +0200
Subject: 8bit mime support? (linked to thunderbird issue)
In-Reply-To: <5706176.rdbgypaU67@daneel>
References: <d0e9c13f-8f1b-41fe-ac47-7531b0357669@dolce-energy.com>
 <26123D2E-D6A5-412A-BC9F-5E14FDCF170C@andrewg.com>
 <d85ef4f1-0fe7-42b9-8be0-a6e88168f7d2@dolce-energy.com>
 <5706176.rdbgypaU67@daneel>
Message-ID: <66506f22-1b80-4f6b-9bc4-ae381593e62e@dolce-energy.com>


Le 01/08/2025 ? 09:48, Ingo Kl?cker a ?crit?:
> On Donnerstag, 31. Juli 2025 23:28:51 Mitteleurop?ische Sommerzeit JL wrote:
>> well it does compress, however you can't compress it on the imap server,
> Pardon my ignorance, but why can't the IMAP server store your emails
> compressed? Maybe you should complain to the operator of the IMAP server if,
> in your opinion, they are wasting storage space. Or complain to its authors if
> you are operate the server yourself.

you have nothing to be pardonned, I'm not the most up-to-date and rather 
ignorant on the matter...

well, they might, but it don't means your quota will (and then that 
you'll benefit) they surely do it, storing on filesystem with 
transparent compression... But they can't compress it serverside without 
rewriting the message (they have to keep the mime file format so that 
the client can read them... a smart server would have received the data, 
and decoded once for all, like this, it'll have been network efficient 
and storage efficient, but this introduce the issue of PGP signing... 
since the message is "modified" and don't know how to scope with this 
matter... is it possible to "sign" each part before encoding? (like a 
checksum file containing all the checksum of every attached file for 
example)

if it's assumed that the message MUST not be altered, you have to go the 
safe way : encode everything in 7bit ASCII

if the signing can be done "part by part in binary" then there is 
nothing to worry about "re-formating" and a SMTP server can choose to 
send it binary (if BINARYMIME and BDAT is supported) or re-code it....

just guessing, how a smtp server will handle if it received a BINARYMIME 
and have to talk to a non BINARYMIME server, what will?they do?

enclose the whole message into a

    Content-Type: message/rfc822;
    boundary="------------AF0V9AuRuaNTRxYbEuAis34i";
    Content-Transfer-Encoding: base64
    XXXXXXXXXXXX
    XXXXXXXXXXXX
    ------------AF0V9AuRuaNTRxYbEuAis34i"

?

best regards


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.gnupg.org/pipermail/gnupg-devel/attachments/20250802/e6fb3226/attachment.html>

From devm23k73ju29h3r at dolce-energy.com  Sat Aug  2 13:05:07 2025
From: devm23k73ju29h3r at dolce-energy.com (JL)
Date: Sat, 2 Aug 2025 13:05:07 +0200
Subject: 8bit mime support? (linked to thunderbird issue)
In-Reply-To: <66506f22-1b80-4f6b-9bc4-ae381593e62e@dolce-energy.com>
References: <d0e9c13f-8f1b-41fe-ac47-7531b0357669@dolce-energy.com>
 <26123D2E-D6A5-412A-BC9F-5E14FDCF170C@andrewg.com>
 <d85ef4f1-0fe7-42b9-8be0-a6e88168f7d2@dolce-energy.com>
 <5706176.rdbgypaU67@daneel>
 <66506f22-1b80-4f6b-9bc4-ae381593e62e@dolce-energy.com>
Message-ID: <03a63bbe-d4eb-4572-8728-78d23c5e6f95@dolce-energy.com>


> ?a smart server would have received the data, and decoded once for 
> all, like this, it'll have been network efficient and storage 
> efficient, but this introduce the issue of PGP signing... since the 
> message is "modified" and don't know how to scope with this matter... 
> is it possible to "sign" each part before encoding? (like a checksum 
> file containing all the checksum of every attached file for example)
>
> if it's assumed that the message MUST not be altered, you have to go 
> the safe way : encode everything in 7bit ASCII
>
> if the signing can be done "part by part in binary" then there is 
> nothing to worry about "re-formating" and a SMTP server can choose to 
> send it binary (if BINARYMIME and BDAT is supported) or re-code it....
>
replying to myself after reading the

  * rfc4880 " OpenPGP Message Format"
  * rfc3156 "MIME Security with OpenPGP"

the latter state that, because the message can be reformated,

 ? ?However, many existing mail gateways will detect if the next
 ? ?hop does not support MIME or 8-bit data and perform conversion to
 ? ?either Quoted-Printable or Base64.? This presents serious problems
 ? ?for multipart/signed, in particular, where the signature is
 ? ?invalidated when such an operation occurs.? For this reason all data
 ? ?signed according to this protocol MUST be constrained to 7 bits (8-
 ? ?bit data MUST be encoded using either Quoted-Printable or Base64).

that's too bad, since in fact the "format" is enforced before signing, 
while they could have chosen the opposite : enforcing all binary fike to 
be presented into binary format in the mime message, and signing 
versification should only be performed once restored to original format....

and in fact a false issue... the signature won't be invalidated if the 
presentation change, since the presentation shall not alter the original 
content and is reversible... so it just needed to force the orginal 
format to create the message with binary data when binary data, 8bit 
strings when 8bits strings, eg removing the formating process.

On reception, the receiver remove the formating, going back to binary 
for things that are binary, and validate the unformatted content... it 
would have been compatible with other RFC effort to allow binary data to 
go through the smtp protocol...

really, too bad...

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.gnupg.org/pipermail/gnupg-devel/attachments/20250802/04f0a78c/attachment-0001.html>

From andrewg at andrewg.com  Sat Aug  2 15:04:41 2025
From: andrewg at andrewg.com (Andrew Gallagher)
Date: Sat, 2 Aug 2025 14:04:41 +0100
Subject: 8bit mime support? (linked to thunderbird issue)
In-Reply-To: <03a63bbe-d4eb-4572-8728-78d23c5e6f95@dolce-energy.com>
References: <03a63bbe-d4eb-4572-8728-78d23c5e6f95@dolce-energy.com>
Message-ID: <9958C9B4-569D-4925-9119-4EDC1E99F432@andrewg.com>

On 2 Aug 2025, at 12:06, JL <devm23k73ju29h3r at dolce-energy.com> wrote:
> 
> that's too bad, since in fact the "format" is enforced before signing, while they could have chosen the opposite : enforcing all binary fike to be presented into binary format in the mime message, and signing versification should only be performed once restored to original format.... 

I think you have misread the spec because that?s already what it requires. Signing is performed before encoding to 7-bit safe format, and verification after decoding. The only time normalisation is performed before signing is with 0x01 text document signatures, when line endings are converted to wire format. This is increasingly a historical curiosity though, and is unnecessary if you are using base64.

A

From andrewg at andrewg.com  Sat Aug  2 15:31:49 2025
From: andrewg at andrewg.com (Andrew Gallagher)
Date: Sat, 2 Aug 2025 14:31:49 +0100
Subject: 8bit mime support? (linked to thunderbird issue)
In-Reply-To: <9958C9B4-569D-4925-9119-4EDC1E99F432@andrewg.com>
References: <9958C9B4-569D-4925-9119-4EDC1E99F432@andrewg.com>
Message-ID: <B2AC0264-8A4E-406A-9FDE-51FC584B9C02@andrewg.com>

On 2 Aug 2025, at 14:06, Andrew Gallagher <andrewg at andrewg.com> wrote:
> 
> ?On 2 Aug 2025, at 12:06, JL <devm23k73ju29h3r at dolce-energy.com> wrote:
>> 
>> that's too bad, since in fact the "format" is enforced before signing, while they could have chosen the opposite : enforcing all binary fike to be presented into binary format in the mime message, and signing versification should only be performed once restored to original format....
> 
> I think you have misread the spec because that?s already what it requires. Signing is performed before encoding to 7-bit safe format, and verification after decoding. The only time normalisation is performed before signing is with 0x01 text document signatures, when line endings are converted to wire format. This is increasingly a historical curiosity though, and is unnecessary if you are using base64.

OK, my turn to reply to myself. Sorry, *I* misread *your* message. :-( The above comment doesn?t apply to your scenario.

Yes, base64 is used because it is relatively immune to mangling by MTAs in transit (although not perfectly so). And while you can transmit an entire signed message as a base64 blob, it?s more common to sign over the mime structure, which may have subparts, and so 7-bit safe encoding can happen before the signing step. It was specified this way so that naive clients could still process the signed-over data and display it without having to understand the details of openpgp.

Sorry for the confusion.

A