From tom at ritter.vg  Mon Apr  4 00:51:42 2011
From: tom at ritter.vg (Tom Ritter)
Date: Sun, 3 Apr 2011 18:51:42 -0400
Subject: Compiling in low-memory conditions
Message-ID: <BANLkTinQHqZa_Fh2OygXLHo1Tem4KiC02A@mail.gmail.com>

I was compiling libgcrypt and gnupg on a low-memory VPS, and ran into
a few problems. First, I compiled libgcrypt, and ran into a gcc crash,
when it ran out of memory trying to compile twofish.o.  It looked like
this:

gcc -DHAVE_CONFIG_H -I. -I.. -I../src -I../src -I/usr/local/include -g
-O2 --param ggc-min-expand=0 --param ggc-min-heapsize=32768
-fvisibility=hidden -Wall -MT twofish.lo -MD -MP -MF .deps/twofish.Tpo
-c twofish.c  -fPIC -DPIC -o .libs/twofish.o

cc1: out of memory allocating 3082800 bytes after a total of 4681728 bytes
make[2]: *** [twofish.lo] Error 1

I worked around that, I'll explain in a minute but I did compile
libgcrypt, and moved onto gnupg.  For gnupg I ran ./configure and
after installing the libraries I needed, I began to compile gnupg.
However it failed with an error about missing zlib.h - I would have
though configure would pick that up.  But no worries, after getting
it, I continued the compile and ran into the same gcc out-of-memory
crashes on :
 - keyedit.o
 - gpg.o
 - keygen.o

To get around the out of memory issue, I tried using ulimit, but this
did not help.  What did work was editing the Makefile and temporarily
setting the optimization level to -O0.  After the module was compiled
I killed the build, and set it back to O2.  Repeating for each module
that crashed it.  I don't know if there's any way this could be
prevented without unnecessary work, so I mainly just wanted to
document it for anyone who ran into the same problem as me.

Finally, i believe the 'status_codes.h' getting deleted that I
reported last summer[1] is still active.

-tom

[1] http://lists.gnupg.org/pipermail/gnupg-devel/2010-July/025642.html


From wk at gnupg.org  Mon Apr  4 09:32:44 2011
From: wk at gnupg.org (Werner Koch)
Date: Mon, 04 Apr 2011 09:32:44 +0200
Subject: Compiling in low-memory conditions
In-Reply-To: <067401cbf28a$02542a30$f9b5a8c0@pii350> (Gilles Espinasse's
	message of "Mon, 4 Apr 2011 07:34:44 +0200")
References: <BANLkTinQHqZa_Fh2OygXLHo1Tem4KiC02A@mail.gmail.com>
	<067401cbf28a$02542a30$f9b5a8c0@pii350>
Message-ID: <8739lybher.fsf@vigenere.g10code.de>

On Mon,  4 Apr 2011 07:34, g.esp at free.fr said:

> Not enabling debugging with '-g' should spare memory for the compilation.

I don't think this helps.  The optimizer is the problem and has always
been.  It tries to unroll the core of the twofish algorithm but it can't
get it better than what we have in the source code.  

Newer gcc versions default to -O2 which exhibits the problem.  Use -O1
or return to an older gcc version.


Shalom-Salam,

   Werner

-- 
Die Gedanken sind frei.  Ausnahmen regelt ein Bundesgesetz.



From g.esp at free.fr  Mon Apr  4 07:34:44 2011
From: g.esp at free.fr (Gilles Espinasse)
Date: Mon, 4 Apr 2011 07:34:44 +0200
Subject: Compiling in low-memory conditions
References: <BANLkTinQHqZa_Fh2OygXLHo1Tem4KiC02A@mail.gmail.com>
Message-ID: <067401cbf28a$02542a30$f9b5a8c0@pii350>


----- Original Message ----- 
From: "Tom Ritter" <tom at ritter.vg>
To: <gcrypt-devel at gnupg.org>; <gnupg-devel at gnupg.org>
Sent: Monday, April 04, 2011 12:51 AM
Subject: Compiling in low-memory conditions


> I was compiling libgcrypt and gnupg on a low-memory VPS, and ran into
> a few problems. First, I compiled libgcrypt, and ran into a gcc crash,
> when it ran out of memory trying to compile twofish.o.  It looked like
> this:
>
> gcc -DHAVE_CONFIG_H -I. -I.. -I../src -I../src -I/usr/local/include -g
> -O2 --param ggc-min-expand=0 --param ggc-min-heapsize=32768
> -fvisibility=hidden -Wall -MT twofish.lo -MD -MP -MF .deps/twofish.Tpo
> -c twofish.c  -fPIC -DPIC -o .libs/twofish.o
>
> cc1: out of memory allocating 3082800 bytes after a total of 4681728 bytes
> make[2]: *** [twofish.lo] Error 1
>
Not enabling debugging with '-g' should spare memory for the compilation.

Gilles



From news at edrusb.is-a-geek.org  Wed Apr  6 20:35:03 2011
From: news at edrusb.is-a-geek.org (Joe the Shmoe)
Date: Wed, 06 Apr 2011 20:35:03 +0200
Subject: gcry_cipher_decrypt usage
Message-ID: <obou68-b7d.ln1@news.edrusb.is-a-geek.org>

Hi,

I hope to ask my question at the appropriate place, if not tell me and
thanks to accept my appologies.

While all works fine using libgcrypt, I have meet a situation where the
submitted buffer containg ciphered data to gcry_cipher_decrypt() may
also contain clear data after the ciphered data.

In that situation, gcry_cipher_decrypt() decipher properly the encrypted
data into the output buffer, but I cannot figure out how much data was
put in it, and cry_cipher_decrypt() does not report any error neither,
while the original clear data located at the end of the input buffer,
could most probably not be "deciphered".

In other words, when a buffer of ciphered data is given to
gcry_cipher_decrypt, how is it possible to know the amount of clear text
data that has been produced?

Thanks in advance for any help,

Regards,
Denis.




From wk at gnupg.org  Thu Apr  7 09:28:04 2011
From: wk at gnupg.org (Werner Koch)
Date: Thu, 07 Apr 2011 09:28:04 +0200
Subject: gcry_cipher_decrypt usage
In-Reply-To: <obou68-b7d.ln1@news.edrusb.is-a-geek.org> (Joe the Shmoe's
	message of "Wed, 06 Apr 2011 20:35:03 +0200")
References: <obou68-b7d.ln1@news.edrusb.is-a-geek.org>
Message-ID: <878vvm8qrf.fsf@vigenere.g10code.de>

On Wed,  6 Apr 2011 20:35, news at edrusb.is-a-geek.org said:

> In other words, when a buffer of ciphered data is given to
> gcry_cipher_decrypt, how is it possible to know the amount of clear text
> data that has been produced?

That depends on the cipher mode.  Almost all modes return the same
number of bytes as you put in.  Can you please give a more explicit
example?


Shalom-Salam,

   Werner

-- 
Die Gedanken sind frei.  Ausnahmen regelt ein Bundesgesetz.



From news at edrusb.is-a-geek.org  Thu Apr  7 10:55:11 2011
From: news at edrusb.is-a-geek.org (Joe the Shmoe)
Date: Thu, 07 Apr 2011 10:55:11 +0200
Subject: gcry_cipher_decrypt usage
In-Reply-To: <obou68-b7d.ln1@news.edrusb.is-a-geek.org> 
References: <obou68-b7d.ln1@news.edrusb.is-a-geek.org> 
Message-ID: <20110407085512.7E1DB2036F@edrusb.is-a-geek.org>

Werner Koch wrote:
> On Wed,  6 Apr 2011 20:35, news at edrusb.is-a-geek.org said:
> 
>> In other words, when a buffer of ciphered data is given to
>> gcry_cipher_decrypt, how is it possible to know the amount of clear text
>> data that has been produced?
> 
> That depends on the cipher mode.  Almost all modes return the same
> number of bytes as you put in.  Can you please give a more explicit
> example?

Sure,

the context is the one of an archiver program ("Disk ARchive") when
comes the time to read an encrypted archive.

Cipher used is blowfish. In this buggy situation, gcry_cipher_decrypt()
receives 10248 bytes of input, which in fact do contain 10240 bytes of
ciphered data followed by 8 bytes of clear data.

The fact that clear data is located after the ciphered data is not
possible to know when gcry_cipher_decrypt() is called that time, thus as
this call do not return any error, it is (wrongly) assumed that
gcry_cipher_decrypt() could decrypt 10248 bytes of data and provided the
same amount of plain text data into the output buffer.

At that time, this decrypted data is sent to its destination and the
next block of data is treated by gcry_cipher_decrypt(). This second
time, gcry_cipher_decrypt() returns an error as expected, because the
data is no more encrypted at all. At that time, reading the clear text
data directly I get the information that the clear data started 8 bytes
before ... but that's too late, the previous 8 bytes have already been
wrongly sent to destination.

Thus, my question is not how much clear data, for a given amount of
ciphered data a given algorithm do provide _in theory_, but rather how
to know whether a given ciphered block of data could successfully be
deciphered completely or partially. The best would be to have
gcry_cipher_decrypt() being able to provide the effective amount of
clear data available in the output buffer. But having
gcry_cipher_decrypt() failing if all the provided data could not be
decrypted would also solve my problem.

I have re-read documentation, reviewed libgcrypt header file for any
undocumented call, but I'm affraid to have to change the implementation
of Disk ARchive after having passed from openssl to libgcrypt dependancy?

Thanks for any help,

Kind Regards,
"Joe the Shmoe" aka Denis Corbin.










From wk at gnupg.org  Thu Apr  7 14:01:56 2011
From: wk at gnupg.org (Werner Koch)
Date: Thu, 07 Apr 2011 14:01:56 +0200
Subject: gcry_cipher_decrypt usage
In-Reply-To: <20110407085512.7E1DB2036F@edrusb.is-a-geek.org> (Joe the Shmoe's
	message of "Thu, 07 Apr 2011 10:55:11 +0200")
References: <obou68-b7d.ln1@news.edrusb.is-a-geek.org>
	<20110407085512.7E1DB2036F@edrusb.is-a-geek.org>
Message-ID: <87pqoy6zij.fsf@vigenere.g10code.de>

On Thu,  7 Apr 2011 10:55, news at edrusb.is-a-geek.org said:

> Cipher used is blowfish. In this buggy situation, gcry_cipher_decrypt()

What cipher mode (CBC, CFB, CTR, ...) and depending on that what kind of
padding to you use.  How do you call gcry_cipher_open?

> The fact that clear data is located after the ciphered data is not

Sure it can't know that and will happlity decrypt this.
gcry_cipher_decrypt can's distinguish between cleartext and ciphertext.
The upper layer (i.e. the protocol) must define how this can be done.

> time, gcry_cipher_decrypt() returns an error as expected, because the

gcry_cipher_decrypt can't return an error except for cases where you you
pass an invalid data length (e.g. not a multiple of the blocklength).

Shalom-Salam,

   Werner

-- 
Die Gedanken sind frei.  Ausnahmen regelt ein Bundesgesetz.



From news at edrusb.is-a-geek.org  Thu Apr  7 22:14:58 2011
From: news at edrusb.is-a-geek.org (Joe the Shmoe)
Date: Thu, 07 Apr 2011 22:14:58 +0200
Subject: gcry_cipher_decrypt usage
In-Reply-To: <87pqoy6zij.fsf@vigenere.g10code.de>
References: <obou68-b7d.ln1@news.edrusb.is-a-geek.org>	<20110407085512.7E1DB2036F@edrusb.is-a-geek.org>
	<87pqoy6zij.fsf@vigenere.g10code.de>
Message-ID: <3ji178-ps4.ln1@news.edrusb.is-a-geek.org>

Werner Koch a ?crit :
> On Thu,  7 Apr 2011 10:55, news at edrusb.is-a-geek.org said:
> 
>> Cipher used is blowfish. In this buggy situation, gcry_cipher_decrypt()
> 
> What cipher mode (CBC, CFB, CTR, ...) and depending on that what kind of
> padding to you use.  How do you call gcry_cipher_open?

gcry_cipher_hd_t clef;

err = gcry_cipher_open(&clef,
	               algo_id, // equal to GCRY_CIPHER_BLOWFISH
	               GCRY_CIPHER_MODE_CBC,
	               GCRY_CIPHER_SECURE);
etc...

> 
>> The fact that clear data is located after the ciphered data is not
> 
> Sure it can't know that and will happlity decrypt this.

OK, I better see the picture now, and understand my mistake.

> gcry_cipher_decrypt can's distinguish between cleartext and ciphertext.
> The upper layer (i.e. the protocol) must define how this can be done.

Yep, I must review this upper layer in Disk ARchive, to handle the
particular and rare situation in which this condition occur  and thus
avoid having clear text after ciphered one being passed to
gcry_cipher_decrypt().

> 
>> time, gcry_cipher_decrypt() returns an error as expected, because the
> 
> gcry_cipher_decrypt can't return an error except for cases where you you
> pass an invalid data length (e.g. not a multiple of the blocklength).
> 
> Shalom-Salam,
> 
>    Werner
> 

Werner, Thank you for your clear explanations. And also, at this
occasion, thank you for GnuPG. ;-)

Best Regards,
Denis Corbin.




From ian at cypherpunks.ca  Sun Apr 10 16:58:57 2011
From: ian at cypherpunks.ca (Ian Goldberg)
Date: Sun, 10 Apr 2011 10:58:57 -0400
Subject: [OTR-dev] Issues with libgcrypt 1.5
In-Reply-To: <4DA195FD.20409@pentabarf.de>
References: <4DA195FD.20409@pentabarf.de>
Message-ID: <20110410145857.GA2073@yoink.cs.uwaterloo.ca>

Werner Koch wrote:
> On Wed, 9 Mar 2011 18:19, arfrever.fta [at] gmail said:
> > I have added Libgcrypt 1.5.0-beta1 to Gentoo's main repository on
> > 2011-02-23.
> > Until now, only problem [1] with libotr [2] has been reported.
> > Could you check if it is a bug in Libgcrypt 1.5.0-beta1 or in libotr?
> 
> Well, I need a bit more of info. Which function returns with "invalid
> length"? I need a small test case (w/o any gentoo or KDE specific
> stuff). 

and Kjell Branded (on the otr-dev list wrote):
> FYI: libgcrypt-1.5.0_beta1 seems to throw GPG_ERR_INV_LENGTH errors when
> encrypting/decrypting messages using AES-CTR when their length is not a
> multiple of the block size (16?). This makes libotr unusable with that
> version.

Werner, would it be possible to revert libgcrypt's behaviour to
internally handle a truncated final block for AES-CTR mode?

Thanks,

   - Ian


From wk at gnupg.org  Mon Apr 11 10:37:46 2011
From: wk at gnupg.org (Werner Koch)
Date: Mon, 11 Apr 2011 10:37:46 +0200
Subject: [OTR-dev] Issues with libgcrypt 1.5
In-Reply-To: <20110410145857.GA2073@yoink.cs.uwaterloo.ca> (Ian Goldberg's
	message of "Sun, 10 Apr 2011 10:58:57 -0400")
References: <4DA195FD.20409@pentabarf.de>
	<20110410145857.GA2073@yoink.cs.uwaterloo.ca>
Message-ID: <87mxjx5gkl.fsf@vigenere.g10code.de>

On Sun, 10 Apr 2011 16:58, ian at cypherpunks.ca said:

> Werner, would it be possible to revert libgcrypt's behaviour to
> internally handle a truncated final block for AES-CTR mode?

Can you pelase specifiy the behaviour you want?  It is still time to
loose this check.


Shalom-Salam,

   Werner

-- 
Die Gedanken sind frei.  Ausnahmen regelt ein Bundesgesetz.



From ian at cypherpunks.ca  Mon Apr 11 13:48:36 2011
From: ian at cypherpunks.ca (Ian Goldberg)
Date: Mon, 11 Apr 2011 07:48:36 -0400
Subject: [OTR-dev] Issues with libgcrypt 1.5
In-Reply-To: <87mxjx5gkl.fsf@vigenere.g10code.de>
References: <4DA195FD.20409@pentabarf.de>
	<20110410145857.GA2073@yoink.cs.uwaterloo.ca>
	<87mxjx5gkl.fsf@vigenere.g10code.de>
Message-ID: <20110411114836.GA30160@thunk.cs.uwaterloo.ca>

On Mon, Apr 11, 2011 at 10:37:46AM +0200, Werner Koch wrote:
> On Sun, 10 Apr 2011 16:58, ian at cypherpunks.ca said:
> 
> > Werner, would it be possible to revert libgcrypt's behaviour to
> > internally handle a truncated final block for AES-CTR mode?
> 
> Can you pelase specifiy the behaviour you want?  It is still time to
> loose this check.

In 1.4, if the plaintext was not a multiple of the block size, the last
encrypted counter block was simply truncated to the appropriate length,
which seems like the correct behaviour to me.  (And indeed, is what the
libotr library expects.)

Ideally, CTR mode would operate like a stream cipher whose keystream was
E(c) || E(c+1) || E(c+2) || ... .  That would mean stashing away the
"truncated" part above for use at the start of the next encryption.
I don't know whether 1.4 did that or not; I didn't try encrypting twice
between resetting the key/counter.

So it would seem to be fairly easy to just revert the 1.4 behaviour for
now, and if that's not the "stream cipher" behaviour, maybe see if that
can get into 1.5 as well?

Thanks,

   - Ian


From wk at gnupg.org  Mon Apr 11 16:37:18 2011
From: wk at gnupg.org (Werner Koch)
Date: Mon, 11 Apr 2011 16:37:18 +0200
Subject: [OTR-dev] Issues with libgcrypt 1.5
In-Reply-To: <20110411114836.GA30160@thunk.cs.uwaterloo.ca> (Ian Goldberg's
	message of "Mon, 11 Apr 2011 07:48:36 -0400")
References: <4DA195FD.20409@pentabarf.de>
	<20110410145857.GA2073@yoink.cs.uwaterloo.ca>
	<87mxjx5gkl.fsf@vigenere.g10code.de>
	<20110411114836.GA30160@thunk.cs.uwaterloo.ca>
Message-ID: <877hb06eht.fsf@vigenere.g10code.de>

On Mon, 11 Apr 2011 13:48, ian at cypherpunks.ca said:

> In 1.4, if the plaintext was not a multiple of the block size, the last
> encrypted counter block was simply truncated to the appropriate length,
> which seems like the correct behaviour to me.  (And indeed, is what the
> libotr library expects.)

Okay, I'll revert that.

> Ideally, CTR mode would operate like a stream cipher whose keystream was
> E(c) || E(c+1) || E(c+2) || ... .  That would mean stashing away the
> "truncated" part above for use at the start of the next encryption.
> I don't know whether 1.4 did that or not; I didn't try encrypting twice

No it did not.  However we(?) once talked about this and my idea was to
add a blocking layer to avoid doing that explicitly for each mode.  This
would also allow to disable this blocking layer to get better
performance.  The problem with not requiring full blocks in CBC etc, is
that we can't used aligned versions of the encryption functions, need to
shift the data in memory and save parts of the input.  A flag to
explicitly enable arbitrary lengths input is IMHO the cleanest solution.


Salam-Shalom,

   Werner

-- 
Die Gedanken sind frei.  Ausnahmen regelt ein Bundesgesetz.



From wk at gnupg.org  Mon Apr 11 17:41:52 2011
From: wk at gnupg.org (Werner Koch)
Date: Mon, 11 Apr 2011 17:41:52 +0200
Subject: [OTR-dev] Issues with libgcrypt 1.5
In-Reply-To: <20110411114836.GA30160@thunk.cs.uwaterloo.ca> (Ian Goldberg's
	message of "Mon, 11 Apr 2011 07:48:36 -0400")
References: <4DA195FD.20409@pentabarf.de>
	<20110410145857.GA2073@yoink.cs.uwaterloo.ca>
	<87mxjx5gkl.fsf@vigenere.g10code.de>
	<20110411114836.GA30160@thunk.cs.uwaterloo.ca>
Message-ID: <8739lo6bi7.fsf@vigenere.g10code.de>

On Mon, 11 Apr 2011 13:48, ian at cypherpunks.ca said:

> So it would seem to be fairly easy to just revert the 1.4 behaviour for
> now, and if that's not the "stream cipher" behaviour, maybe see if that
> can get into 1.5 as well?

(Maybe I can do it without an extra block layer.)

Do you have short test messages, I can use of the regression tests (from
libgcrypt 1.4)?


Shalom-Salam,

   Werner

-- 
Die Gedanken sind frei.  Ausnahmen regelt ein Bundesgesetz.



From ian at cypherpunks.ca  Mon Apr 11 19:03:02 2011
From: ian at cypherpunks.ca (Ian Goldberg)
Date: Mon, 11 Apr 2011 13:03:02 -0400
Subject: [OTR-dev] Issues with libgcrypt 1.5
In-Reply-To: <8739lo6bi7.fsf@vigenere.g10code.de>
References: <4DA195FD.20409@pentabarf.de>
	<20110410145857.GA2073@yoink.cs.uwaterloo.ca>
	<87mxjx5gkl.fsf@vigenere.g10code.de>
	<20110411114836.GA30160@thunk.cs.uwaterloo.ca>
	<8739lo6bi7.fsf@vigenere.g10code.de>
Message-ID: <20110411170302.GP30160@thunk.cs.uwaterloo.ca>

On Mon, Apr 11, 2011 at 05:41:52PM +0200, Werner Koch wrote:
> On Mon, 11 Apr 2011 13:48, ian at cypherpunks.ca said:
> 
> > So it would seem to be fairly easy to just revert the 1.4 behaviour for
> > now, and if that's not the "stream cipher" behaviour, maybe see if that
> > can get into 1.5 as well?
> 
> (Maybe I can do it without an extra block layer.)
> 
> Do you have short test messages, I can use of the regression tests (from
> libgcrypt 1.4)?

Take any message of length, say, 45.  Init the key and counter, and
encrypt.

Then start again with the same message, key, and counter.  Encrypt the
first x bytes, and then the remaining 45-x (both in place).  You should
get the same answer.

If you do this with x = 0,1,7,8,9,15,16,17,22, that should be a good set
of test cases, I think.

Thanks,

   - Ian


From wk at gnupg.org  Mon Apr 11 19:31:55 2011
From: wk at gnupg.org (Werner Koch)
Date: Mon, 11 Apr 2011 19:31:55 +0200
Subject: [OTR-dev] Issues with libgcrypt 1.5
In-Reply-To: <20110411170302.GP30160@thunk.cs.uwaterloo.ca> (Ian Goldberg's
	message of "Mon, 11 Apr 2011 13:03:02 -0400")
References: <4DA195FD.20409@pentabarf.de>
	<20110410145857.GA2073@yoink.cs.uwaterloo.ca>
	<87mxjx5gkl.fsf@vigenere.g10code.de>
	<20110411114836.GA30160@thunk.cs.uwaterloo.ca>
	<8739lo6bi7.fsf@vigenere.g10code.de>
	<20110411170302.GP30160@thunk.cs.uwaterloo.ca>
Message-ID: <87y63g4ruc.fsf@vigenere.g10code.de>

On Mon, 11 Apr 2011 19:03, ian at cypherpunks.ca said:

> If you do this with x = 0,1,7,8,9,15,16,17,22, that should be a good set
> of test cases, I think.

I was more thinking of a testcase you use in libotr.  Anyway, I added a
simple truncation tests which was easy to add to the existsing tests.
Find attached a patch.


Salam-Shalom,

   Werner

-- 
Die Gedanken sind frei.  Ausnahmen regelt ein Bundesgesetz.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Allow-for-truncation-in-CTR-mode.patch
Type: text/x-patch
Size: 9857 bytes
Desc: not available
URL: </pipermail/attachments/20110411/95452bfe/attachment.bin>

From ian at cypherpunks.ca  Mon Apr 11 20:20:23 2011
From: ian at cypherpunks.ca (Ian Goldberg)
Date: Mon, 11 Apr 2011 14:20:23 -0400
Subject: [OTR-dev] Issues with libgcrypt 1.5
In-Reply-To: <87y63g4ruc.fsf@vigenere.g10code.de>
References: <4DA195FD.20409@pentabarf.de>
	<20110410145857.GA2073@yoink.cs.uwaterloo.ca>
	<87mxjx5gkl.fsf@vigenere.g10code.de>
	<20110411114836.GA30160@thunk.cs.uwaterloo.ca>
	<8739lo6bi7.fsf@vigenere.g10code.de>
	<20110411170302.GP30160@thunk.cs.uwaterloo.ca>
	<87y63g4ruc.fsf@vigenere.g10code.de>
Message-ID: <20110411182023.GX30160@thunk.cs.uwaterloo.ca>

On Mon, Apr 11, 2011 at 07:31:55PM +0200, Werner Koch wrote:
> On Mon, 11 Apr 2011 19:03, ian at cypherpunks.ca said:
> 
> > If you do this with x = 0,1,7,8,9,15,16,17,22, that should be a good set
> > of test cases, I think.
> 
> I was more thinking of a testcase you use in libotr.  Anyway, I added a
> simple truncation tests which was easy to add to the existsing tests.
> Find attached a patch.

Excellent, thanks!  I'll have Rob test OTR with this patch, and I'll let
you know whether it all works now.

   - Ian

> >From 3c18377a55085faf4df745034056bac53565effa Mon Sep 17 00:00:00 2001
> From: Werner Koch <wk at gnupg.org>
> Date: Mon, 11 Apr 2011 19:21:47 +0200
> Subject: [PATCH] Allow for truncation in CTR mode.
> 
> This re-enables the behaviour of Libgcrypt 1.4.  Such truncation is
> used by libotr and the current error-ed out here.  The bug was
> introduced due to a rewrite of the function and the undocumented
> feature of truncating OTR data.
> ---
>  cipher/ChangeLog |    5 ++
>  cipher/cipher.c  |   12 ++--
>  tests/ChangeLog  |    6 ++
>  tests/basic.c    |  136 ++++++++++++++++++++++++++++++++++++++++++-----------
>  4 files changed, 124 insertions(+), 35 deletions(-)
> 
> diff --git a/cipher/ChangeLog b/cipher/ChangeLog
> index df27bab..4cde857 100644
> --- a/cipher/ChangeLog
> +++ b/cipher/ChangeLog
> @@ -1,3 +1,8 @@
> +2011-04-11  Werner Koch  <wk at g10code.com>
> +
> +	* cipher.c (do_ctr_encrypt): Allow arbitrary length inputs to
> +	match the 1.4 behaviour.
> +
>  2011-04-04  Werner Koch  <wk at g10code.com>
>  
>  	* ecc.c (compute_keygrip): Release L1 while parsing "curve".
> diff --git a/cipher/cipher.c b/cipher/cipher.c
> index a2f8bb9..e5bb2e0 100644
> --- a/cipher/cipher.c
> +++ b/cipher/cipher.c
> @@ -1453,22 +1453,22 @@ do_ctr_encrypt (gcry_cipher_hd_t c,
>    unsigned int blocksize = c->cipher->blocksize;
>    unsigned int nblocks;
>  
> -  /* FIXME: This code does only work on complete blocks.  */
> -
>    if (outbuflen < inbuflen)
>      return GPG_ERR_BUFFER_TOO_SHORT;
>  
> -  if ((inbuflen % blocksize))
> -    return GPG_ERR_INV_LENGTH;
> -
> +  /* Use a bulk method if available.  */
>    nblocks = inbuflen / blocksize;
>    if (nblocks && c->bulk.ctr_enc)
>      {
>        c->bulk.ctr_enc (&c->context.c, c->u_ctr.ctr, outbuf, inbuf, nblocks);
>        inbuf  += nblocks * blocksize;
>        outbuf += nblocks * blocksize;
> +      inbuflen -= nblocks * blocksize;
>      }
> -  else
> +
> +  /* If we don't have a bulk method use the standard method.  We also
> +     use this method for the a remaining partial block.  */
> +  if (inbuflen)
>      {
>        unsigned char tmp[MAX_BLOCKSIZE];
>  
> diff --git a/tests/ChangeLog b/tests/ChangeLog
> index 0f5918a..3793149 100644
> --- a/tests/ChangeLog
> +++ b/tests/ChangeLog
> @@ -1,3 +1,9 @@
> +2011-04-11  Werner Koch  <wk at g10code.com>
> +
> +	* basic.c (mismatch): New.
> +	(check_ctr_cipher): Remove length error code checks.  Add
> +	truncation checks.
> +
>  2011-04-04  Werner Koch  <wk at g10code.com>
>  
>  	* keygrip.c (main): Add option --repetitions.
> diff --git a/tests/basic.c b/tests/basic.c
> index 185091e..a20e731 100644
> --- a/tests/basic.c
> +++ b/tests/basic.c
> @@ -69,6 +69,22 @@ fail (const char *format, ...)
>  }
>  
>  static void
> +mismatch (const void *expected, size_t expectedlen,
> +          const void *computed, size_t computedlen)
> +{
> +  const unsigned char *p;
> +
> +  fprintf (stderr, "expected:");
> +  for (p = expected; expectedlen; p++, expectedlen--)
> +    fprintf (stderr, " %02x", *p);
> +  fprintf (stderr, "\ncomputed:");
> +  for (p = computed; computedlen; p++, computedlen--)
> +    fprintf (stderr, " %02x", *p);
> +  fprintf (stderr, "\n");
> +}
> +
> +
> +static void
>  die (const char *format, ...)
>  {
>    va_list arg_ptr;
> @@ -349,8 +365,7 @@ check_ctr_cipher (void)
>        unsigned char plaintext[MAX_DATA_LEN];
>        int inlen;
>        char out[MAX_DATA_LEN];
> -    }
> -    data[MAX_DATA_LEN];
> +    } data[5];
>    } tv[] =
>      {
>        /* http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf */
> @@ -369,6 +384,8 @@ check_ctr_cipher (void)
>  	  { "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17\xad\x2b\x41\x7b\xe6\x6c\x37\x10",
>  	    16,
>  	    "\x1e\x03\x1d\xda\x2f\xbe\x03\xd1\x79\x21\x70\xa0\xf3\x00\x9c\xee" },
> +
> +          { "", 0, "" }
>  	}
>        },
>        {	GCRY_CIPHER_AES192,
> @@ -387,6 +404,7 @@ check_ctr_cipher (void)
>  	  { "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17\xad\x2b\x41\x7b\xe6\x6c\x37\x10",
>  	    16,
>  	    "\x4f\x78\xa7\xf6\xd2\x98\x09\x58\x5a\x97\xda\xec\x58\xc6\xb0\x50" },
> +          { "", 0, "" }
>  	}
>        },
>        {	GCRY_CIPHER_AES256,
> @@ -404,7 +422,80 @@ check_ctr_cipher (void)
>  	    "\x2b\x09\x30\xda\xa2\x3d\xe9\x4c\xe8\x70\x17\xba\x2d\x84\x98\x8d" },
>  	  { "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17\xad\x2b\x41\x7b\xe6\x6c\x37\x10",
>  	    16,
> -	    "\xdf\xc9\xc5\x8d\xb6\x7a\xad\xa6\x13\xc2\xdd\x08\x45\x79\x41\xa6" }
> +	    "\xdf\xc9\xc5\x8d\xb6\x7a\xad\xa6\x13\xc2\xdd\x08\x45\x79\x41\xa6" },
> +          { "", 0, "" }
> +	}
> +      },
> +      /* Some truncation tests.  With a truncated second block and
> +         also with a single truncated block.  */
> +      {	GCRY_CIPHER_AES,
> +	"\x2b\x7e\x15\x16\x28\xae\xd2\xa6\xab\xf7\x15\x88\x09\xcf\x4f\x3c",
> +	"\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff",
> +	{{"\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96\xe9\x3d\x7e\x11\x73\x93\x17\x2a",
> +          16,
> +          "\x87\x4d\x61\x91\xb6\x20\xe3\x26\x1b\xef\x68\x64\x99\x0d\xb6\xce" },
> +         {"\xae\x2d\x8a\x57\x1e\x03\xac\x9c\x9e\xb7\x6f\xac\x45\xaf\x8e",
> +          15,
> +          "\x98\x06\xf6\x6b\x79\x70\xfd\xff\x86\x17\x18\x7b\xb9\xff\xfd" },
> +         {"", 0, "" }
> +	}
> +      },
> +      {	GCRY_CIPHER_AES,
> +	"\x2b\x7e\x15\x16\x28\xae\xd2\xa6\xab\xf7\x15\x88\x09\xcf\x4f\x3c",
> +	"\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff",
> +	{{"\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96\xe9\x3d\x7e\x11\x73\x93\x17\x2a",
> +          16,
> +          "\x87\x4d\x61\x91\xb6\x20\xe3\x26\x1b\xef\x68\x64\x99\x0d\xb6\xce" },
> +         {"\xae",
> +          1,
> +          "\x98" },
> +         {"", 0, "" }
> +	}
> +      },
> +      {	GCRY_CIPHER_AES,
> +	"\x2b\x7e\x15\x16\x28\xae\xd2\xa6\xab\xf7\x15\x88\x09\xcf\x4f\x3c",
> +	"\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff",
> +	{{"\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96\xe9\x3d\x7e\x11\x73\x93\x17",
> +          15,
> +          "\x87\x4d\x61\x91\xb6\x20\xe3\x26\x1b\xef\x68\x64\x99\x0d\xb6" },
> +         {"", 0, "" }
> +	}
> +      },
> +      {	GCRY_CIPHER_AES,
> +	"\x2b\x7e\x15\x16\x28\xae\xd2\xa6\xab\xf7\x15\x88\x09\xcf\x4f\x3c",
> +	"\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff",
> +	{{"\x6b",
> +          1,
> +          "\x87" },
> +         {"", 0, "" }
> +	}
> +      },
> +#if USE_CAST5
> +      /* A selfmade test vector using an 64 bit block cipher.  */
> +      {	GCRY_CIPHER_CAST5,
> +	"\x2b\x7e\x15\x16\x28\xae\xd2\xa6\xab\xf7\x15\x88\x09\xcf\x4f\x3c",
> +	"\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8",
> +        {{"\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96\xe9\x3d\x7e\x11\x73\x93\x17\x2a",
> +          16,
> +          "\xe8\xa7\xac\x68\xca\xca\xa0\x20\x10\xcb\x1b\xcc\x79\x2c\xc4\x48" },
> +         {"\xae\x2d\x8a\x57\x1e\x03\xac\x9c",
> +          8,
> +          "\x16\xe8\x72\x77\xb0\x98\x29\x68" },
> +         {"\x9e\xb7\x6f\xac\x45\xaf\x8e\x51",
> +          8,
> +          "\x9a\xb3\xa8\x03\x3b\xb4\x14\xba" },
> +         {"\xae\x2d\x8a\x57\x1e\x03\xac\x9c\xa1\x00",
> +          10,
> +          "\x31\x5e\xd3\xfb\x1b\x8d\xd1\xf9\xb0\x83" },
> +         { "", 0, "" }
> +	}
> +      },
> +#endif /*USE_CAST5*/
> +      {	0,
> +	"",
> +	"",
> +	{
> +         {"", 0, "" }
>  	}
>        }
>      };
> @@ -417,6 +508,9 @@ check_ctr_cipher (void)
>      fprintf (stderr, "  Starting CTR cipher checks.\n");
>    for (i = 0; i < sizeof (tv) / sizeof (tv[0]); i++)
>      {
> +      if (!tv[i].algo)
> +        continue;
> +
>        err = gcry_cipher_open (&hde, tv[i].algo, GCRY_CIPHER_MODE_CTR, 0);
>        if (!err)
>  	err = gcry_cipher_open (&hdd, tv[i].algo, GCRY_CIPHER_MODE_CTR, 0);
> @@ -485,7 +579,11 @@ check_ctr_cipher (void)
>  	    }
>  
>  	  if (memcmp (tv[i].data[j].out, out, tv[i].data[j].inlen))
> -	    fail ("aes-ctr, encrypt mismatch entry %d:%d\n", i, j);
> +            {
> +              fail ("aes-ctr, encrypt mismatch entry %d:%d\n", i, j);
> +              mismatch (tv[i].data[j].out, tv[i].data[j].inlen,
> +                        out, tv[i].data[j].inlen);
> +            }
>  
>  	  err = gcry_cipher_decrypt (hdd, out, tv[i].data[j].inlen, NULL, 0);
>  	  if (err)
> @@ -498,7 +596,11 @@ check_ctr_cipher (void)
>  	    }
>  
>  	  if (memcmp (tv[i].data[j].plaintext, out, tv[i].data[j].inlen))
> -	    fail ("aes-ctr, decrypt mismatch entry %d:%d\n", i, j);
> +            {
> +              fail ("aes-ctr, decrypt mismatch entry %d:%d\n", i, j);
> +              mismatch (tv[i].data[j].plaintext, tv[i].data[j].inlen,
> +                        out, tv[i].data[j].inlen);
> +            }
>  
>          }
>  
> @@ -509,18 +611,6 @@ check_ctr_cipher (void)
>        if (err)
>          fail ("aes-ctr, encryption failed for valid input");
>  
> -      err = gcry_cipher_encrypt (hde, out, MAX_DATA_LEN,
> -                                 "1234567890123456", 15);
> -      if (gpg_err_code (err) != GPG_ERR_INV_LENGTH)
> -        fail ("aes-ctr, too short input returned wrong error: %s\n",
> -              gpg_strerror (err));
> -
> -      err = gcry_cipher_encrypt (hde, out, MAX_DATA_LEN,
> -                                 "12345678901234567", 17);
> -      if (gpg_err_code (err) != GPG_ERR_INV_LENGTH)
> -        fail ("aes-ctr, too long input returned wrong error: %s\n",
> -              gpg_strerror (err));
> -
>        err = gcry_cipher_encrypt (hde, out, 15,
>                                   "1234567890123456", 16);
>        if (gpg_err_code (err) != GPG_ERR_BUFFER_TOO_SHORT)
> @@ -545,18 +635,6 @@ check_ctr_cipher (void)
>        if (err)
>          fail ("aes-ctr, decryption failed for valid input");
>  
> -      err = gcry_cipher_decrypt (hde, out, MAX_DATA_LEN,
> -                                 "1234567890123456", 15);
> -      if (gpg_err_code (err) != GPG_ERR_INV_LENGTH)
> -        fail ("aes-ctr, too short input returned wrong error: %s\n",
> -              gpg_strerror (err));
> -
> -      err = gcry_cipher_decrypt (hde, out, MAX_DATA_LEN,
> -                                 "12345678901234567", 17);
> -      if (gpg_err_code (err) != GPG_ERR_INV_LENGTH)
> -        fail ("aes-ctr, too long input returned wrong error: %s\n",
> -              gpg_strerror (err));
> -
>        err = gcry_cipher_decrypt (hde, out, 15,
>                                   "1234567890123456", 16);
>        if (gpg_err_code (err) != GPG_ERR_BUFFER_TOO_SHORT)
> -- 
> 1.7.2.3
> 



From wk at gnupg.org  Mon Apr 11 21:44:08 2011
From: wk at gnupg.org (Werner Koch)
Date: Mon, 11 Apr 2011 21:44:08 +0200
Subject: [OTR-dev] Issues with libgcrypt 1.5
In-Reply-To: <20110411182023.GX30160@thunk.cs.uwaterloo.ca> (Ian Goldberg's
	message of "Mon, 11 Apr 2011 14:20:23 -0400")
References: <4DA195FD.20409@pentabarf.de>
	<20110410145857.GA2073@yoink.cs.uwaterloo.ca>
	<87mxjx5gkl.fsf@vigenere.g10code.de>
	<20110411114836.GA30160@thunk.cs.uwaterloo.ca>
	<8739lo6bi7.fsf@vigenere.g10code.de>
	<20110411170302.GP30160@thunk.cs.uwaterloo.ca>
	<87y63g4ruc.fsf@vigenere.g10code.de>
	<20110411182023.GX30160@thunk.cs.uwaterloo.ca>
Message-ID: <87tye44lpz.fsf@vigenere.g10code.de>

On Mon, 11 Apr 2011 20:20, ian at cypherpunks.ca said:

> Excellent, thanks!  I'll have Rob test OTR with this patch, and I'll let
> you know whether it all works now.

Meanwhile I added the real stream mode for CTR (like with CFB).  All
pushed to the repo; you may also use the attached patch.

Shalom-Salam,

   Werner

-- 
Die Gedanken sind frei.  Ausnahmen regelt ein Bundesgesetz.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-CTR-mode-may-now-be-used-with-arbitrary-long-data-ch.patch
Type: text/x-patch
Size: 6845 bytes
Desc: not available
URL: </pipermail/attachments/20110411/d598a075/attachment.bin>

From ian at cypherpunks.ca  Wed Apr 13 00:00:11 2011
From: ian at cypherpunks.ca (Ian Goldberg)
Date: Tue, 12 Apr 2011 18:00:11 -0400
Subject: [OTR-dev] Issues with libgcrypt 1.5
In-Reply-To: <87tye44lpz.fsf@vigenere.g10code.de>
References: <4DA195FD.20409@pentabarf.de>
	<20110410145857.GA2073@yoink.cs.uwaterloo.ca>
	<87mxjx5gkl.fsf@vigenere.g10code.de>
	<20110411114836.GA30160@thunk.cs.uwaterloo.ca>
	<8739lo6bi7.fsf@vigenere.g10code.de>
	<20110411170302.GP30160@thunk.cs.uwaterloo.ca>
	<87y63g4ruc.fsf@vigenere.g10code.de>
	<20110411182023.GX30160@thunk.cs.uwaterloo.ca>
	<87tye44lpz.fsf@vigenere.g10code.de>
Message-ID: <20110412220011.GD2073@yoink.cs.uwaterloo.ca>

On Mon, Apr 11, 2011 at 09:44:08PM +0200, Werner Koch wrote:
> On Mon, 11 Apr 2011 20:20, ian at cypherpunks.ca said:
> 
> > Excellent, thanks!  I'll have Rob test OTR with this patch, and I'll let
> > you know whether it all works now.
> 
> Meanwhile I added the real stream mode for CTR (like with CFB).  All
> pushed to the repo; you may also use the attached patch.

Rob confirms that OTR works again with current git.

Thanks!

   - Ian