From wk at gnupg.org Tue Dec 2 18:47:26 2008 From: wk at gnupg.org (Werner Koch) Date: Tue, 02 Dec 2008 18:47:26 +0100 Subject: [Help-gnutls] Alternate random device for certtool In-Reply-To: <4930FB82.2090603@gnutls.org> (Nikos Mavrogiannopoulos's message of "Sat, 29 Nov 2008 10:21:22 +0200") References: <87vdu9qdtt.fsf@squeak.fifthhorseman.net> <87y6z5m2c8.fsf@squeak.fifthhorseman.net> <4930FB82.2090603@gnutls.org> Message-ID: <87tz9mzcy9.fsf@wheatstone.g10code.de> On Sat, 29 Nov 2008 09:21, nmav at gnutls.org said: > I CC to gcrypt-devel since this might be gcrypt related. > Could it be that newer versions from 1.4.1 ignore the control: > gcry_control (GCRYCTL_ENABLE_QUICK_RANDOM, 0); Can you please send me the example code? Salam-Shalom, Werner -- Die Gedanken sind frei. Auschnahme regelt ein Bundeschgesetz. From sega01 at gmail.com Tue Dec 2 20:05:11 2008 From: sega01 at gmail.com (Teran McKinney) Date: Tue, 2 Dec 2008 19:05:11 +0000 Subject: [Help-gnutls] Alternate random device for certtool In-Reply-To: <87tz9mzcy9.fsf@wheatstone.g10code.de> References: <87vdu9qdtt.fsf@squeak.fifthhorseman.net> <87y6z5m2c8.fsf@squeak.fifthhorseman.net> <4930FB82.2090603@gnutls.org> <87tz9mzcy9.fsf@wheatstone.g10code.de> Message-ID: Just verified that I do not have this issue when I use libgcrypt 1.4.1. 1.4.2 also seems to have this problem. Thanks, Teran On Tue, Dec 2, 2008 at 17:47, Werner Koch wrote: > On Sat, 29 Nov 2008 09:21, nmav at gnutls.org said: > >> I CC to gcrypt-devel since this might be gcrypt related. >> Could it be that newer versions from 1.4.1 ignore the control: >> gcry_control (GCRYCTL_ENABLE_QUICK_RANDOM, 0); > > Can you please send me the example code? > > > Salam-Shalom, > > Werner > > > -- > Die Gedanken sind frei. Auschnahme regelt ein Bundeschgesetz. > > From nmav at gnutls.org Wed Dec 3 08:37:50 2008 From: nmav at gnutls.org (Nikos Mavrogiannopoulos) Date: Wed, 03 Dec 2008 09:37:50 +0200 Subject: [Help-gnutls] Alternate random device for certtool In-Reply-To: <87tz9mzcy9.fsf@wheatstone.g10code.de> References: <87vdu9qdtt.fsf@squeak.fifthhorseman.net> <87y6z5m2c8.fsf@squeak.fifthhorseman.net> <4930FB82.2090603@gnutls.org> <87tz9mzcy9.fsf@wheatstone.g10code.de> Message-ID: <4936374E.9030807@gnutls.org> Werner Koch wrote: > On Sat, 29 Nov 2008 09:21, nmav at gnutls.org said: > >> I CC to gcrypt-devel since this might be gcrypt related. >> Could it be that newer versions from 1.4.1 ignore the control: >> gcry_control (GCRYCTL_ENABLE_QUICK_RANDOM, 0); > > Can you please send me the example code? It it the certtool program from gnutls (src/certtool.c) the generate_private_key function. This after all calls: pk-libgcrypt.c: _rsa_generate_params (bigint_t * resarr, int *resarr_len, int bits) gcry_sexp_build (&parms, NULL, "(genkey(rsa(nbits %d)))", bits); gcry_pk_genkey (&key, parms); regards, Nikos From Mahes at sca.nsw.gov.au Thu Dec 4 03:53:01 2008 From: Mahes at sca.nsw.gov.au (Mahes) Date: Thu, 4 Dec 2008 13:53:01 +1100 Subject: Compiling libgcrypt in AIX - get an error cc: 1501-218 file mpih-add1.S contains an incorrect file suffix Message-ID: <32615D33DBD1AE4791CAC08DDB1CE0B00119781F@scamail1.sca.nsw.gov.au> Please help, I get an error on compiling libgcrpt in AIX 5.3L. See the log attached. Thanks in advance Mahes <> ******************************************************************************************************* This e-mail, and any files transmitted, is intended for the use of the individual or entity to whom it is addressed and must not be resent by the recipient unless the permission of the originator is first obtained. It may contain confidential or privileged information and, if you are not the intended recipient, you must immediately destroy the original transmission and its contents. If you have received this e-mail in error, please notify the originator of the message. Any views expressed in this e-mail do not represent the views of the Sydney Catchment Authority unless otherwise stated. ******************************************************************************************************* -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: dum.txt URL: From Mahes at sca.nsw.gov.au Thu Dec 4 05:39:05 2008 From: Mahes at sca.nsw.gov.au (Mahes) Date: Thu, 4 Dec 2008 15:39:05 +1100 Subject: Compiling error libgcrpt :file mpih-add1.S contains an incorrect file suffix Message-ID: <32615D33DBD1AE4791CAC08DDB1CE0B001197822@scamail1.sca.nsw.gov.au> Configured for: AIX (powerpc-ibm-aix5.3.0.0) bash-3.00# make make all-recursive make[1]: Entering directory `/hde/working_dir_chris/libgcrypt-1.2.4' Making all in m4 make[2]: Entering directory `/hde/working_dir_chris/libgcrypt-1.2.4/m4' make[2]: Nothing to be done for `all'. make[2]: Leaving directory `/hde/working_dir_chris/libgcrypt-1.2.4/m4' Making all in mpi make[2]: Entering directory `/hde/working_dir_chris/libgcrypt-1.2.4/mpi' /bin/sh ../libtool --mode=compile cc -qlanglvl=extc89 -g -c -o mpih-add1.lo `test -f 'mpih-add1.S' || echo './'`mpih-add1.S cc -qlanglvl=extc89 -g -c mpih-add1.S -DPIC -o .libs/mpih-add1.o cc: 1501-218 file mpih-add1.S contains an incorrect file suffix make[2]: *** [mpih-add1.lo] Error 1 make[2]: Leaving directory `/hde/working_dir_chris/libgcrypt-1.2.4/mpi' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/hde/working_dir_chris/libgcrypt-1.2.4' make: *** [all] Error 2 ******************************************************************************************************* This e-mail, and any files transmitted, is intended for the use of the individual or entity to whom it is addressed and must not be resent by the recipient unless the permission of the originator is first obtained. It may contain confidential or privileged information and, if you are not the intended recipient, you must immediately destroy the original transmission and its contents. If you have received this e-mail in error, please notify the originator of the message. Any views expressed in this e-mail do not represent the views of the Sydney Catchment Authority unless otherwise stated. ******************************************************************************************************* -------------- next part -------------- An HTML attachment was scrubbed... URL: From wk at gnupg.org Thu Dec 4 12:36:06 2008 From: wk at gnupg.org (Werner Koch) Date: Thu, 04 Dec 2008 12:36:06 +0100 Subject: [Help-gnutls] Alternate random device for certtool In-Reply-To: <4930FB82.2090603@gnutls.org> (Nikos Mavrogiannopoulos's message of "Sat, 29 Nov 2008 10:21:22 +0200") References: <87vdu9qdtt.fsf@squeak.fifthhorseman.net> <87y6z5m2c8.fsf@squeak.fifthhorseman.net> <4930FB82.2090603@gnutls.org> Message-ID: <87skp4xjdl.fsf@wheatstone.g10code.de> On Sat, 29 Nov 2008 09:21, nmav at gnutls.org said: > I upgraded to gcrypt 1.4.4 and I notice the same delay, and strace shows > that /dev/random is being used even with this flag. What you do in certtool is to call if (info.quick_random != 0) gcry_control (GCRYCTL_ENABLE_QUICK_RANDOM, 0); very early. In fact as the first call to libgcrypt. At that point the library is not initialixed and thus it has not checked whether it is in FIPS mode; thus the default is used which is the more restrictive FIPS mode: /* This may be called before full initialization to degrade the quality of the RNG for the sake of a faster running test suite. */ void _gcry_enable_quick_random_gen (void) { if (fips_mode ()) ; /* Not used. */ else _gcry_rngcsprng_enable_quick_gen (); } As you see the flag can't be set in this case. What you need to do is to set this flag during initialization: That is after a first call to gcry_check_version. This is how it is done in by libgcrypt regression tests. Anyway, using this flag is strongly discouraged. It is only useful for testing. gpg for example refuse to use a key if the random number generator is in this mode and the User ID of the key is not flagged as insecure. That is a bit paranoid but older version of libgcrypt even did not used a strong RNG in the quick mode. If you want to use not so strong keys, you better use the transient-key feature available since 1.4.2: @item transient-key This is only meaningful for RSA keys. This is a flag with no value. If given the RSA key is created using a faster and a somewhat less secure random number generator. This flag may be used for keys which are only used for a short time and do not require full cryptographic strength. Usage example: err = gcry_sexp_build (&key_spec, NULL, gcry_fips_mode_active () ? "(genkey (RSA (nbits %d)))" : "(genkey (RSA (nbits %d)(transient-key)))", p_sizes[testno]); You may use that even with older Libgcrypt versions, however it is ignored then. The fips mode test is required because this flag is refused by gcry_pk_genkey in fips mode. Shalom-Salam, Werner -- Die Gedanken sind frei. Auschnahme regelt ein Bundeschgesetz. From wk at gnupg.org Thu Dec 4 13:17:00 2008 From: wk at gnupg.org (Werner Koch) Date: Thu, 04 Dec 2008 13:17:00 +0100 Subject: Use of custom memory allocators Message-ID: <87oczsxhhf.fsf@wheatstone.g10code.de> Hi, I looked at the gnutls code and found a problem. In gnutls_global_init you do this: if (gcry_control (GCRYCTL_ANY_INITIALIZATION_P) == 0) { const char *p = gcry_check_version (GCRYPT_VERSION); gcry_control (GCRYCTL_INITIALIZATION_FINISHED, NULL, 0); #ifdef DEBUG /* applications may want to override that, so we only use * it in debugging mode. */ gcry_set_log_handler (_gnutls_gcry_log_handler, NULL); #endif } /* for gcrypt in order to be able to allocate memory */ gcry_set_allocation_handler (gnutls_malloc, gnutls_secure_malloc, _gnutls_is_secure_memory, gnutls_realloc, gnutls_free); A minor glitch is that you should call gcry_set_log_handler even before gcry_check_version: This function registers FUNC_LOG as `logging handler', which means that it will be called in case Libgcrypt wants to log a message. This function may and should be used prior to calling `gcry_check_version'. This is so that problems during initializaion can be logged. The real problem however is the use of gcry_set_allocation_handler. This installs a new memory allocator defaulting to to standard malloc/free. Well, for an application using just gnutls this might not be a problem (unless in FIPS mode). However if an application is using gnutls directly or indirectly (e.g. through openldap) and also making direct use of libgcrypt this will change the standard Libgcrypt memory allocators or those set by the actual application. This is a security problem because by using a plain malloc and free it is not anymore guaranteed that all sensitive data is zeroes out as soon as needed. If you really, really want to set other Libgcrypt allocation handlers, you need to do it in the above initalization block and before setting the finished flag. (I'll add an extra sentence to the manual.) Salam-Shalom, Werner -- Die Gedanken sind frei. Auschnahme regelt ein Bundeschgesetz. From nmav at gnutls.org Thu Dec 4 19:52:33 2008 From: nmav at gnutls.org (Nikos Mavrogiannopoulos) Date: Thu, 04 Dec 2008 20:52:33 +0200 Subject: [Help-gnutls] Alternate random device for certtool In-Reply-To: <87skp4xjdl.fsf@wheatstone.g10code.de> References: <87vdu9qdtt.fsf@squeak.fifthhorseman.net> <87y6z5m2c8.fsf@squeak.fifthhorseman.net> <4930FB82.2090603@gnutls.org> <87skp4xjdl.fsf@wheatstone.g10code.de> Message-ID: <493826F1.2060206@gnutls.org> Werner Koch wrote: > On Sat, 29 Nov 2008 09:21, nmav at gnutls.org said: > >> I upgraded to gcrypt 1.4.4 and I notice the same delay, and strace shows >> that /dev/random is being used even with this flag. > > What you do in certtool is to call > > if (info.quick_random != 0) > gcry_control (GCRYCTL_ENABLE_QUICK_RANDOM, 0); [...] > you see the flag can't be set in this case. What you need to do is > to set this flag during initialization: That is after a first call to > gcry_check_version. This is how it is done in by libgcrypt regression > tests. > Anyway, using this flag is strongly discouraged. It is only useful for > testing. gpg for example refuse to use a key if the random number > generator is in this mode and the User ID of the key is not flagged as > insecure. That is a bit paranoid but older version of libgcrypt even > did not used a strong RNG in the quick mode. Why is this? As far as I understand the only difference was that it uses /dev/urandom instead of /dev/random. > If you want to use not so strong keys, you better use the transient-key > feature available since 1.4.2: > > @item transient-key > This is only meaningful for RSA keys. This is a flag with no value. If > given the RSA key is created using a faster and a somewhat less secure > random number generator. This flag may be used for keys which are only > used for a short time and do not require full cryptographic strength. Is this stronger than using /dev/urandom? regards, Nikos From nmav at gnutls.org Thu Dec 4 20:02:26 2008 From: nmav at gnutls.org (Nikos Mavrogiannopoulos) Date: Thu, 04 Dec 2008 21:02:26 +0200 Subject: Use of custom memory allocators In-Reply-To: <87oczsxhhf.fsf@wheatstone.g10code.de> References: <87oczsxhhf.fsf@wheatstone.g10code.de> Message-ID: <49382942.4020304@gnutls.org> Werner Koch wrote: > This function registers FUNC_LOG as `logging handler', which means > that it will be called in case Libgcrypt wants to log a message. > This function may and should be used prior to calling > `gcry_check_version'. > > This is so that problems during initializaion can be logged. Hello! Done! > > The real problem however is the use of gcry_set_allocation_handler. > This installs a new memory allocator defaulting to to standard > malloc/free. Well, for an application using just gnutls this might not > be a problem (unless in FIPS mode). However if an application is using > gnutls directly or indirectly (e.g. through openldap) and also making > direct use of libgcrypt this will change the standard Libgcrypt memory > allocators or those set by the actual application. This is a security > problem because by using a plain malloc and free it is not anymore > guaranteed that all sensitive data is zeroes out as soon as needed. > > If you really, really want to set other Libgcrypt allocation handlers, > you need to do it in the above initalization block and before setting > the finished flag. (I'll add an extra sentence to the manual.) To be honest I don't remember why is this code there. I recollect that libgcrypt required to set those allocation functions and didn't work otherwise but this was literally ages ago :) Can libgcrypt work without setting the memory allocation functions? regards, Nikos From simon at josefsson.org Thu Dec 4 20:37:44 2008 From: simon at josefsson.org (Simon Josefsson) Date: Thu, 04 Dec 2008 20:37:44 +0100 Subject: Use of custom memory allocators References: <87oczsxhhf.fsf@wheatstone.g10code.de> <49382942.4020304@gnutls.org> Message-ID: <87prk7enp3.fsf@mocca.josefsson.org> Nikos Mavrogiannopoulos writes: >> The real problem however is the use of gcry_set_allocation_handler. >> This installs a new memory allocator defaulting to to standard >> malloc/free. Well, for an application using just gnutls this might not >> be a problem (unless in FIPS mode). However if an application is using >> gnutls directly or indirectly (e.g. through openldap) and also making >> direct use of libgcrypt this will change the standard Libgcrypt memory >> allocators or those set by the actual application. This is a security >> problem because by using a plain malloc and free it is not anymore >> guaranteed that all sensitive data is zeroes out as soon as needed. >> >> If you really, really want to set other Libgcrypt allocation handlers, >> you need to do it in the above initalization block and before setting >> the finished flag. (I'll add an extra sentence to the manual.) > > To be honest I don't remember why is this code there. I recollect that > libgcrypt required to set those allocation functions and didn't work > otherwise but this was literally ages ago :) Can libgcrypt work without > setting the memory allocation functions? If I remove that code, using any application that uses the GnuTLS library with some functions just dies: jas at mocca:~/src/gnutls/src master$ ./certtool -p Generating a 2048 bit RSA private key... Ohhhh jeeee: operation is not possible without initialized secure memory Aborted jas at mocca:~/src/gnutls/src master$ Is there a recommended way how to resolve this problem? /Simon From wk at gnupg.org Thu Dec 4 22:00:46 2008 From: wk at gnupg.org (Werner Koch) Date: Thu, 04 Dec 2008 22:00:46 +0100 Subject: [Help-gnutls] Alternate random device for certtool In-Reply-To: <493826F1.2060206@gnutls.org> (Nikos Mavrogiannopoulos's message of "Thu, 04 Dec 2008 20:52:33 +0200") References: <87vdu9qdtt.fsf@squeak.fifthhorseman.net> <87y6z5m2c8.fsf@squeak.fifthhorseman.net> <4930FB82.2090603@gnutls.org> <87skp4xjdl.fsf@wheatstone.g10code.de> <493826F1.2060206@gnutls.org> Message-ID: <87hc5jwt8h.fsf@wheatstone.g10code.de> On Thu, 4 Dec 2008 19:52, nmav at gnutls.org said: >> gcry_control (GCRYCTL_ENABLE_QUICK_RANDOM, 0); > Why is this? As far as I understand the only difference was that it uses > /dev/urandom instead of /dev/random. Because this has always been the case. QUICK_RANDOM was and is just a testing hack. >> @item transient-key > Is this stronger than using /dev/urandom? It is not a matter of being stronger but of being a feature. transient-key is suposed to be used for one-off keys and other keys which are not that valuable. In general it is always better to use the defaults for generating a key; see onl the recent BSD problems with their RNG. This would not have been the case with a blocking one. Shalom-Salam, Werner -- Die Gedanken sind frei. Auschnahme regelt ein Bundeschgesetz. From wk at gnupg.org Thu Dec 4 22:08:05 2008 From: wk at gnupg.org (Werner Koch) Date: Thu, 04 Dec 2008 22:08:05 +0100 Subject: Use of custom memory allocators In-Reply-To: <87prk7enp3.fsf@mocca.josefsson.org> (Simon Josefsson's message of "Thu, 04 Dec 2008 20:37:44 +0100") References: <87oczsxhhf.fsf@wheatstone.g10code.de> <49382942.4020304@gnutls.org> <87prk7enp3.fsf@mocca.josefsson.org> Message-ID: <87d4g7wswa.fsf@wheatstone.g10code.de> On Thu, 4 Dec 2008 20:37, simon at josefsson.org said: > If I remove that code, using any application that uses the GnuTLS > library with some functions just dies: > > jas at mocca:~/src/gnutls/src master$ ./certtool -p > Generating a 2048 bit RSA private key... > Ohhhh jeeee: operation is not possible without initialized secure memory Right, because it is not properly initialized. We need to fix the application and not try to work around such problems. If you don't have a need for secure memory, for example if your application does not use secret keys or other confidential data or it runs in a controlled environment where key material floating around in memory is not a problem, you - or weel the application - should initialize Libgcrypt correctly. I recently updated the manual to make this more clear. Here is an excerpt: | If you have to protect your keys or other information in memory | against being swapped out to disk and to enable an automatic overwrite | of used and freed memory, you need to initialize Libgcrypt this way: | | /* Version check should be the very first call because it | makes sure that important subsystems are intialized. */ | if (!gcry_check_version (GCRYPT_VERSION)) | { | fputs ("libgcrypt version mismatch\n", stderr); | exit (2); | } | | /* Disable secure memory. */ | gcry_control (GCRYCTL_DISABLE_SECMEM, 0); | | /* ... If required, other initialization goes here. */ | | /* Tell Libgcrypt that initialization has completed. */ | gcry_control (GCRYCTL_INITIALIZATION_FINISHED, 0); | | If you have to protect your keys or other information in memory | against being swapped out to disk and to enable an automatic overwrite | of used and freed memory, you need to initialize Libgcrypt this way: | | /* Version check should be the very first call because it | makes sure that important subsystems are intialized. */ | if (!gcry_check_version (GCRYPT_VERSION)) | { | fputs ("libgcrypt version mismatch\n", stderr); | exit (2); | } | | /* We don't want to see any warnings, e.g. because we have not yet | parsed program options which might be used to suppress such | warnings. */ | gcry_control (GCRYCTL_SUSPEND_SECMEM_WARN); | | /* ... If required, other initialization goes here. Note that the | process might still be running with increased privileges and that | the secure memory has not been intialized. */ | | /* Allocate a pool of 16k secure memory. This make the secure memory | available and also drops privileges where needed. */ | gcry_control (GCRYCTL_INIT_SECMEM, 16384, 0); | | /* It is now okay to let Libgcrypt complain when there was/is | a problem with the secure memory. */ | gcry_control (GCRYCTL_RESUME_SECMEM_WARN); | | /* ... If required, other initialization goes here. */ | | /* Tell Libgcrypt that initialization has completed. */ | gcry_control (GCRYCTL_INITIALIZATION_FINISHED, 0); | | It is important that these initialization steps are not done by a | library but by the actual application. A library using Libgcrypt might | want to check for finished initialization using: | | if (!gcry_control (GCRYCTL_INITIALIZATION_FINISHED_P)) | { | fputs ("libgcrypt has not been initialized\n", stderr); | abort (); | } | | Instead of terminating the process, the library may instead print a | warning and try to initialize Libgcrypt itself. See also the section on | multi-threading below for more pitfalls. I am sorry that these things are a bit complicated. However I believe that we should better choose the safe side of things. The crypto library is low-level foundation code and problems there would affect a lot of applications. Better some applications break than applications requiring real security are exploitable. Salam-Shalom, Werner -- Die Gedanken sind frei. Auschnahme regelt ein Bundeschgesetz. From nmav at gnutls.org Fri Dec 5 08:17:58 2008 From: nmav at gnutls.org (Nikos Mavrogiannopoulos) Date: Fri, 05 Dec 2008 09:17:58 +0200 Subject: [Help-gnutls] Alternate random device for certtool In-Reply-To: <87hc5jwt8h.fsf@wheatstone.g10code.de> References: <87vdu9qdtt.fsf@squeak.fifthhorseman.net> <87y6z5m2c8.fsf@squeak.fifthhorseman.net> <4930FB82.2090603@gnutls.org> <87skp4xjdl.fsf@wheatstone.g10code.de> <493826F1.2060206@gnutls.org> <87hc5jwt8h.fsf@wheatstone.g10code.de> Message-ID: <4938D5A6.90408@gnutls.org> Werner Koch wrote: >>> gcry_control (GCRYCTL_ENABLE_QUICK_RANDOM, 0); > >> Why is this? As far as I understand the only difference was that it uses >> /dev/urandom instead of /dev/random. > > Because this has always been the case. QUICK_RANDOM was and is just a > testing hack. I don't understand. The issue for certtool that was reported was that it was blocking in /dev/random and taking a lot of time to produce any output. This was the reason I've put QUICK_RANDOM there. >>> @item transient-key > >> Is this stronger than using /dev/urandom? > > It is not a matter of being stronger but of being a feature. > transient-key is suposed to be used for one-off keys and other keys > which are not that valuable. > In general it is always better to use the > defaults for generating a key; see onl the recent BSD problems with > their RNG. This would not have been the case with a blocking one. I don't think so. Block for indefinite time (can be even hours) does not offer anything unless you can wait. If you want to generate keys and you don't care if this will be today or tomorrow it's ok. In all other cases you will not use this rng, it is broken by design[0]. regards, Nikos [0]. Also being blocking does not protect from being a bad algorithm. As far as I know there are known issues to the blocking linux rng (were discussed some years ago in gnutls-dev) and they still cannot gather any entropy from network devices because its state can be compromised! From nmav at gnutls.org Fri Dec 5 08:50:22 2008 From: nmav at gnutls.org (Nikos Mavrogiannopoulos) Date: Fri, 05 Dec 2008 09:50:22 +0200 Subject: Use of custom memory allocators In-Reply-To: <87d4g7wswa.fsf@wheatstone.g10code.de> References: <87oczsxhhf.fsf@wheatstone.g10code.de> <49382942.4020304@gnutls.org> <87prk7enp3.fsf@mocca.josefsson.org> <87d4g7wswa.fsf@wheatstone.g10code.de> Message-ID: <4938DD3E.4060206@gnutls.org> Werner Koch wrote: > On Thu, 4 Dec 2008 20:37, simon at josefsson.org said: > >> If I remove that code, using any application that uses the GnuTLS >> library with some functions just dies: >> >> jas at mocca:~/src/gnutls/src master$ ./certtool -p >> Generating a 2048 bit RSA private key... >> Ohhhh jeeee: operation is not possible without initialized secure memory > > Right, because it is not properly initialized. We need to fix the > application and not try to work around such problems. I also disagree here :) Have a "secure" memory is only of use to limited applications that conform to some different threat models than a typical PC server is faced with. Those special ones know their requirements and will set this behavior explicitly. For others there should be some reasonable defaults. Also if you disagree with my evaluation that common applications do not require a secure memory, the same argument applies. A reasonable default with secure memory initialized should be available. > If you don't have a need for secure memory, for example if your > application does not use secret keys or other confidential data or it > runs in a controlled environment where key material floating around in > memory is not a problem, you - or weel the application - should > initialize Libgcrypt correctly. As far as I know modern operating systems do not leave random memory floating around and being shared between processes. Also memory is being zeroed before being given to an application. Thus the threat-model for having a special memory marked as "secure" is not quite clear for me and this is the reason it was always by default off in gnutls. > I am sorry that these things are a bit complicated. However I believe > that we should better choose the safe side of things. The crypto > library is low-level foundation code and problems there would affect a > lot of applications. Better some applications break than applications > requiring real security are exploitable. I agree choosing the safe side of things. However it should also be clear what the threat-model of using secure memory will protect from. The additional security offered by it might not worth the inconvenience offered by it -limited secure memory that will cause the application to fail in cases where it was exceeded. regards, Nikos From wk at gnupg.org Fri Dec 5 09:13:24 2008 From: wk at gnupg.org (Werner Koch) Date: Fri, 05 Dec 2008 09:13:24 +0100 Subject: [Help-gnutls] Alternate random device for certtool In-Reply-To: <4938D5A6.90408@gnutls.org> (Nikos Mavrogiannopoulos's message of "Fri, 05 Dec 2008 09:17:58 +0200") References: <87vdu9qdtt.fsf@squeak.fifthhorseman.net> <87y6z5m2c8.fsf@squeak.fifthhorseman.net> <4930FB82.2090603@gnutls.org> <87skp4xjdl.fsf@wheatstone.g10code.de> <493826F1.2060206@gnutls.org> <87hc5jwt8h.fsf@wheatstone.g10code.de> <4938D5A6.90408@gnutls.org> Message-ID: <874p1jvy3f.fsf@wheatstone.g10code.de> On Fri, 5 Dec 2008 08:17, nmav at gnutls.org said: > I don't understand. The issue for certtool that was reported was that it > was blocking in /dev/random and taking a lot of time to produce any > output. This was the reason I've put QUICK_RANDOM there. Right, it is blocking because it needs to generate random numbers and do do this we need to gather entropy from physical sources. If the bandwidth of these sources is that small it just takes a long time. If the box is idle it may even not finish at all. Recall that a computer is a deterministic machine and that it is hard to extract unpredictable events from a deterministic machines (actually impossible, but fortunately a general purpose computer is not completely deterministic.) Ask the user to work on the box to give it a chnace to collect entroy. For example "find /usr -type f | xargs cat >/dev/null" gets the disk to work and thus floods the box with interrupts. > I don't think so. Block for indefinite time (can be even hours) does not > offer anything unless you can wait. If you want to generate keys and you I disagree: If you want a secure key, you need to invest something, be it time to wait for sporadic interrupts, keep on working on the box or even install a hardware RNG. > [0]. Also being blocking does not protect from being a bad algorithm. As > far as I know there are known issues to the blocking linux rng (were > discussed some years ago in gnutls-dev) and they still cannot gather any > entropy from network devices because its state can be compromised! And thus your solution is to give up on it and use a a deterministc source like /dev/urandom? If /dev/random blocks, /dev/urandom will only return a sequence of bytes which is predictable if you know the initial state of the RNG. It al depends on what you want. The default for Libgcrypt is to make sure that there is really strong random available for key generation and to do with a not so strong (read /dev/urandom) for session keys etc. If you don't want that (transient-key) gives you a way to degrade random quality for key generation. Salam-Shalom, Werner -- Die Gedanken sind frei. Auschnahme regelt ein Bundeschgesetz. From wk at gnupg.org Fri Dec 5 11:07:12 2008 From: wk at gnupg.org (Werner Koch) Date: Fri, 05 Dec 2008 11:07:12 +0100 Subject: Use of custom memory allocators In-Reply-To: <4938DD3E.4060206@gnutls.org> (Nikos Mavrogiannopoulos's message of "Fri, 05 Dec 2008 09:50:22 +0200") References: <87oczsxhhf.fsf@wheatstone.g10code.de> <49382942.4020304@gnutls.org> <87prk7enp3.fsf@mocca.josefsson.org> <87d4g7wswa.fsf@wheatstone.g10code.de> <4938DD3E.4060206@gnutls.org> Message-ID: <87myfbue9b.fsf@wheatstone.g10code.de> On Fri, 5 Dec 2008 08:50, nmav at gnutls.org said: > Also if you disagree with my evaluation that common applications do not > require a secure memory, the same argument applies. A reasonable default > with secure memory initialized should be available. There are good defaults available. With recent Linux versions there is no more need to setuid the application to again access to mlock. Thus you get mlock-able memory for free. Just make sure to initialize Libgcrypt properly! Proper initialization is required anyway. > zeroed before being given to an application. Thus the threat-model for > having a special memory marked as "secure" is not quite clear for me and > this is the reason it was always by default off in gnutls. The threat model is that keys war swapped to disk. You may mitigate that by using an encrypted swap space - but how many installations do that? Even OpenBSD does not use encrypted swap by default. Thus mlock-ed memory is the best solution we have. You don't think that is an issue for servers? I tend to agree for SSL keys, however a lot of security policies require zeroization of keys as early as possible. Also, the majority of applications using GNuTLS are client applications and there you really want to safe your keys: User Certificates and keys used by the application not related to gnutls. For example gnupg uses gnutls and has a need to keep its keys safe (granted, gnupg uses an external process to access ldaps and https but other apps might not want to go into that trouble). > The additional security offered by it might not worth the inconvenience > offered by it -limited secure memory that will cause the application to > fail in cases where it was exceeded. You see a problem in the limited amount of secure memory when used with servers? That is a different problem we can solve. However you need to initialize Libgcrypt properly: In the server case with disabled secure memory and let the server application do the libgcrypt initialization and not gnutls. As it stands now, gnutls just overrides good defaults. Shalom-Salam, Werner -- Die Gedanken sind frei. Auschnahme regelt ein Bundeschgesetz. From simon at josefsson.org Fri Dec 5 12:04:17 2008 From: simon at josefsson.org (Simon Josefsson) Date: Fri, 05 Dec 2008 12:04:17 +0100 Subject: Use of custom memory allocators References: <87oczsxhhf.fsf@wheatstone.g10code.de> <49382942.4020304@gnutls.org> <87prk7enp3.fsf@mocca.josefsson.org> <87d4g7wswa.fsf@wheatstone.g10code.de> Message-ID: <873ah2evda.fsf@mocca.josefsson.org> Werner Koch writes: > On Thu, 4 Dec 2008 20:37, simon at josefsson.org said: > >> If I remove that code, using any application that uses the GnuTLS >> library with some functions just dies: >> >> jas at mocca:~/src/gnutls/src master$ ./certtool -p >> Generating a 2048 bit RSA private key... >> Ohhhh jeeee: operation is not possible without initialized secure memory > > Right, because it is not properly initialized. We need to fix the > application and not try to work around such problems. Application? GnuTLS is a library. We don't want to require all applications that use GnuTLS to call libgcrypt explicitly. > If you don't have a need for secure memory, for example if your > application does not use secret keys or other confidential data or it > runs in a controlled environment where key material floating around in > memory is not a problem, you - or weel the application - should > initialize Libgcrypt correctly. I recently updated the manual to make > this more clear. Here is an excerpt: > > | If you have to protect your keys or other information in memory > | against being swapped out to disk and to enable an automatic overwrite > | of used and freed memory, you need to initialize Libgcrypt this way: > | > | /* Version check should be the very first call because it > | makes sure that important subsystems are intialized. */ > | if (!gcry_check_version (GCRYPT_VERSION)) > | { > | fputs ("libgcrypt version mismatch\n", stderr); > | exit (2); > | } > | > | /* Disable secure memory. */ > | gcry_control (GCRYCTL_DISABLE_SECMEM, 0); > | > | /* ... If required, other initialization goes here. */ > | > | /* Tell Libgcrypt that initialization has completed. */ > | gcry_control (GCRYCTL_INITIALIZATION_FINISHED, 0); Ok, this would argue for the following solution: diff --git a/lib/gnutls_global.c b/lib/gnutls_global.c index f59a47f..2c2a91f 100644 --- a/lib/gnutls_global.c +++ b/lib/gnutls_global.c @@ -222,8 +222,7 @@ gnutls_global_init (void) } /* for gcrypt in order to be able to allocate memory */ - gcry_set_allocation_handler (gnutls_malloc, gnutls_secure_malloc, - _gnutls_is_secure_memory, gnutls_realloc, gnutls_free); + gcry_control (GCRYCTL_DISABLE_SECMEM, NULL, 0); gcry_control (GCRYCTL_INITIALIZATION_FINISHED, NULL, 0); > | > | If you have to protect your keys or other information in memory > | against being swapped out to disk and to enable an automatic overwrite > | of used and freed memory, you need to initialize Libgcrypt this way: This is the exact same text as above. What is the difference between these two modes? > | /* Version check should be the very first call because it > | makes sure that important subsystems are intialized. */ > | if (!gcry_check_version (GCRYPT_VERSION)) > | { > | fputs ("libgcrypt version mismatch\n", stderr); > | exit (2); > | } > | > | /* We don't want to see any warnings, e.g. because we have not yet > | parsed program options which might be used to suppress such > | warnings. */ > | gcry_control (GCRYCTL_SUSPEND_SECMEM_WARN); > | > | /* ... If required, other initialization goes here. Note that the > | process might still be running with increased privileges and that > | the secure memory has not been intialized. */ > | > | /* Allocate a pool of 16k secure memory. This make the secure memory > | available and also drops privileges where needed. */ > | gcry_control (GCRYCTL_INIT_SECMEM, 16384, 0); > | > | /* It is now okay to let Libgcrypt complain when there was/is > | a problem with the secure memory. */ > | gcry_control (GCRYCTL_RESUME_SECMEM_WARN); > | > | /* ... If required, other initialization goes here. */ > | > | /* Tell Libgcrypt that initialization has completed. */ > | gcry_control (GCRYCTL_INITIALIZATION_FINISHED, 0); What does "increased privileges" mean? Does the application needs to be setuid for this to work? That would also be a non-starter, we can't require all applications using GnuTLS to be setuid. > | It is important that these initialization steps are not done by a > | library but by the actual application. That seems like a non-starter for GnuTLS. If it is important for libgcrypt that GnuTLS doesn't initialize libgcrypt, it seems we can't really use libgcrypt in GnuTLS. > | A library using Libgcrypt might want to check for finished > | initialization using: > | > | if (!gcry_control (GCRYCTL_INITIALIZATION_FINISHED_P)) > | { > | fputs ("libgcrypt has not been initialized\n", stderr); > | abort (); > | } > | > | Instead of terminating the process, the library may instead print a > | warning and try to initialize Libgcrypt itself. This is what GnuTLS is doing now, except it is not printing a warning. What use is there in printing a warning? > I am sorry that these things are a bit complicated. However I believe > that we should better choose the safe side of things. The crypto > library is low-level foundation code and problems there would affect a > lot of applications. Better some applications break than applications > requiring real security are exploitable. The best would that things just work _and_ be secure. I don't see why it isn't possible to reach that goal here? /Simon From wk at gnupg.org Fri Dec 5 12:41:28 2008 From: wk at gnupg.org (Werner Koch) Date: Fri, 05 Dec 2008 12:41:28 +0100 Subject: Use of custom memory allocators In-Reply-To: <873ah2evda.fsf@mocca.josefsson.org> (Simon Josefsson's message of "Fri, 05 Dec 2008 12:04:17 +0100") References: <87oczsxhhf.fsf@wheatstone.g10code.de> <49382942.4020304@gnutls.org> <87prk7enp3.fsf@mocca.josefsson.org> <87d4g7wswa.fsf@wheatstone.g10code.de> <873ah2evda.fsf@mocca.josefsson.org> Message-ID: <8763lyvogn.fsf@wheatstone.g10code.de> On Fri, 5 Dec 2008 12:04, simon at josefsson.org said: > Application? GnuTLS is a library. We don't want to require all > applications that use GnuTLS to call libgcrypt explicitly. Right, this is why we have this GCRYCTL_ANY_INITIALIZATION_P feature. It allows to do an initialization if the application missed to do it. > Ok, this would argue for the following solution: > > diff --git a/lib/gnutls_global.c b/lib/gnutls_global.c > index f59a47f..2c2a91f 100644 > --- a/lib/gnutls_global.c > +++ b/lib/gnutls_global.c > @@ -222,8 +222,7 @@ gnutls_global_init (void) > } > > /* for gcrypt in order to be able to allocate memory */ > - gcry_set_allocation_handler (gnutls_malloc, gnutls_secure_malloc, > - _gnutls_is_secure_memory, gnutls_realloc, gnutls_free); > + gcry_control (GCRYCTL_DISABLE_SECMEM, NULL, 0); > > gcry_control (GCRYCTL_INITIALIZATION_FINISHED, NULL, 0); Fine. This is a proper solution. If an applications needs to use secure memory it can do so and gnutls uses it too. You might want to state int the manual that gnutls disables the secure memory unless an application has already initialzied libgcrypt. > This is the exact same text as above. What is the difference between > these two modes? Sorry, cut and paste error. It should read: If you don't have a need for secure memory, for example if your application does not use secret keys or other confidential data or it runs in a controlled environment where key material floating around in memory is not a problem, you should initialize Libgcrypt this way: [... GCRYCTL_DISABLE_SECMEM ...] If you have to protect your keys or other information in memory against being swapped out to disk and to enable an automatic overwrite of used and freed memory, you need to initialize Libgcrypt this way: [... GCRYCTL_INIT_SECMEM ...] > What does "increased privileges" mean? Does the application needs to be > setuid for this to work? That would also be a non-starter, we can't > require all applications using GnuTLS to be setuid. Depends on the number of pages allocated for the secure memory and the OS. Current linux version can controll the mlock-able pages with ulimit. Right, the application needs to initialize that. A library can't know how much secure memory is required. Another library initialized later might have a different idea of the required amount of secure memory and thus it would be unpredictable. The only solution is that the application decides and thus initializes libgcrypt. > That seems like a non-starter for GnuTLS. If it is important for > libgcrypt that GnuTLS doesn't initialize libgcrypt, it seems we can't > really use libgcrypt in GnuTLS. We discussed that in the past ad nauseam when talking about thread library initialization. In contrast to W32, the shared libray system used in GNU/Linux is not up to handle certain tasks. The problem is elevated by using libraries indirectly (app->libfoo->libldap->libgnutls->libgcrypt->libc) without APP being aware that it uses libgnutls. As soon as a library needs a specific global initialization you need to pass the intialization up to the application. For libgcrypt we found a somewhat working solution using GCRYCTL_INITIALIZATION_FINISHED_P. It allows a default to catch the naive use of crypto without the need for specific initialization. > This is what GnuTLS is doing now, except it is not printing a warning. > What use is there in printing a warning? Except that gnutls does not intialize the secure memory. Printing a warning is good to inform the application that it should take care of initialization. I agree that for gnutls such a waning does not make sense if you change the code as you proposed. > The best would that things just work _and_ be secure. I don't see why > it isn't possible to reach that goal here? See above. Also think of the case if you do not want to use threads - all will break if you let the libratry decide which threading model to use. We need some state in libgcrypt: For selftests, for hardware crypto, for the thread library and so on. Shalom-Salam, Werner -- Die Gedanken sind frei. Auschnahme regelt ein Bundeschgesetz. From simon at josefsson.org Fri Dec 5 14:13:55 2008 From: simon at josefsson.org (Simon Josefsson) Date: Fri, 05 Dec 2008 14:13:55 +0100 Subject: Use of custom memory allocators References: <87oczsxhhf.fsf@wheatstone.g10code.de> <49382942.4020304@gnutls.org> <87prk7enp3.fsf@mocca.josefsson.org> <87d4g7wswa.fsf@wheatstone.g10code.de> <873ah2evda.fsf@mocca.josefsson.org> <8763lyvogn.fsf@wheatstone.g10code.de> Message-ID: <87wseebw8c.fsf@mocca.josefsson.org> Werner Koch writes: > On Fri, 5 Dec 2008 12:04, simon at josefsson.org said: > >> Application? GnuTLS is a library. We don't want to require all >> applications that use GnuTLS to call libgcrypt explicitly. > > Right, this is why we have this GCRYCTL_ANY_INITIALIZATION_P feature. > It allows to do an initialization if the application missed to do it. Great. I got the impression from the documentation that this isn't the recommend approach though. >> Ok, this would argue for the following solution: >> >> diff --git a/lib/gnutls_global.c b/lib/gnutls_global.c >> index f59a47f..2c2a91f 100644 >> --- a/lib/gnutls_global.c >> +++ b/lib/gnutls_global.c >> @@ -222,8 +222,7 @@ gnutls_global_init (void) >> } >> >> /* for gcrypt in order to be able to allocate memory */ >> - gcry_set_allocation_handler (gnutls_malloc, gnutls_secure_malloc, >> - _gnutls_is_secure_memory, gnutls_realloc, gnutls_free); >> + gcry_control (GCRYCTL_DISABLE_SECMEM, NULL, 0); >> >> gcry_control (GCRYCTL_INITIALIZATION_FINISHED, NULL, 0); > > Fine. This is a proper solution. If an applications needs to use > secure memory it can do so and gnutls uses it too. You might want to > state int the manual that gnutls disables the secure memory unless an > application has already initialzied libgcrypt. Yup. And we could also recommend applications to initialize secure memory if they want. I'll test the patch above more. >> This is the exact same text as above. What is the difference between >> these two modes? > > Sorry, cut and paste error. It should read: > > If you don't have a need for secure memory, for example if your > application does not use secret keys or other confidential data or it > runs in a controlled environment where key material floating around in > memory is not a problem, you should initialize Libgcrypt this way: > > [... GCRYCTL_DISABLE_SECMEM ...] > > If you have to protect your keys or other information in memory against > being swapped out to disk and to enable an automatic overwrite of used > and freed memory, you need to initialize Libgcrypt this way: > > [... GCRYCTL_INIT_SECMEM ...] Ah, that makes more sense. >> What does "increased privileges" mean? Does the application needs to be >> setuid for this to work? That would also be a non-starter, we can't >> require all applications using GnuTLS to be setuid. > > Depends on the number of pages allocated for the secure memory and the > OS. Current linux version can controll the mlock-able pages with > ulimit. > > Right, the application needs to initialize that. A library can't know > how much secure memory is required. Another library initialized later > might have a different idea of the required amount of secure memory and > thus it would be unpredictable. The only solution is that the > application decides and thus initializes libgcrypt. Ok. Using the patch above, and documenting this, seems to be the best we can achieve then. >> That seems like a non-starter for GnuTLS. If it is important for >> libgcrypt that GnuTLS doesn't initialize libgcrypt, it seems we can't >> really use libgcrypt in GnuTLS. > > We discussed that in the past ad nauseam when talking about thread > library initialization. In contrast to W32, the shared libray system > used in GNU/Linux is not up to handle certain tasks. The problem is > elevated by using libraries indirectly > (app->libfoo->libldap->libgnutls->libgcrypt->libc) without APP being > aware that it uses libgnutls. As soon as a library needs a specific > global initialization you need to pass the intialization up to the > application. > > For libgcrypt we found a somewhat working solution using > GCRYCTL_INITIALIZATION_FINISHED_P. It allows a default to catch the > naive use of crypto without the need for specific initialization. Still, it seems like it would be possible to implement this in libgcrypt without the requirement to modify applications or workarounds. >> This is what GnuTLS is doing now, except it is not printing a warning. >> What use is there in printing a warning? > > Except that gnutls does not intialize the secure memory. Printing a > warning is good to inform the application that it should take care of > initialization. I agree that for gnutls such a waning does not make > sense if you change the code as you proposed. OK. >> The best would that things just work _and_ be secure. I don't see why >> it isn't possible to reach that goal here? > > See above. Also think of the case if you do not want to use threads - > all will break if you let the libratry decide which threading model to > use. We need some state in libgcrypt: For selftests, for hardware > crypto, for the thread library and so on. OK. /Simon From linux at paip.net Fri Dec 5 14:14:41 2008 From: linux at paip.net (Ian Goldberg) Date: Fri, 5 Dec 2008 08:14:41 -0500 Subject: [Help-gnutls] Alternate random device for certtool In-Reply-To: <874p1jvy3f.fsf@wheatstone.g10code.de> References: <87vdu9qdtt.fsf@squeak.fifthhorseman.net> <87y6z5m2c8.fsf@squeak.fifthhorseman.net> <4930FB82.2090603@gnutls.org> <87skp4xjdl.fsf@wheatstone.g10code.de> <493826F1.2060206@gnutls.org> <87hc5jwt8h.fsf@wheatstone.g10code.de> <4938D5A6.90408@gnutls.org> <874p1jvy3f.fsf@wheatstone.g10code.de> Message-ID: <20081205131441.GA11159@thunk.cs.uwaterloo.ca> On Fri, Dec 05, 2008 at 09:13:24AM +0100, Werner Koch wrote: > It al depends on what you want. The default for Libgcrypt is to make > sure that there is really strong random available for key generation and > to do with a not so strong (read /dev/urandom) for session keys etc. If > you don't want that (transient-key) gives you a way to degrade random > quality for key generation. It's my understanding that (transient-key) only works for RSA. Can it be made to work for DSA as well? We hit this problem in libotr. Thanks, - Ian From wk at gnupg.org Fri Dec 5 19:00:29 2008 From: wk at gnupg.org (Werner Koch) Date: Fri, 05 Dec 2008 19:00:29 +0100 Subject: [Help-gnutls] Alternate random device for certtool In-Reply-To: <20081205131441.GA11159@thunk.cs.uwaterloo.ca> (Ian Goldberg's message of "Fri, 5 Dec 2008 08:14:41 -0500") References: <87vdu9qdtt.fsf@squeak.fifthhorseman.net> <87y6z5m2c8.fsf@squeak.fifthhorseman.net> <4930FB82.2090603@gnutls.org> <87skp4xjdl.fsf@wheatstone.g10code.de> <493826F1.2060206@gnutls.org> <87hc5jwt8h.fsf@wheatstone.g10code.de> <4938D5A6.90408@gnutls.org> <874p1jvy3f.fsf@wheatstone.g10code.de> <20081205131441.GA11159@thunk.cs.uwaterloo.ca> Message-ID: <87skp2tsci.fsf@wheatstone.g10code.de> On Fri, 5 Dec 2008 14:14, linux at paip.net said: > It's my understanding that (transient-key) only works for RSA. Can it > be made to work for DSA as well? We hit this problem in libotr. Done. SVN trunk revision 1371; will go into 1.4.4. Sample code: rc = gcry_sexp_new (&key_spec, transient_key ? "(genkey (dsa (nbits 4:1024)(transient-key)))" : "(genkey (dsa (nbits 4:1024)))", 0, 1); Shalom-Salam, Werner -- Die Gedanken sind frei. Auschnahme regelt ein Bundeschgesetz. From nmav at gnutls.org Fri Dec 5 21:06:38 2008 From: nmav at gnutls.org (Nikos Mavrogiannopoulos) Date: Fri, 05 Dec 2008 22:06:38 +0200 Subject: [Help-gnutls] Alternate random device for certtool In-Reply-To: <874p1jvy3f.fsf@wheatstone.g10code.de> References: <87vdu9qdtt.fsf@squeak.fifthhorseman.net> <87y6z5m2c8.fsf@squeak.fifthhorseman.net> <4930FB82.2090603@gnutls.org> <87skp4xjdl.fsf@wheatstone.g10code.de> <493826F1.2060206@gnutls.org> <87hc5jwt8h.fsf@wheatstone.g10code.de> <4938D5A6.90408@gnutls.org> <874p1jvy3f.fsf@wheatstone.g10code.de> Message-ID: <493989CE.908@gnutls.org> Werner Koch wrote: >> I don't understand. The issue for certtool that was reported was that it >> was blocking in /dev/random and taking a lot of time to produce any >> output. This was the reason I've put QUICK_RANDOM there. > Right, it is blocking because it needs to generate random numbers and do > do this we need to gather entropy from physical sources. If the > bandwidth of these sources is that small it just takes a long time. If > the box is idle it may even not finish at all. Recall that a computer > is a deterministic machine and that it is hard to extract unpredictable > events from a deterministic machines (actually impossible, but > fortunately a general purpose computer is not completely deterministic.) There are many parts in a typical PC that can feed a prng with non-deterministic data. Typical examples are the network card and sound card (mic etc), hard disks, memory accesses, interrupts, thermal sensors etc. Those provide lots of information that can feed a prng even on unattended system. There are two problems with the linux prng: 1. It needs to block when it thinks it does not have enough randomness 2. It does not use all available random data sources because its state could be compromised by a malicious or broken source. Fortuna [0] is a suitable PRNG replacement, because it has none of these issues. It can work even with some sources being malicious and will use a block cipher to produce large series of data without compromising or its state. [0]. http://en.wikipedia.org/wiki/Fortuna_(PRNG) > Ask the user to work on the box to give it a chnace to collect entroy. > For example "find /usr -type f | xargs cat >/dev/null" gets the disk to > work and thus floods the box with interrupts. The problem is that programs should be able to run both interactive and not. Moreover the blocking interface makes it's easy to prevent someone from creating a key... Just cat /dev/random, or open many tcp connections to a linux host. >> [0]. Also being blocking does not protect from being a bad algorithm. As >> far as I know there are known issues to the blocking linux rng (were >> discussed some years ago in gnutls-dev) and they still cannot gather any >> entropy from network devices because its state can be compromised! > And thus your solution is to give up on it and use a a deterministc > source like /dev/urandom? If /dev/random blocks, /dev/urandom will only > return a sequence of bytes which is predictable if you know the initial > state of the RNG. /dev/urandom is not deterministic it just has worse PR. /dev/random is the SAME as /dev/urandom with the exception that it blocks when it THINKS randomness gathered is not enough. If it thinks wrong (like when I control one source of randomness) it exactly the same as /dev/urandom. regards, Nikos From wk at gnupg.org Sat Dec 6 14:09:26 2008 From: wk at gnupg.org (Werner Koch) Date: Sat, 06 Dec 2008 14:09:26 +0100 Subject: [Help-gnutls] Alternate random device for certtool In-Reply-To: <493989CE.908@gnutls.org> (Nikos Mavrogiannopoulos's message of "Fri, 05 Dec 2008 22:06:38 +0200") References: <87vdu9qdtt.fsf@squeak.fifthhorseman.net> <87y6z5m2c8.fsf@squeak.fifthhorseman.net> <4930FB82.2090603@gnutls.org> <87skp4xjdl.fsf@wheatstone.g10code.de> <493826F1.2060206@gnutls.org> <87hc5jwt8h.fsf@wheatstone.g10code.de> <4938D5A6.90408@gnutls.org> <874p1jvy3f.fsf@wheatstone.g10code.de> <493989CE.908@gnutls.org> Message-ID: <87iqpxtpq1.fsf@wheatstone.g10code.de> On Fri, 5 Dec 2008 21:06, nmav at gnutls.org said: > There are many parts in a typical PC that can feed a prng with > non-deterministic data. Typical examples are the network card and sound Please read Peter's papers on this subject. In particular, network traffic does not yield any usable entropy. > 1. It needs to block when it thinks it does not have enough randomness Right, that is the correct behaviour. Actually I believe that current linux even estimates a too high entropy. > 2. It does not use all available random data sources because its state > could be compromised by a malicious or broken source. Currect behaviour. > Fortuna [0] is a suitable PRNG replacement, because it has none of these Well, as you say, this is a PRNG. It needs to be seeded. And the seed is the most problematic part. Almost all evaluations are handwaving the problem. The use of a continuously seeded PRNG is a pragmatical solution towards these problems. IIRC, NIST's special publication 800-90 suggest to re-seed a PRNG as ofthe as possible. FIPS 140-2 allows and suggest for re-seeded. For a real entropy source you need a *reliable* hardware entropy source. > Moreover the blocking interface makes it's easy to prevent someone from > creating a key... Just cat /dev/random, or open many tcp connections to > a linux host. So what? You are under attack and you still want to create a key on that attcked box? > /dev/urandom is not deterministic it just has worse PR. > /dev/random is the SAME as /dev/urandom with the exception that it > blocks when it THINKS randomness gathered is not enough. If it thinks That is simply not true. Read the 2006 paper by Gutterman, Pinkas and Reinman on the Linux RNG. Yes, I have a pretty conservative POV on entropy gathering. Shalom-Salam, Werner -- Die Gedanken sind frei. Auschnahme regelt ein Bundeschgesetz. From wk at gnupg.org Wed Dec 10 13:11:16 2008 From: wk at gnupg.org (Werner Koch) Date: Wed, 10 Dec 2008 13:11:16 +0100 Subject: DSA key generation using domain parameters In-Reply-To: <20081205131441.GA11159@thunk.cs.uwaterloo.ca> (Ian Goldberg's message of "Fri, 5 Dec 2008 08:14:41 -0500") References: <87vdu9qdtt.fsf@squeak.fifthhorseman.net> <87y6z5m2c8.fsf@squeak.fifthhorseman.net> <4930FB82.2090603@gnutls.org> <87skp4xjdl.fsf@wheatstone.g10code.de> <493826F1.2060206@gnutls.org> <87hc5jwt8h.fsf@wheatstone.g10code.de> <4938D5A6.90408@gnutls.org> <874p1jvy3f.fsf@wheatstone.g10code.de> <20081205131441.GA11159@thunk.cs.uwaterloo.ca> Message-ID: <87bpvkqlgb.fsf_-_@wheatstone.g10code.de> Hi Ian, I don't know whether this is useful for you: The latest Libgcrypt supports the specification of domain parameters to create DSA keys: `domain' This is only meaningful for DLP algorithms. If specified keys are generated with domain parameters taken from this list. The exact format of this parameter depends on the actual algorithm. It is currently only implemented for DSA using this format: (genkey (dsa (domain (p P-MPI) (q Q-MPI) (g Q-MPI)))) `nbits' and `qbits' may not be specified because they are derived from the domain parameters. Example: rc = gcry_sexp_new (&key_spec, "(genkey (dsa (transient-key)(domain" "(p #d3aed1876054db831d0c1348fbb1ada72507e5fbf9a62cbd47a63aeb7859d6921" "4adeb9146a6ec3f43520f0fd8e3125dd8bbc5d87405d1ac5f82073cd762a3f8d7" "74322657c9da88a7d2f0e1a9ceb84a39cb40876179e6a76e400498de4bb9379b0" "5f5feb7b91eb8fea97ee17a955a0a8a37587a272c4719d6feb6b54ba4ab69#)" "(q #9c916d121de9a03f71fb21bc2e1c0d116f065a4f#)" "(g #8157c5f68ca40b3ded11c353327ab9b8af3e186dd2e8dade98761a0996dda99ab" "0250d3409063ad99efae48b10c6ab2bba3ea9a67b12b911a372a2bba260176fad" "b4b93247d9712aad13aa70216c55da9858f7a298deb670a403eb1e7c91b847f1e" "ccfbd14bd806fd42cf45dbb69cd6d6b43add2a78f7d16928eaa04458dea44#)" ")))", 0, 1); if (rc) die ("error creating S-expression: %s\n", gcry_strerror (rc)); rc = gcry_pk_genkey (&key, key_spec); gcry_sexp_release (key_spec); if (rc) die ("error generating DSA key: %s\n", gcry_strerror (rc)); This should speed up key generation a lot because we don't need to search for primes. Salam-Shalom, Werner -- Die Gedanken sind frei. Auschnahme regelt ein Bundeschgesetz. From robert at roberthogan.net Mon Dec 29 22:07:16 2008 From: robert at roberthogan.net (Robert Hogan) Date: Mon, 29 Dec 2008 21:07:16 +0000 Subject: [patch] allow ctr mode to handle 'unaligned' plaintext blocks and improve ctr benchmarks Message-ID: <200812292107.21918.robert@roberthogan.net> Hi there, I needed to use AES in CTR mode but found that libgcrypt's current implementation does not allow for 'unaligned' blocks of plaintext, i.e. where the plaintext is not a multiple of the context's blocksize. I adapted an implementation found at c. line 289 of: https://svn.torproject.org/svn/tor/trunk/src/common/aes.c to add this functionality to libgcrypt and have supplied the patch below. The code there is licensed under 3-clause BSD which is GPL-compatible. I also ran it by Nick Mathewson of Tor (at #tor at irc.oftc.net) and he was fine with it too. (Though any errors remaining in the patch are very much my own.) I haven't tested the change to destruction but it does pass the existing unit tests and libgcrypt now decrypts correctly the unaligned plaintext blocks it was previously choking on in my application. The results from the unit tests I've added to basic.c are the output from patched do_ctr_encrypt() on my machine so shouldn't be taken as validating the new code in itself! I'm sure there are numerous other tests that could/should be added to validate the patch and I'm also sure it will be subject to careful review! Finally, you can see the patch results in a 33% performance improvement: Old do_ctr_encrypt(): $ ./benchmark --cipher-repetition 10 --verbose cipher AES Running each test 10 times. CTR --------------- AES 370ms 360ms Patched do_ctr_encrypt(): $ ./benchmark --cipher-repetition 10 --verbose cipher AES Running each test 10 times. CTR --------------- AES 240ms 240ms The Tor code also finds some optimization while incrementing the counter. I will test this out later and see if the gains are appreciable. Alternatively, you're welcome to try them out for yourself. The patch isn't very readable so here is the new do_ctr_encrypt() function by itself: static void do_ctr_encrypt( gcry_cipher_hd_t c, byte *outbuf, const byte *inbuf, unsigned int nbytes ) { unsigned int n; int i; n = c->ctrpos; while (1) { if (n == 0) c->cipher->encrypt (&c->context.c, c->ctrbuf, c->ctr); do { if (nbytes-- == 0) { c->ctrpos = n; return; } *(outbuf++) = *(inbuf++) ^ c->ctrbuf[n]; } while (++n != c->cipher->blocksize); c->ctrpos = n = 0; for (i = c->cipher->blocksize; i > 0; i--) { c->ctr[i-1]++; if (c->ctr[i-1] != 0) break; } if (nbytes == 0) { c->ctrpos = n; return; } } } Index: tests/basic.c =================================================================== --- tests/basic.c (revision 1375) +++ tests/basic.c (working copy) @@ -363,6 +363,16 @@ { "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17\xad\x2b\x41\x7b\xe6\x6c\x37\x10", 16, "\x1e\x03\x1d\xda\x2f\xbe\x03\xd1\x79\x21\x70\xa0\xf3\x00\x9c\xee" }, + { "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17\xad\x2b\x41\x7b\xe6\x6c\x37\x10" + "\x6c\x37\x10", + 19, + "\x46\x92\x63\xbd\xcb\xc5\x0a\x19\x5d\x43\x71\xec\x76\x27\x92\x12" + "\x34\xae\x54"}, + { "\xf1\x92\x24\x35\xaf\x4f\x9b\x17\xad\x2b\x41\x7b\xe6\x6c\x37\x10" + "\x6c\x37\x11", + 19, + "\xab\xdf\xc5\x34\x5a\x5c\x51\xc6\x35\x56\xc8\x92\xfd\x57\xee\xbc" + "\x15\x7e\xcf"}, } }, { GCRY_CIPHER_AES192, Index: cipher/cipher.c =================================================================== --- cipher/cipher.c (revision 1375) +++ cipher/cipher.c (working copy) @@ -205,7 +205,19 @@ unsigned char ctr[MAX_BLOCKSIZE]; /* For Counter (CTR) mode. */ + /*This is the context's current position in ctrbuf. Where the number + of bytes encrypted are not a multiple of c->cipher_block_size this + stores the position of the next byte in the keystream to be xor'd + by do_encrypt_ctr. ctr gets incremented every time ctrpos reaches + c->cipher_block_size. */ + unsigned int ctrpos; /* For Counter (CTR) mode. */ + /* This contains the context's current keystream. ctrpos stores the + next position to be xor'd in ctrbuf when do_ctr_encrypt is called. + The contents of ctr are encrypted every time ctrpos reaches + zero in do_ctr_encrypt, and the result is stored in ctrbuf. */ + byte ctrbuf[MAX_BLOCKSIZE]; /* For Counter (CTR) mode. */ + /* What follows are two contexts of the cipher in use. The first one needs to be aligned well enough for the cipher operation whereas the second one is a copy created by cipher_setkey and @@ -921,6 +933,8 @@ memset (c->u_iv.iv, 0, c->cipher->blocksize); memset (c->lastiv, 0, c->cipher->blocksize); memset (c->ctr, 0, c->cipher->blocksize); + memset (c->ctrbuf, 0, c->cipher->blocksize); + c->ctrpos=0; } @@ -1355,32 +1369,31 @@ } } - static void do_ctr_encrypt( gcry_cipher_hd_t c, byte *outbuf, const byte *inbuf, unsigned int nbytes ) { unsigned int n; - byte tmp[MAX_BLOCKSIZE]; int i; + n = c->ctrpos; - for(n=0; n < nbytes; n++) - { - if ((n % c->cipher->blocksize) == 0) - { - c->cipher->encrypt (&c->context.c, tmp, c->ctr); + while (1) { + if (n == 0) + c->cipher->encrypt (&c->context.c, c->ctrbuf, c->ctr); + do { + if (nbytes-- == 0) { c->ctrpos = n; return; } + *(outbuf++) = *(inbuf++) ^ c->ctrbuf[n]; + } while (++n != c->cipher->blocksize); + c->ctrpos = n = 0; + for (i = c->cipher->blocksize; i > 0; i--) + { + c->ctr[i-1]++; + if (c->ctr[i-1] != 0) + break; + } + if (nbytes == 0) { c->ctrpos = n; return; } + } - for (i = c->cipher->blocksize; i > 0; i--) - { - c->ctr[i-1]++; - if (c->ctr[i-1] != 0) - break; - } - } - - /* XOR input with encrypted counter and store in output. */ - outbuf[n] = inbuf[n] ^ tmp[n % c->cipher->blocksize]; - } } static void -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From wk at gnupg.org Tue Dec 30 14:42:31 2008 From: wk at gnupg.org (Werner Koch) Date: Tue, 30 Dec 2008 14:42:31 +0100 Subject: [patch] allow ctr mode to handle 'unaligned' plaintext blocks and improve ctr benchmarks In-Reply-To: <200812292107.21918.robert@roberthogan.net> (Robert Hogan's message of "Mon, 29 Dec 2008 21:07:16 +0000") References: <200812292107.21918.robert@roberthogan.net> Message-ID: <874p0lhj94.fsf@wheatstone.g10code.de> On Mon, 29 Dec 2008 22:07, robert at roberthogan.net said: > I needed to use AES in CTR mode but found that libgcrypt's current > implementation does not allow for 'unaligned' blocks of plaintext, i.e. > where the plaintext is not a multiple of the context's blocksize. That is clearly a bug and needs to be fixed. > to add this functionality to libgcrypt and have supplied the patch below. > The code there is licensed under 3-clause BSD which is GPL-compatible. I As per the GNU coding standards we would need to exchange legal papers with the orginal author and you to include this code - this would be a hassle for such a bug. Thus, I am going to implement it of my own, probably as the first task next year. See https://bugs.g10code.com/gnupg/issue983 . > The results from the unit tests I've added to basic.c are the output from > patched do_ctr_encrypt() on my machine so shouldn't be taken as validating I'll add this test to the regression test of course. Thanks. > The Tor code also finds some optimization while incrementing the counter. I > will test this out later and see if the gains are appreciable. Would it be helpful for you or TOR to have the code further optimized? We already have CFB and CBS optimizations for AES and adding CTR should not be a big problem. However, I can do further optimizations only after the release of 1.4.4. Salam-Shalom, Werner p.s. Now I need to prepare tomorrows shutdown of our TOR server allium.gnupg.org due to the German data retention laws :-((. Artikel 10 Grundgesetz, you served as well since May 23, 1949. Bye, bye for now and lets hope that the Federal Constitutional Court will decide soon. -- Die Gedanken sind frei. Auschnahme regelt ein Bundeschgesetz. From robert at roberthogan.net Tue Dec 30 19:38:10 2008 From: robert at roberthogan.net (Robert Hogan) Date: Tue, 30 Dec 2008 18:38:10 +0000 Subject: [patch] allow ctr mode to handle 'unaligned' plaintext blocks and improve ctr benchmarks In-Reply-To: <874p0lhj94.fsf@wheatstone.g10code.de> References: <200812292107.21918.robert@roberthogan.net> <874p0lhj94.fsf@wheatstone.g10code.de> Message-ID: <200812301838.15251.robert@roberthogan.net> On Tuesday 30 December 2008 13:42:31 Werner Koch wrote: > > to add this functionality to libgcrypt and have supplied the patch > > below. The code there is licensed under 3-clause BSD which is > > GPL-compatible. I > > As per the GNU coding standards we would need to exchange legal papers > with the orginal author and you to include this code - this would be a > hassle for such a bug. Thus, I am going to implement it of my own, > probably as the first task next year. See > https://bugs.g10code.com/gnupg/issue983 . > Sounds complicated.. > > The Tor code also finds some optimization while incrementing the > > counter. I will test this out later and see if the gains are > > appreciable. > > Would it be helpful for you or TOR to have the code further optimized? > We already have CFB and CBS optimizations for AES and adding CTR should > not be a big problem. However, I can do further optimizations only > after the release of 1.4.4. > Tor ships a number of AES-CTR implementations and tries to choose the optimal one at compile time. All of them are optimized to the nth degree because Tor spends a lot of time there. I'm not part of the Tor project but as far as I can see they've helped themselves in this regard. > > p.s. > Now I need to prepare tomorrows shutdown of our TOR server > allium.gnupg.org due to the German data retention laws :-((. > You might be interested to read: http://archives.seul.org/or/talk/Nov-2008/msg00262.html > Artikel 10 Grundgesetz, you served as well since May 23, 1949. Bye, bye > for now and lets hope that the Federal Constitutional Court will decide > soon. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: