From jas at extundo.com Thu Dec 8 21:01:27 2005 From: jas at extundo.com (Simon Josefsson) Date: Thu Dec 8 21:01:57 2005 Subject: [gnutls-dev] Experimental: GnuTLS 1.3.1 Message-ID: We are pleased to announce the availability of GnuTLS version 1.3.1, the second release on the experimental 1.3.x branch. The goal of 1.3.x will be to merge work currently done on CVS branches, for TLS Pre-Shared-Keys and TLS Inner Application. Other planned improvements in 1.3.x are system-independent resume data structures, modularization of the bignum operations, and TLS OpenPGP improvements. So far, TLS-PSK and system-independent resume data has been implemented. GnuTLS is a modern C library that implement the standard network security protocol Transport Layer Security (TLS), for use by network applications. Improving GnuTLS is costly, but you can help! We are looking for organizations that find GnuTLS useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for GnuTLS are available, and they help finance continued maintenance. Simon Josefsson Datakonsult, a Stockholm based privately held company, is currently funding GnuTLS maintenance. We are always looking for interesting development projects. If you need help to use GnuTLS, or want to help others, you are invited to join our help-gnutls mailing list, see: . The project page of the library is available at: http://www.gnutls.org/ http://www.gnu.org/software/gnutls/ http://josefsson.org/gnutls/ (updated fastest) Here are the compressed sources: http://josefsson.org/gnutls/releases/gnutls-1.3.1.tar.gz (3.0MB) ftp://ftp.gnutls.org/pub/gnutls/gnutls-1.3.1.tar.bz2 (3.0MB) Here are GPG detached signatures signed using key 0xB565716F: http://josefsson.org/gnutls/releases/gnutls-1.3.1.tar.bz2.sig ftp://ftp.gnutls.org/pub/gnutls/gnutls-1.3.1.tar.bz2.sig The software is cryptographically signed by the author using an OpenPGP key identified by the following information: 1280R/B565716F 2002-05-05 [expires: 2006-02-28] Key fingerprint = 0424 D4EE 81A0 E3D1 19C6 F835 EDA2 1E94 B565 716F The key is available from: http://josefsson.org/key.txt dns:b565716f.josefsson.org?TYPE=CERT Here are the build reports for various platforms: http://josefsson.org/autobuild-logs/gnutls.html Here are the SHA-1 checksums: 80eb527cf981344778d0dd6cb2ed25f379d8785c gnutls-1.3.1.tar.bz2 5b260e5d3594a8cf8ea79376bd97775a5f566920 gnutls-1.3.1.tar.bz2.sig Enjoy, Nikos and Simon Noteworthy changes since version 1.3.0: ** Support for DHE-PSK cipher suites has been added. This method offers perfect forward secrecy. ** Fix gnutls-cli STARTTLS hang when SIGINT is sent too quickly, thanks to Otto Maddox and Nozomu Ando . ** Corrected a bug in certtool for 64 bit machines. Reported by Max Kellermann . ** New function to set a X.509 private key and certificate pairs, and/or CRLs, from an PKCS#12 file, suggested by Emile van Bergen . The integrity of the PKCS#12 file is protected through a password based MAC; public-key based signatures for integrity protection are not supported. PKCS#12 bags may be encrypted using password derived symmetric keys, public-key based encryption is not supported. The PKCS#8 keys may be encrypted using passwords. The API use the same password for all operations. We believe that any more flexibility create too much complexity that would hurt overall security, but may add more PKCS#12 related APIs if real-world experience indicate otherwise. ** gnutls_x509_privkey_import_pkcs8 now accept unencrypted PEM PKCS#8 keys, reported by Emile van Bergen . This will enable "certtool -k -8" to parse those keys. ** Certtool now generate keys in unencrypted PKCS#8 format for empty passwords. Use "certtool -p -8" and press press enter at the prompt. Earlier, certtool would have encrypted the key using an empty password. ** Certtool now accept --password for --key-info and encrypted PKCS#8 keys. Earlier it would have prompted the user for it, even if --password was supplied. ** Added self test of PKCS#8 parsing. Unencrypted and encrypted (pbeWithSHAAnd3-KeyTripleDES-CBC and pbeWithSHAAnd40BitRC2-CBC) formats are tested. The test is in tests/pkcs8. ** API and ABI modifications: New function to set X.509 credentials from a PKCS#12 file: gnutls_certificate_set_x509_simple_pkcs12_file New gnutls_kx_algorithm_t enum type: GNUTLS_KX_DHE_PSK New API to return session data (better data types than gnutls_session_get_data): gnutls_session_get_data2 New API to set PSK Diffie-Hellman parameters: gnutls_psk_set_server_dh_params -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 423 bytes Desc: not available Url : /pipermail/attachments/20051208/8d0f2b3c/attachment.pgp From jas at extundo.com Wed Dec 14 14:37:02 2005 From: jas at extundo.com (Simon Josefsson) Date: Wed Dec 14 14:37:07 2005 Subject: [gnutls-dev] GnuTLS 1.2.8 with TLS Inner Application (TLS/IA) support Message-ID: We are pleased to present a customized version of GnuTLS 1.2.8 that adds an implementation of the TLS Inner Application (TLS/IA) protocol. The TLS/IA protocol was designed to be used in the EAP-TTLSv1 protocol, to perform user authentication of Wireless LAN network nodes using IEEE 802.1x. The TLS/IA and TTLSv1 protocols were published through the IETF and descriptions can be found at: http://josefsson.org/tlsia/draft-funk-tls-inner-application-extension-01.txt http://josefsson.org/tlsia/draft-ietf-pppext-eap-ttls-05.txt The goal is to merge this TLS/IA branch with the main development branch (1.3.x) and then to investigate how EAP-TTLSv1 can be implemented. We invite suggestions and comments on these matters. This work was done by Simon Josefsson Datakonsult in close co-operation with Emile van Bergen of E-advies, under commission for Lumiad. Lumiad is a Dutch based privately held company. Lumiad is specialized in wireless applications and wireless security solutions. Lumiad supports open source projects, from which large parts will be used in Lumiad products. Lumiad was happy to sponsor this specific TLS/IA module. We see this module as a first step for the correct implementation of the EAP-TTLSV1 standard in open source products. http://www.lumiad.nl/ E-advies is a privately held company based in the Netherlands that designs and develops software and solutions, and provides consultancy in telecommunications and storage. Its flagship product is OpenRADIUS, an industrial strength RADIUS server that offers complete freedom in policy definition, and is available under the GNU General Public License. Simon Josefsson Datakonsult, a Stockholm based privately held company that specialize in development and standardization of security and internationalization technologies, is currently funding GnuTLS maintenance. Commercial support contracts for GnuTLS are available, and they help finance continued maintenance. GnuTLS is a modern C library that implement the standard network security protocol Transport Layer Security (TLS), for use by network applications. The NEWS entries for this release are: - GnuTLS now support TLS Inner application (TLS/IA) as per draft-funk-tls-inner-application-extension-01. This functionality is added to libgnutls-extra, so it is licensed under the GPL. - API and ABI modifications: gnutls_ia_handshake: New function, to perform TLS/IA handshake. gnutls_ia_handshake_p: New function, a predicate to decide whether to TLS/IA handshake. gnutls_ia_free_client_credentials, gnutls_ia_allocate_client_credentials, gnutls_ia_free_server_credentials, gnutls_ia_allocate_server_credentials: New functions to allocate a TLS/IA credential. gnutls_ia_set_client_avp_function, gnutls_ia_set_server_avp_function, gnutls_ia_set_client_avp_ptr, gnutls_ia_get_client_avp_ptr, gnutls_ia_set_server_avp_ptr, gnutls_ia_get_server_avp_ptr: New functions to handle the AVP callback. gnutls_ia_require_inner_phase: New functions, to toggle TLS/IA application phases. gnutls_ia_permute_inner_secret: New function to mix session keys with inner secret. gnutls_ia_endphase_send, gnutls_ia_send, gnutls_ia_recv: Low-level API. gnutls_ia_generate_challenge, gnutls_ia_extract_inner_secret: New functions that can be used after successful TLS/IA negotiation. gnutls_ia_mode_t: Enum type with TLS/IA modes. gnutls_ia_apptype_t: Enum type with TLS/IA packet types. GNUTLS_A_INNER_APPLICATION_FAILURE, GNUTLS_A_INNER_APPLICATION_VERIFICATION: Enum values for TLS/IA alerts. GNUTLS_E_WARNING_IA_IPHF_RECEIVED, GNUTLS_E_WARNING_IA_FPHF_RECEIVED: New error codes, to signal when an application phase has finished. GNUTLS_E_IA_VERIFY_FAILED: New error code to signal TLS/IA verify failure. If you need help to use GnuTLS, or want to help others, you are invited to join our help-gnutls mailing list, see: . The project page of the library is available at: http://josefsson.org/gnutls/ Here are the compressed sources: http://josefsson.org/gnutls/releases/tlsia/gnutls-1.2.8+ia.1.tar.bz2 (2.5MB) Here are GPG detached signatures signed using key 0xB565716F: http://josefsson.org/gnutls/releases/tlsia/gnutls-1.2.8+ia.1.tar.bz2.sig The software is cryptographically signed by the author using an OpenPGP key identified by the following information: 1280R/B565716F 2002-05-05 [expires: 2006-02-28] Key fingerprint = 0424 D4EE 81A0 E3D1 19C6 F835 EDA2 1E94 B565 716F The key is available from: http://josefsson.org/key.txt dns:b565716f.josefsson.org?TYPE=CERT Here are the SHA-1 checksums: 4296d3bcdd32f11df9b3ea16f1811f4bc6569fd9 gnutls-1.2.8+ia.1.tar.bz2 da6445dfb716adbbcb696a205f0361c0add2a9e1 gnutls-1.2.8+ia.1.tar.bz2.sig Enjoy, Simon From jas at extundo.com Thu Dec 15 10:01:27 2005 From: jas at extundo.com (Simon Josefsson) Date: Thu Dec 15 10:01:49 2005 Subject: [gnutls-dev] Re: GnuTLS 1.2.8 with TLS Inner Application (TLS/IA) support In-Reply-To: <20051215041945.GA9138@jm.kir.nu> (Jouni Malinen's message of "Wed, 14 Dec 2005 20:19:46 -0800") References: <20051215041945.GA9138@jm.kir.nu> Message-ID: Jouni Malinen writes: > I haven't yet looked into details of the TLS/IA changes. However, I have > one thing in my mind from the earlier experience of adding GnuTLS > support. EAP-TLS/TTLS/PEAP/FAST require access to couple of values that > are internal to the TLS handshake, namely, master_secret, client_random, > and server_random. However, I have not found a supported, public API > call for getting these from GnuTLS. These are stored in > security_parameters_st, but that seems to be only used internally > (defined in gnutls_int.h). > > My current workaround is to make a copy of this structure and fetch the > parameters with direct read through gnutls_session::security_parameters. > Another option would be to parse the values from debug messages (using > log function callback). Neither option seems very nice since they > are relying on the internal structure not changing and/or debug messages > remaining in the same format. I'm actually keeping a list of tested > GnuTLS versions and only allow tested versions to be used because of > this workaround. Adding a public API call or structure definition for > getting these values would be quite useful for EAP implementations. Right. I noticed this in your tls_gnutls.c, and it seemed ugly. Of course, it was the only solution for you. I am working on adding two APIs, one to access the TLS PRF and one to retrieve the master secret, the client/server-random and potentially other useful data as well. This will be part of the next GnuTLS 1.3.x release (which will also include TLS/IA), hopefully due before the end of this year. Hopefully this will remove the need for your workaround. Thanks, Simon From jas at extundo.com Thu Dec 15 12:54:28 2005 From: jas at extundo.com (Simon Josefsson) Date: Thu Dec 15 12:54:23 2005 Subject: [gnutls-dev] Re: GnuTLS 1.2.8 with TLS Inner Application (TLS/IA) support In-Reply-To: (Simon Josefsson's message of "Thu, 15 Dec 2005 10:01:27 +0100") References: <20051215041945.GA9138@jm.kir.nu> Message-ID: I have added the API below which should remove the need for your hack. Let me know if you have any thoughts on the API, or can think of any improvements. This will be part of 1.3.2 which will likely be released later today. Thanks, Simon ** New APIs to access the TLS Pseudo-Random-Function (PRF). The PRF is used by some protocols building on TLS, such as EAP-PEAP and EAP-TTLS. One function to access the raw PRF and one to access the PRF seeded with the client/server random fields are provided. Suggested by Jouni Malinen . ** New APIs to acceess the client and server random fields in a session. These fields can be useful by protocols using TLS. Note that these fields are typically used as input to the TLS PRF, and if this is your intended use, you should use the TLS PRF API that use the client/server random field directly. Suggested by Jouni Malinen . New functions to invoke the TLS Pseudo-Random-Function (PRF): gnutls_prf gnutls_prf_raw New functions to retrieve the session's client and server random values: gnutls_session_get_server_random gnutls_session_get_client_random /** * gnutls_prf_raw - access the TLS PRF directly * @session: is a #gnutls_session_t structure. * @label_size: length of the @label variable. * @label: label used in PRF computation, typically a short string. * @seed_size: length of the @seed variable. * @seed: optional extra data to seed the PRF with. * @outsize: size of pre-allocated output buffer to hold the output. * @out: pre-allocate buffer to hold the generated data. * * Apply the TLS Pseudo-Random-Function (PRF) using the master secret * on some data. * * The @label variable usually contain a string denoting the purpose * for the generated data. The @seed usually contain data such as the * client and server random, perhaps together with some additional * data that is added to guarantee uniqueness of the output for a * particular purpose. * * Because the output is not guaranteed to be unique for a particular * session unless @seed include the client random and server random * fields (the PRF would output the same data on another connection * resumed from the first one), it is not recommended to use this * function directly. The gnutls_prf() function seed the PRF with the * client and server random fields directly, and is recommended if you * want to generate pseudo random data unique for each session. * * Return value: Return 0 on success, or an error code. **/ int gnutls_prf_raw (gnutls_session_t session, size_t label_size, const char *label, size_t seed_size, const char *seed, size_t outsize, char *out); /** * gnutls_prf - derive pseudo-random data using the TLS PRF * @session: is a #gnutls_session_t structure. * @label_size: length of the @label variable. * @label: label used in PRF computation, typically a short string. * @server_random_first: non-0 if server random field should be first in seed * @extra_size: length of the @extra variable. * @extra: optional extra data to seed the PRF with. * @outsize: size of pre-allocated output buffer to hold the output. * @out: pre-allocate buffer to hold the generated data. * * Apply the TLS Pseudo-Random-Function (PRF) using the master secret * on some data, seeded with the client and server random fields. * * The @label variable usually contain a string denoting the purpose * for the generated data. The @server_random_first indicate whether * the client random field or the server random field should be first * in the seed. Non-0 indicate that the server random field is first, * 0 that the client random field is first. * * The @extra variable can be used to add more data to the seed, after * the random variables. It can be used to tie make sure the * generated output is strongly connected to some additional data * (e.g., a string used in user authentication). * * The output is placed in *@OUT, which must be pre-allocated. * * Return value: Return 0 on success, or an error code. **/ int gnutls_prf (gnutls_session_t session, size_t label_size, const char *label, int server_random_first, size_t extra_size, const char *extra, size_t outsize, char *out); /** * gnutls_session_get_client_random - get the session's client random value * @session: is a #gnutls_session_t structure. * * Return a pointer to the 32-byte client random field used in the * session. The pointer must not be modified or deallocated. * * If a client random value has not yet been established, the output * will be garbage, and in particular a %NULL return value should not * be expected. * * Return value: pointer to client random. **/ const char * gnutls_session_get_client_random (gnutls_session_t session); /** * gnutls_session_get_server_random - get the session's server random value * @session: is a #gnutls_session_t structure. * * Return a pointer to the 32-byte server random field used in the * session. The pointer must not be modified or deallocated. * * If a server random value has not yet been established, the output * will be garbage, and in particular a %NULL return value should not * be expected. * * Return value: pointer to server random. **/ const char * gnutls_session_get_server_random (gnutls_session_t session); From jas at extundo.com Thu Dec 15 14:18:37 2005 From: jas at extundo.com (Simon Josefsson) Date: Thu Dec 15 14:18:35 2005 Subject: [gnutls-dev] Experimental: GnuTLS 1.3.2 Message-ID: We are pleased to announce the availability of GnuTLS version 1.3.2, another release on the experimental 1.3.x branch. The goal of 1.3.x will be to merge work currently done on CVS branches, for TLS Pre-Shared-Keys and TLS Inner Application (TLS/IA). Other planned improvements in 1.3.x are system-independent resume data structures, modularization of the bignum operations, and TLS OpenPGP improvements. With this release, the TLS-PSK, TLS/IA and system-independent resume data goals have been met. Currently, http://www.gnutls.org/ and ftp://ftp.gnutls.org/ appear to be down. The web pages on http://www.gnu.org/software/gnutls/ are no longer updated automatically, presumably due to Savannah's recent CVS changes. This means http://josefsson.org/gnutls/ is the only distribution point right now. I'm considering using ftp.gnu.org as the canonical distribution point in the future. GnuTLS is a modern C library that implement the standard network security protocol Transport Layer Security (TLS), for use by network applications. Improving GnuTLS is costly, but you can help! We are looking for organizations that find GnuTLS useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for GnuTLS are available, and they help finance continued maintenance. Simon Josefsson Datakonsult, a Stockholm based privately held company, is currently funding GnuTLS maintenance. We are always looking for interesting development projects. If you need help to use GnuTLS, or want to help others, you are invited to join our help-gnutls mailing list, see: . The project page of the library is available at: http://josefsson.org/gnutls/ (updated fastest) http://www.gnu.org/software/gnutls/ Here are the compressed sources: http://josefsson.org/gnutls/releases/gnutls-1.3.2.tar.gz (3.1MB) Here are GPG detached signatures signed using key 0xB565716F: http://josefsson.org/gnutls/releases/gnutls-1.3.2.tar.bz2.sig The software is cryptographically signed by the author using an OpenPGP key identified by the following information: 1280R/B565716F 2002-05-05 [expires: 2006-02-28] Key fingerprint = 0424 D4EE 81A0 E3D1 19C6 F835 EDA2 1E94 B565 716F The key is available from: http://josefsson.org/key.txt dns:b565716f.josefsson.org?TYPE=CERT Here are the build reports for various platforms: http://josefsson.org/autobuild-logs/gnutls.html Here are the SHA-1 checksums: f0bc87bb29591b710d63699896cb26f539a47e6b gnutls-1.3.2.tar.bz2 2f7293627dd39968d7341b793c4bf252ac366379 gnutls-1.3.2.tar.bz2.sig Enjoy, Nikos and Simon Noteworthy changes since version 1.3.1: ** GnuTLS now support TLS Inner application (TLS/IA). This is per draft-funk-tls-inner-application-extension-01. This functionality is added to libgnutls-extra, so it is licensed under the GNU General Public License. ** New APIs to access the TLS Pseudo-Random-Function (PRF). The PRF is used by some protocols building on TLS, such as EAP-PEAP and EAP-TTLS. One function to access the raw PRF and one to access the PRF seeded with the client/server random fields are provided. Suggested by Jouni Malinen . ** New APIs to acceess the client and server random fields in a session. These fields can be useful by protocols using TLS. Note that these fields are typically used as input to the TLS PRF, and if this is your intended use, you should use the TLS PRF API that use the client/server random field directly. Suggested by Jouni Malinen . ** Internal type cleanups. The uint8, uint16, uint32 types have been replaced by uint8_t, uint16_t, uint32_t. Gnulib is used to guarantee the presence of correct types on platforms that lack them. The uint type have been replaced by unsigned. ** API and ABI modifications: New functions to invoke the TLS Pseudo-Random-Function (PRF): gnutls_prf gnutls_prf_raw New functions to retrieve the session's client and server random values: gnutls_session_get_server_random gnutls_session_get_client_random New function, to perform TLS/IA handshake: gnutls_ia_handshake New function to decide whether to do a TLS/IA handshake: gnutls_ia_handshake_p New functions to allocate a TLS/IA credential: gnutls_ia_allocate_client_credentials gnutls_ia_free_client_credentials gnutls_ia_allocate_server_credentials gnutls_ia_free_server_credentials New functions to handle the AVP callback: gnutls_ia_set_client_avp_function gnutls_ia_set_client_avp_ptr gnutls_ia_get_client_avp_ptr gnutls_ia_set_server_avp_function gnutls_ia_set_server_avp_ptr gnutls_ia_get_server_avp_ptr New functions, to toggle TLS/IA application phases: gnutls_ia_require_inner_phase New function to mix session keys with inner secret: gnutls_ia_permute_inner_secret Low-level API (used internally by gnutls_ia_handshake): gnutls_ia_endphase_send gnutls_ia_send gnutls_ia_recv New functions that can be used after successful TLS/IA negotiation: gnutls_ia_generate_challenge gnutls_ia_extract_inner_secret Enum type with TLS/IA modes: gnutls_ia_mode_t Enum type with TLS/IA packet types: gnutls_ia_apptype_t Enum values for TLS/IA alerts: GNUTLS_A_INNER_APPLICATION_FAILURE GNUTLS_A_INNER_APPLICATION_VERIFICATION New error codes, to signal when an application phase has finished: GNUTLS_E_WARNING_IA_IPHF_RECEIVED GNUTLS_E_WARNING_IA_FPHF_RECEIVED New error code to signal TLS/IA verify failure: GNUTLS_E_IA_VERIFY_FAILED -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 423 bytes Desc: not available Url : /pipermail/attachments/20051215/bc992d36/attachment.pgp From jas at extundo.com Fri Dec 16 12:24:24 2005 From: jas at extundo.com (Simon Josefsson) Date: Fri Dec 16 12:24:12 2005 Subject: [gnutls-dev] Re: GnuTLS 1.2.8 with TLS Inner Application (TLS/IA) support In-Reply-To: <20051216040120.GA8943@jm.kir.nu> (Jouni Malinen's message of "Thu, 15 Dec 2005 20:01:21 -0800") References: <20051215041945.GA9138@jm.kir.nu> <20051216040120.GA8943@jm.kir.nu> Message-ID: Jouni Malinen writes: > On Thu, Dec 15, 2005 at 12:54:28PM +0100, Simon Josefsson wrote: > >> I have added the API below which should remove the need for your hack. >> Let me know if you have any thoughts on the API, or can think of any >> improvements. >> >> This will be part of 1.3.2 which will likely be released later today. > > Thanks! This was indeed enough to get rid of the need for including > internal structure definitions. Using gnutls_prf() means that the master > secret is not actually exported from the library, so I added a new > wrapper function for doing this. Somewhat similar functionality was > already needed for Schannel, so this was a good opportunity to clean up > the interface by making it more generic. I have added gnutls_session_get_master_secret() for completeness, it might be useful in some older protocols. It will be in 1.3.3. > gnutls_session_get_server_random() and > gnutls_session_get_client_random() actually not needed for > EAP-{TLS,PEAP,TTLS} since gnutls_prf() takes care of that part, too. Right, that was my hope. > However, these functions may end up being quite useful if EAP-FAST > support is added at some point. That would require support for > SessionTicket TLS extension (draft-salowey-tls-ticket-05.txt) and a > callback for setting master secret during ServerHello processing so that > full certificate-based authentication can be skipped. GnuTLS seems to > have some support for TLS extensions, but I haven't yet looked into > details of whether this particular use (mainly the callback) could be > implemented now or whether some changes would be needed in the library. That document look rather useful, even for non-EAP stuff. It seem possible to implement it in GnuTLS. I have added it to the TODO list... Regards, Simon From bryanh at giraffe-data.com Wed Dec 21 19:04:01 2005 From: bryanh at giraffe-data.com (Bryan Henderson) Date: Wed Dec 21 19:04:03 2005 Subject: [gnutls-dev] living without global variables Message-ID: It would be nice for GnuTLS not to use program-global variables. That way, it could be used in a modular program -- i.e. two different parts of a program that don't know about each other could use GnuTLS under the covers without interfering with each other. I'm especially concerned about independently distributed libraries that use Gnu TLS. GnuTLS contains two global variables of interest: the handles of the two ASN1 trees: _gnutls_pkix1_asn and _gnutls_gnutls_asn. (I'm not concerned about debugging features or alternative memory allocators). To work without these, I propose having a new session group object that contains the two trees. A program creates that object instead of calling gnutls_global_init() and passes that object as a parameter when creating a session (with a new variation of gnutls_init()). All functions that access those global variables get a session group object as a parameter. gnutls_global_init() would still be around for programs that want the simplicity of global variables and don't care about modularity. gnutls_global_init() would create a session group object and assign its handle to a global variable. gnutls_init() uses the global session group handle. Would this be acceptable? I'm willing to write whatever code and documentation is necessary. -- Bryan Henderson Phone 408-621-2000 San Jose, California From nmav at gnutls.org Wed Dec 21 22:23:53 2005 From: nmav at gnutls.org (Nikos Mavrogiannopoulos) Date: Wed Dec 21 22:23:59 2005 Subject: [gnutls-dev] living without global variables In-Reply-To: <200512211805.jBLI5GeK032128@yxa.extundo.com> References: <200512211805.jBLI5GeK032128@yxa.extundo.com> Message-ID: <200512212223.53584.nmav@gnutls.org> On Wednesday 21 December 2005 19:05, Bryan Henderson wrote: > It would be nice for GnuTLS not to use program-global variables. > That way, it could be used in a modular program -- i.e. two different > parts of a program that don't know about each other could use GnuTLS > under the covers without interfering with each other. I'm especially > concerned about independently distributed libraries that use Gnu TLS. > GnuTLS contains two global variables of interest: the handles of the > two ASN1 trees: _gnutls_pkix1_asn and _gnutls_gnutls_asn. (I'm not > concerned about debugging features or alternative memory allocators). These variables are constant during the lifetime of the application. What would such a change offer as an advantage? -- Nikos Mavrogiannopoulos From bryanh at giraffe-data.com Thu Dec 22 18:22:38 2005 From: bryanh at giraffe-data.com (Bryan Henderson) Date: Thu Dec 22 18:22:40 2005 Subject: [gnutls-dev] living without global variables In-Reply-To: <200512212223.53584.nmav@gnutls.org> References: <200512211805.jBLI5GeK032128@yxa.extundo.com> <200512212223.53584.nmav@gnutls.org> Message-ID: >These variables are constant during the lifetime of the application. I didn't really mean to refer to those two root pointers as the global variables -- the entire tree each points to is the global variable. These trees are specifically designed (according to what I see in the user's guide) to make separate sessions share stuff as it changes. That's great for sessions that all belong to the same context, but modules shouldn't be sharing with other modules inadvertently. The most serious way that these "variables" change during the lifetime of the application is when a piece (module) of the application does a gnutls_global_deinit(). That would blow up some other module in the same application that happens to be using gnutls. In case it isn't clear what sort of "module" I'm talking about: I (and plenty of others, I'm sure) want to distribute a programming library that uses gnutls to provide some of its function. It should be possible for my user to be blissfully ignorant of the gnutls stuff going on inside the library, which means he doesn't worry about initializing and deinitializing the gnutls library. His program may use gnutls for its own purposes, and may use some other library that uses gnutls under the covers like mine. It's common in modern libraries to avoid process-global variables completely as a matter of good programming style (it helps not only with modularity but thread safety and recursion as well). -- Bryan Henderson Phone 408-621-2000 San Jose, California From jas at extundo.com Fri Dec 23 14:47:27 2005 From: jas at extundo.com (Simon Josefsson) Date: Fri Dec 23 14:47:16 2005 Subject: [gnutls-dev] Re: living without global variables In-Reply-To: <41414.1438821845$1135272425@news.gmane.org> (Bryan Henderson's message of "Thu, 22 Dec 2005 17:27:05 +0000 (UTC)") References: <200512211805.jBLI5GeK032128@yxa.extundo.com> <200512212223.53584.nmav@gnutls.org> <41414.1438821845$1135272425@news.gmane.org> Message-ID: bryanh@giraffe-data.com (Bryan Henderson) writes: >>These variables are constant during the lifetime of the application. > > I didn't really mean to refer to those two root pointers as the global > variables -- the entire tree each points to is the global variable. The entire tree is constant as well. > The most serious way that these "variables" change during the lifetime > of the application is when a piece (module) of the application does a > gnutls_global_deinit(). That would blow up some other module in the > same application that happens to be using gnutls. Yes. gnutls_global_deinit() should not be called if you want to continue to use the library. No other function modify that variable. > In case it isn't clear what sort of "module" I'm talking about: I (and > plenty of others, I'm sure) want to distribute a programming library > that uses gnutls to provide some of its function. It should be > possible for my user to be blissfully ignorant of the gnutls stuff > going on inside the library, which means he doesn't worry about > initializing and deinitializing the gnutls library. His program may use > gnutls for its own purposes, and may use some other library that uses > gnutls under the covers like mine. > > It's common in modern libraries to avoid process-global variables > completely as a matter of good programming style (it helps not only > with modularity but thread safety and recursion as well). I think it would be useful to see exactly what changes you are thinking of in more detail. Remember that API changes are bad. I'm not sure exactly what real problem you are having. Without understanding the problem more, it is difficult to justify API changes (which if I understood you correctly would be necessary). Thanks, Simon From nmav at gnutls.org Fri Dec 23 20:43:33 2005 From: nmav at gnutls.org (Nikos Mavrogiannopoulos) Date: Fri Dec 23 20:43:37 2005 Subject: [gnutls-dev] Re: living without global variables In-Reply-To: References: <200512211805.jBLI5GeK032128@yxa.extundo.com> <41414.1438821845$1135272425@news.gmane.org> Message-ID: <200512232043.33524.nmav@gnutls.org> On Friday 23 December 2005 14:47, Simon Josefsson wrote: > > The most serious way that these "variables" change during the > > lifetime of the application is when a piece (module) of the > > application does a gnutls_global_deinit(). That would blow up some > > other module in the same application that happens to be using > > gnutls. > Yes. gnutls_global_deinit() should not be called if you want to > continue to use the library. No other function modify that variable. Note also that gnutls_global_init() and deinit() count the number of times called. So for these variables to be freed the counter must be zero. Thus most of these kind of problems are minimized. -- Nikos Mavrogiannopoulos From nmav at gnutls.org Fri Dec 23 20:54:33 2005 From: nmav at gnutls.org (Nikos Mavrogiannopoulos) Date: Fri Dec 23 20:54:36 2005 Subject: [gnutls-dev] Re: living without global variables In-Reply-To: <200512232043.33524.nmav@gnutls.org> References: <200512211805.jBLI5GeK032128@yxa.extundo.com> <200512232043.33524.nmav@gnutls.org> Message-ID: <200512232054.33633.nmav@gnutls.org> On Friday 23 December 2005 20:43, Nikos Mavrogiannopoulos wrote: > > Yes. gnutls_global_deinit() should not be called if you want to > > continue to use the library. No other function modify that > > variable. > Note also that gnutls_global_init() and deinit() count the number of > times called. So for these variables to be freed the counter must be > zero. Thus most of these kind of problems are minimized. I just found out that this wasn't the case. It seems I was sure that this was the behaviour but I was wrong. Thanks to some changes of Simon, this should work now. -- Nikos Mavrogiannopoulos From bryanh at giraffe-data.com Sat Dec 24 10:44:53 2005 From: bryanh at giraffe-data.com (Bryan Henderson) Date: Sat Dec 24 10:44:57 2005 Subject: [gnutls-dev] Re: living without global variables In-Reply-To: (jas@extundo.com) References: <200512211805.jBLI5GeK032128@yxa.extundo.com> <200512212223.53584.nmav@gnutls.org> <41414.1438821845$1135272425@news.gmane.org> Message-ID: >The entire tree is constant as well. That's great. I really couldn't tell just what information is in there. What is it, anyway? Something you can explain quickly? The information from Nikos that reference counting will be implemented solves another part of the problem (but I'm a little skeptical about that because the old code seemed to be pretty deliberate in not doing reference counting; it actually looks like there was reference counting before and someone undid it for some reason). But there's still more work to do. It isn't thread-safe. Often, a program is modular in that two threads are executing separate code independently. If both of those threads decide to use the facilities of the gnutls library, they might simultaneously call gnutls_global_init() and the reference count would get screwed up. On the other hand, in what I propose each thread has its own memory and there can't possibly be a conflict. (The synchronization point moves to malloc(), which is a reliable place for it). >Remember that API changes are bad. But API _additions_ are no big deal. I'm talking about adding three new functions for people to use optionally in place of three existing ones, and people who use the existing ones will see no difference at all. >I think it would be useful to see exactly what changes you are >thinking of in more detail. I don't want to go so far as coding a patch if you don't buy into the basic idea, because it would be too much work. But let me try a more detailed explanation. This is nothing revolutionary -- it's how lots of libraries work in order to cater to threaded, modular, object oriented programs. A program calls gnutls_create_session_group() instead of gnutls_global_init(). This new function returns a "session group" handle. The program then calls gnutls_session_init_group() instead of gnutls_session_init(). This new function takes that session group handle as an argument. When it's all done, the program calls gnutls_destroy_session_group(handle) instead of gnutls_global_deinit(). The session handle is really a pointer to a descriptor (struct) that contains root pointers for the "global" trees. So each session group has its own copy of the trees. The session group handle gets added to the session descriptor, and everything in gnutls that references the trees references them via the relevant session group descriptor instead of direct reference to global variables. Some internal functions need to get an extra parameter so they know which tree to access. So that the existing API keeps working, there is a global session group descriptor that takes the place of the two global tree pointers. gnutls_global_init() does a gnutls_create_session_group() of that descriptor and gnutls_session_init() does a gnutls_session_init_group() with the global handle. >I'm not sure exactly what real problem you are having. I thought the basic global variable issue was widely enough known that there wouldn't be a need to describe a particular scenario, but I'll tell you where I'm coming from. I don't have a program that crashed. I distribute a library (XML-RPC for C/C++) that uses gnutls. (Actually, it's worse. It uses Curl, and Curl uses gnutls). My users want to write object oriented programs with multiple threads. I know that because Curl has a global init function, and one reason it has that is because gnutls does, I can't give my users that because my code has to initialize gnutls and I have no way to keep my gnutls initialization from conflicting with gnutls initializations that the user does for other purposes (it's the simultaneous reference count update problem I described above). The best I can do is expose the deep underlying use of gnutls (no easy thing, since gnutls is only one of Curl's SSH options and Curl is only one of Xmlrpc-c's HTTP options) and tell the user he has to supply an already initialized gnutls environment to me. But with the session group concept, it's a snap. I just give each Xmlrpc-c client object its own Curl client object and each Curl client object its own gnutls session group, and I can't possible interfere with the user's other uses of gnutls. The only thing I'm sharing with them is the code of gnutls. -- Bryan Henderson Phone 408-621-2000 San Jose, California From jas at extundo.com Sun Dec 25 15:13:15 2005 From: jas at extundo.com (Simon Josefsson) Date: Sun Dec 25 15:13:24 2005 Subject: [gnutls-dev] Re: living without global variables In-Reply-To: <43404.6885501177$1135417620@news.gmane.org> (Bryan Henderson's message of "Sat, 24 Dec 2005 09:47:00 +0000 (UTC)") References: <200512211805.jBLI5GeK032128@yxa.extundo.com> <200512212223.53584.nmav@gnutls.org> <41414.1438821845$1135272425@news.gmane.org> <43404.6885501177$1135417620@news.gmane.org> Message-ID: bryanh@giraffe-data.com (Bryan Henderson) writes: >>The entire tree is constant as well. > > That's great. I really couldn't tell just what information is in there. > What is it, anyway? Something you can explain quickly? It is the ASN.1 schema for various PKCS and PKIX standards, in a compiled form for libtasn1. The variables are constant and read-only throughout the lifetime of the GnuTLS library, except for when it is deallocated in gnutls_global_deinit(). Perhaps GnuTLS should deallocate these variables in an `atexit' hook instead. Then it seem all of this problem would go away. In GNU libc, atexit hooks are even called if GnuTLS was dlopen+dlclose'd. gnutls_global_deinit() would then only call gc_done(). > The information from Nikos that reference counting will be implemented > solves another part of the problem (but I'm a little skeptical about that > because the old code seemed to be pretty deliberate in not doing reference > counting; it actually looks like there was reference counting before and > someone undid it for some reason). It simply looked half-finished to me. With my latest fix, I think this problem has been solved. If you are concerned with the theoretical memory leak, we could adopt the atexit solution. > But there's still more work to do. It isn't thread-safe. Often, a > program is modular in that two threads are executing separate code > independently. If both of those threads decide to use the facilities > of the gnutls library, they might simultaneously call > gnutls_global_init() and the reference count would get screwed up. I'm not sure I see the problem here. GnuTLS should in general be thread safe. Allocating and deallocating these two global read-only variables may be the only thing causing problems. Let's consider if both threads call gnutls_global_init() at the same time. As far as I can see, the worst thing that can happen is that there is a race in: if (_gnutls_init++) goto out; between checking the variable and incrementing it. If there is a thread context switch between checking the variable and increment it, so that both threads read the value 0 from the variable, the worst thing that seem to happen is that the global variables are allocated twice. That is a memory leak, but a pretty insignificant one, and one that would happen rarely. There is no serious problem, that cannot easily be solved by the caller, as far as I can see. Nothing else bad will happen, as far as I can see. > On the other hand, in what I propose each thread has its own memory > and there can't possibly be a conflict. (The synchronization point > moves to malloc(), which is a reliable place for it). Alternatively, each thread could do a mutex around the call to gnutls_global_init: mutex_get() gnutls_global_init(); mutex_release(); Although I wonder if the complexity is worth it. The only danger in not doing this is that if there is a thread race, you could leak some memory. >>Remember that API changes are bad. > > But API _additions_ are no big deal. I'm talking about adding three > new functions for people to use optionally in place of three existing > ones, and people who use the existing ones will see no difference at > all. Good! >>I think it would be useful to see exactly what changes you are >>thinking of in more detail. > > I don't want to go so far as coding a patch if you don't buy into the > basic idea, because it would be too much work. But let me try a more > detailed explanation. This is nothing revolutionary -- it's how lots > of libraries work in order to cater to threaded, modular, object > oriented programs. There are a lot of different solutions to this problem, so I wanted to know exactly which one you thought of. For example, many libraries has chosen to only support one thread library, and then use mutexes from that thread library to solve problems like this. GnuTLS could have done that too, and wrapped gnutls_global_init and gnutls_global_deinit around thread mutexes. But I believe that is too inflexible for GnuTLS, consider this: Application A (uses NPTL) Library B (uses pthreads) GnuTLS Library C (uses GNU Pth) GnuTLS If GnuTLS used a pthread mutex internally to protect against this, it would not work well. I believe it is best to write thread safe code. If that can't be done (and I think we may change the GnuTLS code here), declaring that the function is thread-unsafe and relying on the caller to protect the call around a mutex is the second best bet. Of course, if there is a good argument for another approach, I could change my mind. > A program calls gnutls_create_session_group() instead of > gnutls_global_init(). This new function returns a "session group" handle. > The program then calls gnutls_session_init_group() instead of > gnutls_session_init(). This new function takes that session group handle > as an argument. When it's all done, the program calls > gnutls_destroy_session_group(handle) instead of gnutls_global_deinit(). > > The session handle is really a pointer to a descriptor (struct) that > contains root pointers for the "global" trees. So each session group > has its own copy of the trees. > > The session group handle gets added to the session descriptor, and > everything in gnutls that references the trees references them via the > relevant session group descriptor instead of direct reference to > global variables. Some internal functions need to get an extra > parameter so they know which tree to access. > > So that the existing API keeps working, there is a global session > group descriptor that takes the place of the two global tree pointers. > gnutls_global_init() does a gnutls_create_session_group() of that > descriptor and gnutls_session_init() does a > gnutls_session_init_group() with the global handle. This could work, but I don't exactly see what the problem is that would warrant all those changes. The only gain I see is that you protect against a potential memory leak in gnutls_global_init() that will most likely never happen in real code. For comparison, the constant memory leak inside libgcrypt should be more worrying to you... >>I'm not sure exactly what real problem you are having. > > I thought the basic global variable issue was widely enough known that > there wouldn't be a need to describe a particular scenario, but I'll > tell you where I'm coming from. Read-only constant global variables are not a serious problem as far as I know. I would have completely agreed with you if these variables were modified by any GnuTLS functions. gnutls_global_deinit() does that, but it is a rather special case to me. And the atexit approach seems like an easier solution anyway. > I don't have a program that crashed. I distribute a library > (XML-RPC for C/C++) that uses gnutls. (Actually, it's worse. It > uses Curl, and Curl uses gnutls). My users want to write object > oriented programs with multiple threads. I know that because Curl > has a global init function, and one reason it has that is because > gnutls does, I can't give my users that because my code has to > initialize gnutls and I have no way to keep my gnutls initialization > from conflicting with gnutls initializations that the user does for > other purposes (it's the simultaneous reference count update problem > I described above). Understood. > The best I can do is expose the deep underlying use of gnutls (no easy > thing, since gnutls is only one of Curl's SSH options and Curl is only > one of Xmlrpc-c's HTTP options) and tell the user he has to supply an > already initialized gnutls environment to me. That seem to be a layer violation. I think it could be avoided. > But with the session group concept, it's a snap. I just give each > Xmlrpc-c client object its own Curl client object and each Curl client > object its own gnutls session group, and I can't possible interfere > with the user's other uses of gnutls. The only thing I'm sharing with > them is the code of gnutls. My claim is that with the current code, it should be a snap anyway. Thanks for discussing this, Simon From bryanh at giraffe-data.com Sun Dec 25 19:47:18 2005 From: bryanh at giraffe-data.com (Bryan Henderson) Date: Sun Dec 25 19:47:20 2005 Subject: [gnutls-dev] Re: living without global variables In-Reply-To: (jas@extundo.com) References: <200512211805.jBLI5GeK032128@yxa.extundo.com> <200512212223.53584.nmav@gnutls.org> <41414.1438821845$1135272425@news.gmane.org> <43404.6885501177$1135417620@news.gmane.org> Message-ID: >It is the ASN.1 schema for various PKCS and PKIX standards, in a >compiled form for libtasn1. I have no idea what those words mean, but I guess I don't have to. >Perhaps GnuTLS should deallocate these variables in an `atexit' hook >instead. Then it seem all of this problem would go away. In GNU >libc, atexit hooks are even called if GnuTLS was dlopen+dlclose'd. >gnutls_global_deinit() would then only call gc_done(). If you're willing to do that, you should be willing just to skip the deallocate altogether. On process termination, memory allocation becomes irrelevant. Most engineers I know prefer to have more modularity -- the library doesn't know about process termination or program lifetime; it has a matching setup and shutdown and after shutdown, everything is back the way it was before setup. But this would be acceptable to me (acceptable means I would still use GnuTLS). You know, I've seen the code of dozens of Unix libraries and hundreds of Unix programs, and I don't think I've ever seen atexit() used. And while I've been reminded of its existence from time to time, I've never been able to come with a use for it in my own code. >That is a memory leak, but a pretty insignificant one, and one >that would happen rarely. There is no serious problem, that cannot >easily be solved by the caller, as far as I can see. Nothing else bad >will happen, as far as I can see. You're probably right, though I prefer not to attempt analyses of this kind. It's easier just to say it's thread safe or it's not. In this case, it's not, and you're pointing out that the user can tolerate that as he does any non-thread-safe library -- make sure he doesn't call it simultaneously from two different threads. I also agree that the damage is slight and very unlikely, and a user may be willing just to sweep it under the carpet. But I've never seen anyone take the position that just a little thread unsafety or a little memory leak is OK in published code -- computer people are always interested in precision. >Alternatively, each thread could do a mutex around the call to >gnutls_global_init: And that's the nonmodularity I'm trying to get around. Each thread has to be aware that other modules in other threads might be using the same library and agree with them on a locking protocol. >For example, many libraries has chosen to only support one thread >library, and then use mutexes from that thread library to solve >problems like this. GnuTLS could have done that too, and wrapped >gnutls_global_init and gnutls_global_deinit around thread mutexes. >But I believe that is too inflexible for GnuTLS, consider this: Yes, I agree. There are only two methods I've seen to make a stateful library thread-safe: 1) the library routines use locks 2) the library routines each take a complete context as an argument (usually thought of as the object oriented approach). I prefer (2). >Read-only constant global variables are not a serious problem as far >as I know. These aren't read only, of course -- they get set twice. But the exemption still holds as long as a global variable is repeatably settable. For example, static struct { int version; int maxConnections; } globalStuff; void global_init() { globalStuff.version = 3; globalStuff.maxConnections = 1024; } is acceptable as an alternative to having the loader set those constants with a C initial value declaration. But when there's a malloc involved, you don't have the repeatability (i.e. subsequent sets are not idempotent). What I haven't seen so far in all the fishing for an alternative is what is the drawback to adding the 3 functions to allow a thread to have totally private context if it wants? -- Bryan Henderson Phone 408-621-2000 San Jose, California From nmav at gnutls.org Mon Dec 26 12:16:46 2005 From: nmav at gnutls.org (Nikos Mavrogiannopoulos) Date: Mon Dec 26 12:16:43 2005 Subject: [gnutls-dev] living without global variables In-Reply-To: <200512211805.jBLI5GeK032128@yxa.extundo.com> References: <200512211805.jBLI5GeK032128@yxa.extundo.com> Message-ID: <200512261216.46973.nmav@gnutls.org> On Wednesday 21 December 2005 19:05, Bryan Henderson wrote: > gnutls_global_init() would still be around for programs that want the > simplicity of global variables and don't care about modularity. > gnutls_global_init() would create a session group object and assign > its handle to a global variable. gnutls_init() uses the global > session group handle. This will not help. I don't think there is a practical advantage of that, but if you want to do something like that you have to change all the x509 parsing functions to accept the "so called" session group. The constant stuff that gets initialized by gnutls_global_init() has nothing to do with tls sessions, they are only used for the ASN.1 parser, which is used in certificate parsing. So as far as you are concerned gnutls sessions are thread safe, except for the libgcrypt part of course. -- Nikos Mavrogiannopoulos From nmav at gnutls.org Mon Dec 26 12:20:20 2005 From: nmav at gnutls.org (Nikos Mavrogiannopoulos) Date: Mon Dec 26 12:20:16 2005 Subject: [gnutls-dev] Re: living without global variables In-Reply-To: References: <200512211805.jBLI5GeK032128@yxa.extundo.com> <43404.6885501177$1135417620@news.gmane.org> Message-ID: <200512261220.20431.nmav@gnutls.org> On Sunday 25 December 2005 15:13, Simon Josefsson wrote: > Perhaps GnuTLS should deallocate these variables in an `atexit' hook > instead. Then it seem all of this problem would go away. In GNU > libc, atexit hooks are even called if GnuTLS was dlopen+dlclose'd. > gnutls_global_deinit() would then only call gc_done(). I don't see how this can be usefull. Adding a sentance in the global_init() function that it is not thread safe I think should be sufficient. This is a one time call initialization function, so it is perfectly ok to leave the thread stuff to the called application. -- Nikos Mavrogiannopoulos From Frediano.Ziglio at vodafone.com Mon Dec 26 13:10:44 2005 From: Frediano.Ziglio at vodafone.com (ZIGLIO, Frediano, VF-IT) Date: Mon Dec 26 14:05:28 2005 Subject: [gnutls-dev] Re: living without global variables Message-ID: > > bryanh@giraffe-data.com (Bryan Henderson) writes: > > >>The entire tree is constant as well. > > > > That's great. I really couldn't tell just what information > is in there. > > What is it, anyway? Something you can explain quickly? > > It is the ASN.1 schema for various PKCS and PKIX standards, in a > compiled form for libtasn1. The variables are constant and read-only > throughout the lifetime of the GnuTLS library, except for when it is > deallocated in gnutls_global_deinit(). > > Perhaps GnuTLS should deallocate these variables in an `atexit' hook > instead. Then it seem all of this problem would go away. In GNU > libc, atexit hooks are even called if GnuTLS was dlopen+dlclose'd. > gnutls_global_deinit() would then only call gc_done(). > Use of atexit is not portable. On many platforms atexit routines get call when program terminate. Using in a DSO can lead calling not existing code. I did some research in how to handle this but I didn't find a portable way to do this... If you detect gcc the better solution is to use __attribute__((destructor)) (not easy to check this on configure script). ... > > > But there's still more work to do. It isn't thread-safe. Often, a > > program is modular in that two threads are executing separate code > > independently. If both of those threads decide to use the > facilities > > of the gnutls library, they might simultaneously call > > gnutls_global_init() and the reference count would get screwed up. > > I'm not sure I see the problem here. GnuTLS should in general be > thread safe. Allocating and deallocating these two global read-only > variables may be the only thing causing problems. Let's consider if > both threads call gnutls_global_init() at the same time. As far as I > can see, the worst thing that can happen is that there is a race in: > > if (_gnutls_init++) > goto out; > > between checking the variable and incrementing it. If there is a > thread context switch between checking the variable and increment it, > so that both threads read the value 0 from the variable, the worst > thing that seem to happen is that the global variables are allocated > twice. That is a memory leak, but a pretty insignificant one, and one > that would happen rarely. There is no serious problem, that cannot > easily be solved by the caller, as far as I can see. Nothing else bad > will happen, as far as I can see. > No, during initialization you set up some linked list so is not a good idea to initialize them twice at the same time. > > On the other hand, in what I propose each thread has its own memory > > and there can't possibly be a conflict. (The synchronization point > > moves to malloc(), which is a reliable place for it). > > Alternatively, each thread could do a mutex around the call to > gnutls_global_init: > > mutex_get() > gnutls_global_init(); > mutex_release(); > > Although I wonder if the complexity is worth it. The only danger in > not doing this is that if there is a thread race, you could leak some > memory. > This do not solve problems if gnutls_global_init is called by two libraries using two separate mutexes (assuming library A does not know library B you cannot share same mutex). > >>Remember that API changes are bad. > > > > But API _additions_ are no big deal. I'm talking about adding three > > new functions for people to use optionally in place of > three existing > > ones, and people who use the existing ones will see no difference at > > all. > > Good! > IMHO with some changes is possible to retain also ABI... > >>I think it would be useful to see exactly what changes you are > >>thinking of in more detail. > > > > I don't want to go so far as coding a patch if you don't > buy into the > > basic idea, because it would be too much work. But let me > try a more > > detailed explanation. This is nothing revolutionary -- > it's how lots > > of libraries work in order to cater to threaded, modular, object > > oriented programs. > > There are a lot of different solutions to this problem, so I wanted to > know exactly which one you thought of. > > For example, many libraries has chosen to only support one thread > library, and then use mutexes from that thread library to solve > problems like this. GnuTLS could have done that too, and wrapped > gnutls_global_init and gnutls_global_deinit around thread mutexes. > But I believe that is too inflexible for GnuTLS, consider this: > > Application A (uses NPTL) > Library B (uses pthreads) > GnuTLS > Library C (uses GNU Pth) > GnuTLS > > If GnuTLS used a pthread mutex internally to protect against this, it > would not work well. > > I believe it is best to write thread safe code. If that can't be done > (and I think we may change the GnuTLS code here), declaring that the > function is thread-unsafe and relying on the caller to protect the > call around a mutex is the second best bet. Of course, if there is a > good argument for another approach, I could change my mind. > An easy way to make GnuTLS thread-safe is use mutex directly from GnuTLS. > > A program calls gnutls_create_session_group() instead of > > gnutls_global_init(). This new function returns a "session > group" handle. > > The program then calls gnutls_session_init_group() instead of > > gnutls_session_init(). This new function takes that > session group handle > > as an argument. When it's all done, the program calls > > gnutls_destroy_session_group(handle) instead of > gnutls_global_deinit(). > > > > The session handle is really a pointer to a descriptor (struct) that > > contains root pointers for the "global" trees. So each > session group > > has its own copy of the trees. > > > > The session group handle gets added to the session descriptor, and > > everything in gnutls that references the trees references > them via the > > relevant session group descriptor instead of direct reference to > > global variables. Some internal functions need to get an extra > > parameter so they know which tree to access. > > > > So that the existing API keeps working, there is a global session > > group descriptor that takes the place of the two global > tree pointers. > > gnutls_global_init() does a gnutls_create_session_group() of that > > descriptor and gnutls_session_init() does a > > gnutls_session_init_group() with the global handle. > > This could work, but I don't exactly see what the problem is that > would warrant all those changes. The only gain I see is that you > protect against a potential memory leak in gnutls_global_init() that > will most likely never happen in real code. For comparison, the > constant memory leak inside libgcrypt should be more worrying to > you... > The session idea works however it require extra parameters in many functions... This has also an advantage. Assuming someone can add ciphers or other informations to GnuTLS these informations are not used globally but only limited for each session. Also it fix all DSO problems cause memory deallocation is demanded to caller, not to GnuTLS. This also make it possible to demand thread-safety to caller. ... freddy77 From jas at extundo.com Mon Dec 26 18:51:21 2005 From: jas at extundo.com (Simon Josefsson) Date: Mon Dec 26 18:51:43 2005 Subject: [gnutls-dev] Re: living without global variables In-Reply-To: <36953.281882643$1135536521@news.gmane.org> (Bryan Henderson's message of "Sun, 25 Dec 2005 18:48:41 +0000 (UTC)") References: <200512211805.jBLI5GeK032128@yxa.extundo.com> <200512212223.53584.nmav@gnutls.org> <41414.1438821845$1135272425@news.gmane.org> <43404.6885501177$1135417620@news.gmane.org> <36953.281882643$1135536521@news.gmane.org> Message-ID: bryanh@giraffe-data.com (Bryan Henderson) writes: >>It is the ASN.1 schema for various PKCS and PKIX standards, in a >>compiled form for libtasn1. > > I have no idea what those words mean, but I guess I don't have to. Regard them as read-only data structures. The variables are used by code that has to parse X.509 certificates or other PKCS structures. Several of those functions doesn't have a gnutls_session parameter. >>Perhaps GnuTLS should deallocate these variables in an `atexit' hook >>instead. Then it seem all of this problem would go away. In GNU >>libc, atexit hooks are even called if GnuTLS was dlopen+dlclose'd. >>gnutls_global_deinit() would then only call gc_done(). > > If you're willing to do that, you should be willing just to skip the > deallocate altogether. On process termination, memory allocation > becomes irrelevant. Most engineers I know prefer to have more > modularity -- the library doesn't know about process termination or > program lifetime; it has a matching setup and shutdown and after > shutdown, everything is back the way it was before setup. But this > would be acceptable to me (acceptable means I would still use GnuTLS). > > You know, I've seen the code of dozens of Unix libraries and hundreds > of Unix programs, and I don't think I've ever seen atexit() used. And > while I've been reminded of its existence from time to time, I've > never been able to come with a use for it in my own code. I was merely suggesting it as a solution, and as Freddy pointed out, using atexit in a library is probably a bad idea, so I agree with you. >>That is a memory leak, but a pretty insignificant one, and one >>that would happen rarely. There is no serious problem, that cannot >>easily be solved by the caller, as far as I can see. Nothing else bad >>will happen, as far as I can see. > > You're probably right, though I prefer not to attempt analyses of this > kind. It's easier just to say it's thread safe or it's not. In this > case, it's not, and you're pointing out that the user can tolerate > that as he does any non-thread-safe library -- make sure he doesn't > call it simultaneously from two different threads. I also agree that > the damage is slight and very unlikely, and a user may be willing just > to sweep it under the carpet. But I've never seen anyone take the > position that just a little thread unsafety or a little memory leak is > OK in published code -- computer people are always interested in > precision. If you want precision, gnutls_global_init() is not thread safe. >>Alternatively, each thread could do a mutex around the call to >>gnutls_global_init: > > And that's the nonmodularity I'm trying to get around. Each thread has > to be aware that other modules in other threads might be using the same > library and agree with them on a locking protocol. I think you need to show us how to do the things you describe. I don't see a way to make things better than they are now. I believe libgcrypt initialization has a similar same race-condition. >>For example, many libraries has chosen to only support one thread >>library, and then use mutexes from that thread library to solve >>problems like this. GnuTLS could have done that too, and wrapped >>gnutls_global_init and gnutls_global_deinit around thread mutexes. >>But I believe that is too inflexible for GnuTLS, consider this: > > Yes, I agree. > > There are only two methods I've seen to make a stateful library thread-safe: > > 1) the library routines use locks > > 2) the library routines each take a complete context as an argument > (usually thought of as the object oriented approach). > > I prefer (2). There is another popular approach: 3) Declare functions thread-unsafe, and let the caller handle the thread issue. This approach is used by many POSIX and libc functions. Another possibility is: 4) Store the global variables in thread-local storage. There is a gnulib module that works with pthread, pth, solaris threads and win32 threads to do this. >>Read-only constant global variables are not a serious problem as far >>as I know. > > These aren't read only, of course -- they get set twice. I mean read-only except in gnutls_global_init and gnutls_global_deinit. Those two functions are not thread safe, and were not intended to be (hence "global"). Doesn't your library has a global initialization method? > But the exemption still holds as long as a global variable is > repeatably settable. For example, > > static struct { > int version; > int maxConnections; > } globalStuff; > > void > global_init() { > globalStuff.version = 3; > globalStuff.maxConnections = 1024; > } > > is acceptable as an alternative to having the loader set those > constants with a C initial value declaration. But when there's a > malloc involved, you don't have the repeatability (i.e. subsequent > sets are not idempotent). > > What I haven't seen so far in all the fishing for an alternative is > what is the drawback to adding the 3 functions to allow a thread to > have totally private context if it wants? Someone has to write the patch, and test it, and I proposed something that I thought would be simpler. If you write the patch, we can look at it, and if it works, and doesn't have other problems, we could use it. However, remember that gnutls_global_init would still be thread unsafe, since libgcrypt initialization is thread-unsafe, so your problem would persist. I am trying to rewrite the crypto engine so that we can use crypto modules from gnulib, to avoid libgcrypt, to fix that problem. Help on doing that would also be appreciated, and might solve your concerns in the long run. Thanks, Simon From jas at extundo.com Mon Dec 26 18:59:09 2005 From: jas at extundo.com (Simon Josefsson) Date: Mon Dec 26 18:59:12 2005 Subject: [gnutls-dev] Re: living without global variables In-Reply-To: <200512261220.20431.nmav@gnutls.org> (Nikos Mavrogiannopoulos's message of "Mon, 26 Dec 2005 12:20:20 +0100") References: <200512211805.jBLI5GeK032128@yxa.extundo.com> <43404.6885501177$1135417620@news.gmane.org> <200512261220.20431.nmav@gnutls.org> Message-ID: Nikos Mavrogiannopoulos writes: > On Sunday 25 December 2005 15:13, Simon Josefsson wrote: > >> Perhaps GnuTLS should deallocate these variables in an `atexit' hook >> instead. Then it seem all of this problem would go away. In GNU >> libc, atexit hooks are even called if GnuTLS was dlopen+dlclose'd. >> gnutls_global_deinit() would then only call gc_done(). > > I don't see how this can be usefull. Adding a sentance in the > global_init() function that it is not thread safe I think should be > sufficient. This is a one time call initialization function, so it is > perfectly ok to leave the thread stuff to the called application. Agreed. I have installed the patch below. Thanks, Simon --- gnutls_global.c 23 Dec 2005 15:49:27 +0100 2.110 +++ gnutls_global.c 26 Dec 2005 18:58:22 +0100 @@ -171,6 +171,21 @@ * you must do it before calling this function. This is useful in cases you * want to disable libgcrypt's internal lockings etc. * + * This function increment a global counter, so that + * gnutls_global_deinit() only releases resources when it has been + * called as many times as gnutls_global_init(). This is useful when + * GnuTLS is used by more than one library in an application. This + * function can be called many times, but will only do something the + * first time. + * + * Note! This function is not thread safe. If two threads call this + * function simultaneously, they can cause a race between checking + * the global counter and incrementing it, causing both threads to + * execute the library initialization code. That would lead to a + * memory leak. To handle this, your application could invoke this + * function after aquiring a thread mutex. To ignore the potential + * memory leak is also an option. + * **/ int gnutls_global_init (void) @@ -265,8 +280,10 @@ * This function deinitializes the global data, that were initialized * using gnutls_global_init(). * + * Note! This function is not thread safe. See the discussion for + * gnutls_global_init() for more information. + * **/ - void gnutls_global_deinit (void) { From jas at extundo.com Mon Dec 26 19:16:56 2005 From: jas at extundo.com (Simon Josefsson) Date: Mon Dec 26 19:16:58 2005 Subject: [gnutls-dev] Re: living without global variables In-Reply-To: (Frediano ZIGLIO's message of "Mon, 26 Dec 2005 13:10:44 +0100") References: Message-ID: "ZIGLIO, Frediano, VF-IT" writes: > Use of atexit is not portable. On many platforms atexit routines get > call when program terminate. Using in a DSO can lead calling not > existing code. I did some research in how to handle this but I didn't > find a portable way to do this... If you detect gcc the better solution > is to use __attribute__((destructor)) (not easy to check this on > configure script). This sounds bad. Hopefully other solutions will be suggested that work better. >> > On the other hand, in what I propose each thread has its own memory >> > and there can't possibly be a conflict. (The synchronization point >> > moves to malloc(), which is a reliable place for it). >> >> Alternatively, each thread could do a mutex around the call to >> gnutls_global_init: >> >> mutex_get() >> gnutls_global_init(); >> mutex_release(); >> >> Although I wonder if the complexity is worth it. The only danger in >> not doing this is that if there is a thread race, you could leak some >> memory. >> > > This do not solve problems if gnutls_global_init is called by two > libraries using two separate mutexes (assuming library A does not know > library B you cannot share same mutex). Right, the function is not thread safe. >> >>Remember that API changes are bad. >> > >> > But API _additions_ are no big deal. I'm talking about adding three >> > new functions for people to use optionally in place of >> three existing >> > ones, and people who use the existing ones will see no difference at >> > all. >> >> Good! >> > > IMHO with some changes is possible to retain also ABI... I'm interested in seeing what changes you are thinking of. To me, it sounds like a lot of work to rewrite all of GnuTLS to use non-global ASN.1 structures. I think it is easier to solve the problem by letting the caller of gnutls_global_init make sure it isn't called twice at the same time. > An easy way to make GnuTLS thread-safe is use mutex directly from > GnuTLS. But as you note above, this doesn't work. If library A use GnuTLS but doesn't use threads, and library B use GnuTLS but does use threads, the mutex will be useless. > The session idea works however it require extra parameters in many > functions... This has also an advantage. Assuming someone can add > ciphers or other informations to GnuTLS these informations are not used > globally but only limited for each session. Also it fix all DSO problems > cause memory deallocation is demanded to caller, not to GnuTLS. This > also make it possible to demand thread-safety to caller. Patches are welcome... /Simon From jas at extundo.com Mon Dec 26 19:24:33 2005 From: jas at extundo.com (Simon Josefsson) Date: Mon Dec 26 19:24:33 2005 Subject: [gnutls-dev] Another solution Message-ID: Here is another solution for removing the global variables without changing any APIs [1]: replace all uses of the global variable with a function call to a function that generate a local copy of the ASN.1 structures. This will be slower, but would work. However, gnutls_global_init() will not become thread safe by this (because libgcrypt is used) but all the global variables in GnuTLS would then be removed. Thus, the above proposal to get rid of all global variables would not solve your problem with thread safety anyway, so I propose we wait with this idea until libgcrypt is thread safe or we support alternatives to libgcrypt. I again wish to stress that I believe this is a insignificant problem. Simply have all libraries using GnuTLS call gnutls_global_init() in a thread-unsafe initialization function, ultimately mutex synchronized by the application. [1] Changing APIs is really bad, that is why I'm having trouble with your proposed approach. From bryanh at giraffe-data.com Mon Dec 26 19:25:59 2005 From: bryanh at giraffe-data.com (Bryan Henderson) Date: Mon Dec 26 19:26:00 2005 Subject: [gnutls-dev] living without global variables In-Reply-To: <200512261216.46973.nmav@gnutls.org> References: <200512211805.jBLI5GeK032128@yxa.extundo.com> <200512261216.46973.nmav@gnutls.org> Message-ID: >if you want to do something like that you have to change >all the x509 parsing functions to accept the "so called" session group. I see. I suspected there'd be something outside of session context like this that uses the global variables. So those functions would have to have private-context versions as well. (The name "session group" is not great, but I couldn't think of a better name for the context -- I believe the whole point of using GnuTLS is to make sessions, so that's where I got that. I notice OpenSSL just gives up and calls it "context"! >So as far as you are concerned gnutls sessions are thread safe, except >for the libgcrypt part of course. I agree the sessions are thread safe. The problem I'm having is with gnutls_global_init() not being thread safe. >except for the libgcrypt part of course. libgcrypt is in the same boat? I was afraid of that. -- Bryan Henderson Phone 408-621-2000 San Jose, California From bryanh at giraffe-data.com Mon Dec 26 19:35:30 2005 From: bryanh at giraffe-data.com (Bryan Henderson) Date: Mon Dec 26 19:35:31 2005 Subject: [gnutls-dev] Re: living without global variables In-Reply-To: <200512261220.20431.nmav@gnutls.org> References: <200512211805.jBLI5GeK032128@yxa.extundo.com> <43404.6885501177$1135417620@news.gmane.org> <200512261220.20431.nmav@gnutls.org> Message-ID: >This is a one time call initialization function, so it is >perfectly ok to leave the thread stuff to the called application. The reason it's not _perfectly_ OK is what I started this whole discussion with: lack of modularity. The calling code may be a module of a larger program. That larger program may contain another module which also uses GnuTLS. The two modules run in their own threads. As modules, they don't know about each other and cannot coordinate their calls to gnutls_global_init(). Modularity also forbids the code which is the parent of the threads/modules from knowing that GNUtls is down inside those modules, so the parent code can't call gnutls_global_init() either. It's _reasonable_ to produce a GnuTLS that has a nonthreadsafe gnutls_global_init() and not threadsafe alternative -- it just means the library isn't all it could be. -- Bryan Henderson Phone 408-621-2000 San Jose, California From bryanh at giraffe-data.com Mon Dec 26 20:45:18 2005 From: bryanh at giraffe-data.com (Bryan Henderson) Date: Mon Dec 26 20:45:20 2005 Subject: [gnutls-dev] Re: living without global variables In-Reply-To: (jas@extundo.com) References: Message-ID: >The variables are used by code that has to parse X.509 certificates or >other PKCS structures. Now I'm starting to wonder just how constant this information is (I'm talking about the information now, not the program variables that represent it). We know it's constant for the life of a program, but is it constant for all programs? I.e. could it conceivably be generated at library build time and loaded whole by the program/library loader? >> There are only two methods I've seen to make a stateful library thread-safe: > >There is another popular approach: > >3) Declare functions thread-unsafe, I hate to be pedantic (well, not really), but (3) is not an _approach_ to making a library thread-safe; it's a _retreat_. >4) Store the global variables in thread-local storage. There is a > gnulib module that works with pthread, pth, solaris threads and > win32 threads to do this. Thanks; I forgot that one. I've only ever seen it in libc, probably because of the portability difficulty. >Here is another solution for removing the global variables without >changing any APIs [1]: replace all uses of the global variable with a >function call to a function that generate a local copy of the ASN.1 >structures. This will be slower, but would work. It _sounds_ impractically slow. But then I don't know enough about the code to know. >Simply have all libraries using GnuTLS call gnutls_global_init() in a >thread-unsafe initialization function, ultimately mutex synchronized >by the application. That initialization function (call it foo_global_init()) is more than thread-unsafe. Thread-unsafe means you can't call foo_global_init() simultaneously from two threads. But foo_global_init() not only can't be called while another thread is calling it, but also not while another thread is running at all. Because that thread might be calling gnutls_global_init() via something other than foo_global_init(). But I guess this is probably the way to go, considering how much easier it is to implement. The paradigm is that the global initialization/termination must propagate through all the libraries up to the main program initialization code where the program is a single thread. The object oriented libraries are a little less object oriented, since you can't just haul off and create an object whenever you want (you have to think about the global state), but it's a whole lot less coding. And I guess the program can't dynamically load any of these libraries. I note that even removing global variables from GnuTLS and libgcrypt just begins to solve my problem, because libcurl also uses 4 other libraries, in various environments, that have the same global initialization requirement. But the single-threaded reference-counted global init thing can work for all of them. >[1] Changing APIs is really bad, that is why I'm having trouble with >your proposed approach. I thought I cleared this up before, but here it is again, so I guess not. Is it your opinion that changing an API by adding optional functions to it is really bad? That's what I proposed. In contrast, the "really bad" change to an API most people mean is doing something that makes programs that use the existing API have to change. -- Bryan Henderson Phone 408-621-2000 San Jose, California From Frediano.Ziglio at vodafone.com Tue Dec 27 11:25:58 2005 From: Frediano.Ziglio at vodafone.com (ZIGLIO, Frediano, VF-IT) Date: Tue Dec 27 11:25:38 2005 Subject: [gnutls-dev] constification patch Message-ID: This simple patch add some const definition. It also assure that global variables are really const. Frediano -------------- next part -------------- A non-text attachment was scrubbed... Name: gnutls.diff.gz Type: application/x-gzip Size: 4888 bytes Desc: gnutls.diff.gz Url : /pipermail/attachments/20051227/ee904940/gnutls.diff.bin From jas at extundo.com Wed Dec 28 10:07:12 2005 From: jas at extundo.com (Simon Josefsson) Date: Wed Dec 28 10:07:34 2005 Subject: [gnutls-dev] Re: living without global variables In-Reply-To: <3642.39705163545$1135622219@news.gmane.org> (Bryan Henderson's message of "Mon, 26 Dec 2005 18:36:59 +0000 (UTC)") References: <200512211805.jBLI5GeK032128@yxa.extundo.com> <43404.6885501177$1135417620@news.gmane.org> <200512261220.20431.nmav@gnutls.org> <3642.39705163545$1135622219@news.gmane.org> Message-ID: bryanh@giraffe-data.com (Bryan Henderson) writes: >>This is a one time call initialization function, so it is >>perfectly ok to leave the thread stuff to the called application. > > The reason it's not _perfectly_ OK is what I started this whole > discussion with: lack of modularity. The calling code may be a module > of a larger program. That larger program may contain another module > which also uses GnuTLS. The two modules run in their own threads. As > modules, they don't know about each other and cannot coordinate their > calls to gnutls_global_init(). Modularity also forbids the code which > is the parent of the threads/modules from knowing that GNUtls is down > inside those modules, so the parent code can't call > gnutls_global_init() either. > > It's _reasonable_ to produce a GnuTLS that has a nonthreadsafe > gnutls_global_init() and not threadsafe alternative -- it just means the > library isn't all it could be. If there is a way to fix the problem, without causing other problems, of course we will do it. From jas at extundo.com Wed Dec 28 10:13:52 2005 From: jas at extundo.com (Simon Josefsson) Date: Wed Dec 28 10:14:41 2005 Subject: [gnutls-dev] Re: living without global variables References: <7997.31945521689$1135626397@news.gmane.org> Message-ID: bryanh@giraffe-data.com (Bryan Henderson) writes: >>The variables are used by code that has to parse X.509 certificates or >>other PKCS structures. > > Now I'm starting to wonder just how constant this information is (I'm > talking about the information now, not the program variables that > represent it). We know it's constant for the life of a program, but > is it constant for all programs? I.e. could it conceivably be > generated at library build time and loaded whole by the > program/library loader? Yes, that is roughly how it works now. Read the code, the global variables are another representation of the information in the generated lib/gnutls_asn1_tab.c and lib/pkix_asn1_tab.c files. It may be possible to make those two files contain more complex data structures that is easier to use directly by the ASN.1 library. That would avoid the allocation/deallocation of the intermediary format. >>4) Store the global variables in thread-local storage. There is a >> gnulib module that works with pthread, pth, solaris threads and >> win32 threads to do this. > > Thanks; I forgot that one. I've only ever seen it in libc, probably > because of the portability difficulty. I think gettext uses it. I don't like the portability problems, and the problems of being in an application that uses two thread packages at the same time. So I don't think that is a good solution. >>Here is another solution for removing the global variables without >>changing any APIs [1]: replace all uses of the global variable with a >>function call to a function that generate a local copy of the ASN.1 >>structures. This will be slower, but would work. > > It _sounds_ impractically slow. But then I don't know enough about the > code to know. Modularity comes with a price... if we can't come up with something else, we at least know there is a solution. >>Simply have all libraries using GnuTLS call gnutls_global_init() in a >>thread-unsafe initialization function, ultimately mutex synchronized >>by the application. > > That initialization function (call it foo_global_init()) is more than > thread-unsafe. Thread-unsafe means you can't call foo_global_init() > simultaneously from two threads. But foo_global_init() not only can't > be called while another thread is calling it, but also not while > another thread is running at all. Because that thread might be > calling gnutls_global_init() via something other than > foo_global_init(). > > But I guess this is probably the way to go, considering how much > easier it is to implement. The paradigm is that the global > initialization/termination must propagate through all the libraries up > to the main program initialization code where the program is a single > thread. The object oriented libraries are a little less object > oriented, since you can't just haul off and create an object whenever > you want (you have to think about the global state), but it's a whole > lot less coding. And I guess the program can't dynamically load any > of these libraries. > > I note that even removing global variables from GnuTLS and libgcrypt > just begins to solve my problem, because libcurl also uses 4 other > libraries, in various environments, that have the same global > initialization requirement. But the single-threaded reference-counted > global init thing can work for all of them. And I think that is a reasonable approach, unless we can think of a better solution. Many complex libraries I've seen have a global setup function, e.g., gettext, glib. When we have a replacement for libgcrypt, we can reconsider this issue, and then I think it would even be possible to get a thread safe gnutls_global_init() -- the only remaining problem then would be the two global variables. And that might be fixed by generating the *_asn1_tab.c files in a different format, that the asn1 library can read without having to allocate and set up a linked list for it. >>[1] Changing APIs is really bad, that is why I'm having trouble with >>your proposed approach. > > I thought I cleared this up before, but here it is again, so I guess > not. Is it your opinion that changing an API by adding optional > functions to it is really bad? That's what I proposed. In contrast, > the "really bad" change to an API most people mean is doing something > that makes programs that use the existing API have to change. Changing APIs are bad, adding APIs are fine. I simply could not see how you would fix the problem with your approach without changing the API, that's why I repeated it. Thanks, Simon From jas at extundo.com Wed Dec 28 10:28:58 2005 From: jas at extundo.com (Simon Josefsson) Date: Wed Dec 28 10:28:54 2005 Subject: [gnutls-dev] Re: constification patch In-Reply-To: (Frediano ZIGLIO's message of "Tue, 27 Dec 2005 11:25:58 +0100") References: Message-ID: "ZIGLIO, Frediano, VF-IT" writes: > This simple patch add some const definition. It also assure that global > variables are really const. I installed most of the patch. The lib/minitasn1/ changes didn't look right, so I didn't install them. First, 'inline' can't be used because libtasn1 is supposed to be C89 portable. Also, isn't it possible to solve this without creating new *_const functions? Moving typecasts into a special function just hide the problems. I think warnings are better here because then we know that there is a problem. Thanks, Simon From Frediano.Ziglio at vodafone.com Wed Dec 28 14:00:03 2005 From: Frediano.Ziglio at vodafone.com (ZIGLIO, Frediano, VF-IT) Date: Wed Dec 28 14:00:14 2005 Subject: [gnutls-dev] RE: constification patch Message-ID: > > "ZIGLIO, Frediano, VF-IT" writes: > > > This simple patch add some const definition. It also assure > that global > > variables are really const. > > I installed most of the patch. > > The lib/minitasn1/ changes didn't look right, so I didn't install > them. First, 'inline' can't be used because libtasn1 is supposed to > be C89 portable. Also, isn't it possible to solve this without > creating new *_const functions? Moving typecasts into a special > function just hide the problems. I think warnings are better here > because then we know that there is a problem. > > Thanks, > Simon > Hi Simon, I would never expected that you ever apply the patch so fast! Well, I came from "C++ world" were const is very important but easier to use than in C however after some year developing OpenSource C applications I have to say that const is important even on C... The *_const functions reflect how C++ work... but in C. In C++ you can declare two functions like char *strchr(char *, int); const char *strchr(const char *, int); without problems, in C this give error... so I added _const. The advantage is that compiler help you. Usually many people do not pay many attention to warning but some time ignoring warnings can lead to disaster... so usually I remove any type of warning I found. In this case I call a function for no-const object knowing that this function do not change the data. Well, after explaining my reasons let's came to a possible (and portable) fix. A way to fix this issue is to use static inline for compilers that support this or define for others. As I think that many people will use gcc so they will see the error I would convert static inline const node_asn * _asn1_find_up_const(const node_asn *node) { return _asn1_find_up((node_asn*) node); } to #if defined(__GNUC__) && __GNUC__ >= 3 static inline const node_asn * _asn1_find_up_const(const node_asn *node) { return _asn1_find_up((node_asn*) node); } #else #define _asn1_find_up_const(node) _asn1_find_up((node_asn*) (node)) #endif freddy77 From jas at extundo.com Wed Dec 28 15:02:28 2005 From: jas at extundo.com (Simon Josefsson) Date: Wed Dec 28 15:02:30 2005 Subject: [gnutls-dev] Re: constification patch In-Reply-To: (Frediano ZIGLIO's message of "Wed, 28 Dec 2005 14:00:03 +0100") References: Message-ID: "ZIGLIO, Frediano, VF-IT" writes: >> >> "ZIGLIO, Frediano, VF-IT" writes: >> >> > This simple patch add some const definition. It also assure >> that global >> > variables are really const. >> >> I installed most of the patch. >> >> The lib/minitasn1/ changes didn't look right, so I didn't install >> them. First, 'inline' can't be used because libtasn1 is supposed to >> be C89 portable. Also, isn't it possible to solve this without >> creating new *_const functions? Moving typecasts into a special >> function just hide the problems. I think warnings are better here >> because then we know that there is a problem. >> >> Thanks, >> Simon >> > > Hi Simon, > I would never expected that you ever apply the patch so fast! > > Well, I came from "C++ world" were const is very important but easier to > use than in C however after some year developing OpenSource C > applications I have to say that const is important even on C... The > *_const functions reflect how C++ work... but in C. In C++ you can > declare two functions like > > char *strchr(char *, int); > const char *strchr(const char *, int); > > without problems, in C this give error... so I added _const. The > advantage is that compiler help you. Usually many people do not pay many > attention to warning but some time ignoring warnings can lead to > disaster... so usually I remove any type of warning I found. In this > case I call a function for no-const object knowing that this function do > not change the data. > > Well, after explaining my reasons let's came to a possible (and > portable) fix. A way to fix this issue is to use static inline for > compilers that support this or define for others. As I think that many > people will use gcc so they will see the error I would convert > > static inline const node_asn * > _asn1_find_up_const(const node_asn *node) > { > return _asn1_find_up((node_asn*) node); > } > > to > > #if defined(__GNUC__) && __GNUC__ >= 3 > static inline const node_asn * > _asn1_find_up_const(const node_asn *node) > { > return _asn1_find_up((node_asn*) node); > } > #else > #define _asn1_find_up_const(node) _asn1_find_up((node_asn*) (node)) > #endif How about changing the prototype of _asn1_find_up to include the const keyword instead? That seem more correct. Using typecasts like you do hide problems: what if _asn1_find_up does not preserve const? That would lead to a disaster, and the compiler will not warn about it. Further, it has not been written down, but the coding style for minitasn1 is slightly different than GnuTLS. Minitasn1 should be portable C89 code, and #if's like that make the code unreadable. Unless profiling show that you'd gain a non-negligible amount of time making that particular function inline (I highly doubt that), then it is worth the extra CPU cycles to have readable code. Optimize only when you need. In contrast, in GnuTLS we can use 'inline' unconditionally because there is a gnulib M4 test that make sure 'inline' is defined to "" if it is not available. If you update the patch, to change the prototype of the existing functions to use 'const' too, I'll review it. I know there is a lot of missing 'const' keywords in various places. Thanks, Simon From Frediano.Ziglio at vodafone.com Wed Dec 28 16:07:41 2005 From: Frediano.Ziglio at vodafone.com (ZIGLIO, Frediano, VF-IT) Date: Wed Dec 28 16:07:48 2005 Subject: [gnutls-dev] RE: constification patch Message-ID: > > "ZIGLIO, Frediano, VF-IT" writes: > > >> > >> "ZIGLIO, Frediano, VF-IT" writes: > >> > >> > This simple patch add some const definition. It also assure > >> that global > >> > variables are really const. > >> > >> I installed most of the patch. > >> > >> The lib/minitasn1/ changes didn't look right, so I didn't install > >> them. First, 'inline' can't be used because libtasn1 is > supposed to > >> be C89 portable. Also, isn't it possible to solve this without > >> creating new *_const functions? Moving typecasts into a special > >> function just hide the problems. I think warnings are better here > >> because then we know that there is a problem. > >> > >> Thanks, > >> Simon > >> > > > > Hi Simon, > > I would never expected that you ever apply the patch so fast! > > > > Well, I came from "C++ world" were const is very important > but easier to > > use than in C however after some year developing OpenSource C > > applications I have to say that const is important even on C... The > > *_const functions reflect how C++ work... but in C. In C++ you can > > declare two functions like > > > > char *strchr(char *, int); > > const char *strchr(const char *, int); > > > > without problems, in C this give error... so I added _const. The > > advantage is that compiler help you. Usually many people do > not pay many > > attention to warning but some time ignoring warnings can lead to > > disaster... so usually I remove any type of warning I found. In this > > case I call a function for no-const object knowing that > this function do > > not change the data. > > > > Well, after explaining my reasons let's came to a possible (and > > portable) fix. A way to fix this issue is to use static inline for > > compilers that support this or define for others. As I > think that many > > people will use gcc so they will see the error I would convert > > > > static inline const node_asn * > > _asn1_find_up_const(const node_asn *node) > > { > > return _asn1_find_up((node_asn*) node); > > } > > > > to > > > > #if defined(__GNUC__) && __GNUC__ >= 3 > > static inline const node_asn * > > _asn1_find_up_const(const node_asn *node) > > { > > return _asn1_find_up((node_asn*) node); > > } > > #else > > #define _asn1_find_up_const(node) _asn1_find_up((node_asn*) (node)) > > #endif > > How about changing the prototype of _asn1_find_up to include the const > keyword instead? That seem more correct. > Yes and not... you end up changing some const* to no const in _asn1_find_up... the reason is that for const node_asn* I mean that all ASN tree is constant so is a bad idea to return a node_asn *... > Using typecasts like you do hide problems: what if _asn1_find_up does > not preserve const? That would lead to a disaster, and the compiler > will not warn about it. > Perhaps it would be more polite if: we define a _asn1_find_up_const function in C source (renaming _asn1_find_up to const) which take const and return const and a _asn1_find_up function that take no-const and return no-const (a wrapper to previous one). This would also allow easy detection of invalid changes in _asn1_find_up_const. > Further, it has not been written down, but the coding style for > minitasn1 is slightly different than GnuTLS. Minitasn1 should be > portable C89 code, and #if's like that make the code unreadable. > Unless profiling show that you'd gain a non-negligible amount of time > making that particular function inline (I highly doubt that), then it > is worth the extra CPU cycles to have readable code. Optimize only > when you need. In contrast, in GnuTLS we can use 'inline' > unconditionally because there is a gnulib M4 test that make sure > 'inline' is defined to "" if it is not available. > No problem for optimizazion, if we use inline only for gcc gcc can do a good work. Using define however do not check for const/no-const (C don't have a const_cast like C++...). I don't find however a solution that can be portable and readable as you want and also optimized and with "efficient" warnings... but we can catch three of them (portable, readable, clean warning) with a "normal" _asn1_find_up_const function (no static, no inline, in source file with declaration). > If you update the patch, to change the prototype of the existing > functions to use 'const' too, I'll review it. I know there is a lot > of missing 'const' keywords in various places. > > Thanks, > Simon > Ok I'll look a bit deeper. I have to CVS direct access here so I'll have to wait tomorrow to integrate my patch... Frediano Ziglio From bryanh at giraffe-data.com Wed Dec 28 18:29:53 2005 From: bryanh at giraffe-data.com (Bryan Henderson) Date: Wed Dec 28 18:29:54 2005 Subject: [gnutls-dev] Re: constification patch In-Reply-To: (jas@extundo.com) References: Message-ID: I've spent some time investigating the issue of "inline" keywords in portable code. For those who haven't had the pleasure: First of all, the "inline" keyword rarely affects speed. Why? Because a modern compiler inlines on its own when it helps. (And if you disable optimization, many will _not_ inline even if you use the keyword). Some compilers actually ignore the "inline" keyword completely, and others don't have one. So no one should be afraid to take some lines out of the middle of a function and make it a separate function for readability, out of fear that the subroutine call will take too long. And functions should be used instead of macros where possible because they're a lot easier to read and debug and the compiler can type-check arguments. The main reason for "inline" these days is to avoid a warning that you've defined a function and not used it (because when you put a static function in a header file, many programs that include that header file won't use it). Compilers variously have "inline", "__inline", "__inline__", and nothing at all. "__inline" and "__inline__" are sometimes macros and sometimes native keywords. The best thing to do is to put a mess of ifdefs in a central file that define __inline__ as the appropriate keyword or undefined, and use __inline__ in functions that are defined in header files. I don't know GNU Autoconf, but I presume it provides a service to help with that. -- Bryan Henderson Phone 408-621-2000 San Jose, California