From gniibe at fsij.org Tue Oct 2 07:39:34 2018 From: gniibe at fsij.org (NIIBE Yutaka) Date: Tue, 02 Oct 2018 14:39:34 +0900 Subject: gnupg_reopen_std - descriptor state problem In-Reply-To: References: <707fc580-b13b-4a0b-a1c9-353c7063a00b@fork.pl> <87va7iau9v.fsf@iwagami.gniibe.org> Message-ID: <87in2kaovd.fsf@iwagami.gniibe.org> Marcin Gryszkalis writes: > So I guess you could merge the patch. Thanks a lot for your confirmation. I applied it to 2.2 branch as well as master. In GnuPG, to check fd, fstat had been used for years. (GnuPG 1.4 still uses fstat.) I don't know the reason why we haven't got a bug report until today. Perhaps, it may be unusual case, as you said. -- From wk at gnupg.org Tue Oct 2 17:35:40 2018 From: wk at gnupg.org (Werner Koch) Date: Tue, 02 Oct 2018 17:35:40 +0200 Subject: export-drop-uids Message-ID: <87h8i4744z.fsf@wheatstone.g10code.de> Hi! Recent problems with the keyserver network may require that we rethink how we can use the keyservers in another way. To help evaluating such other ways I added two utility features to gpg in master: --export-options has the new sub-option: export-drop-uids Do no export any user id or attribute packets or their associates signatures. Note that due to missing user ids the resulting output is not strictly RFC-4880 compliant. and --import-options has the new sub-option: import-drop-uids Do not import any user ids or their binding signatures. This option can be used to update only the subkeys or other non-user id related information. One idea is to use the new export option to send keys stripped of all user ids and their self- and key-signatures. to the servers. The servers would then not anymore carry mail addresses or other personal information [1], However it should still be possible to query the keyservers for revocation certificates and new subkeys. User-ids will be kept away from them as well as key-signatures. The complete key including the user ids and maybe key-signatures can still be queried by other means (e.g. Web Key Directory). When creating a new key the user could upload the entire key to the Web Key Directory and use the new export option to upload the key sans user ids to the keyservers. Sending encrypted mail will work because the peer's key can be looked up by mail address in the mail provider's directory. Checking for revocations works by uploading a revocation to the keyservers and due to regular checking the keyservers for updates. Subkey rollover works by uploading the updated key to the keyserver (sans user id). Verifying signatures can be made to work because the key can be looked up by fingerprint from the keyservers. However, it does not identify the user - this can however be done on user demand by looking up the original key via the From address at the Web Key Directory. The new import option would be useful to avoid importing unwanted key-signatures from a keyserver, If such a scheme turns out to be useful these options can be added to the keyserver related commands. Problems right now are that gpg will reject encrypting to a key without a user-id, even so that the key has been imported. I have not yet tested whether the keyservers are able to handle keys without user-ids. We also need to create direct key signatures to convey properties of the key even without user ids. And well, --search-keys will not be useful anymore because keyservers won't carry user ids anymore. A slightly different approach would be to use dummy user ids and a non-keyserver-uploaded mapping of these dummy-user id to the real user ids. This has the problem of complexity and that it will be easy to test whether a certain user id can be mapped to such a dummy user id and a key. Salam-Shalom, Werner [1] According to some interpretation of data protection rules. -- Die Gedanken sind frei. Ausnahmen regelt ein Bundesgesetz. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 227 bytes Desc: not available URL: From wk at gnupg.org Thu Oct 4 12:43:14 2018 From: wk at gnupg.org (Werner Koch) Date: Thu, 04 Oct 2018 12:43:14 +0200 Subject: export-drop-uids In-Reply-To: <87h8i4744z.fsf@wheatstone.g10code.de> (Werner Koch's message of "Tue, 02 Oct 2018 17:35:40 +0200") References: <87h8i4744z.fsf@wheatstone.g10code.de> Message-ID: <87va6iyou5.fsf@wheatstone.g10code.de> Hi! For those not ware of it: You can put export and import options also into the keyserver options, like gpg --keyserver-options export-drop-uids --send-keys KEYIDS The SKS keyservers don't fail on sending keys without a user id but it is currently not possible to retrieve them. FWIW, I uploaded this key pub rsa3072 2018-09-10 [SCEA] 8888840D56AB5E1574A8CEA5526C1080EC5EA42F sub rsa3072 2018-09-10 [E] sub rsa1024 2018-10-02 [S] Shalom-Salam, Werner -- Die Gedanken sind frei. Ausnahmen regelt ein Bundesgesetz. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 227 bytes Desc: not available URL: From kristian.fiskerstrand at sumptuouscapital.com Thu Oct 4 13:02:57 2018 From: kristian.fiskerstrand at sumptuouscapital.com (Kristian Fiskerstrand) Date: Thu, 4 Oct 2018 13:02:57 +0200 Subject: export-drop-uids In-Reply-To: <87va6iyou5.fsf@wheatstone.g10code.de> References: <87h8i4744z.fsf@wheatstone.g10code.de> <87va6iyou5.fsf@wheatstone.g10code.de> Message-ID: On 10/4/18 12:43 PM, Werner Koch wrote: > The SKS keyservers don't fail on sending keys without a user id but it > is currently not possible to retrieve them. FWIW, I uploaded this key It could actually be used with existing network if a non-cryptographically correct UID was used as a placeholder, just with nodata style .. A bit time constrained atm but will look into things a bit more ahead of the october meeting so that we have something more to discuss then. -- ---------------------------- Kristian Fiskerstrand Blog: https://blog.sumptuouscapital.com Twitter: @krifisk ---------------------------- Public OpenPGP keyblock at hkp://pool.sks-keyservers.net fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3 ---------------------------- Aurum est Potestas Gold is power -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From wk at gnupg.org Fri Oct 5 12:08:41 2018 From: wk at gnupg.org (Werner Koch) Date: Fri, 05 Oct 2018 12:08:41 +0200 Subject: Libgcrypt Android Compatibility In-Reply-To: (Md Monjur Alam's message of "Mon, 1 Oct 2018 23:10:10 +0000") References: Message-ID: <87lg7cyac6.fsf@wheatstone.g10code.de> On Tue, 2 Oct 2018 01:10, md.monjur at gatech.edu said: > I Can build the libgcrypt library at ARM core embedded device (with > Linux OS). I want the same with Android OS. Does libgcrypt support for > Android? Is there any tool-chain to produce libgcrypt.so which can run Yes. However, for build instructions I would suggest to leap over to https://guardianproject.info who ported GnuPG and thus also Libgcrypt to Android. All their changes should be upstream despite that their GnuPG project is currently unmaintained. You should still find information on how to setup a tool chain and build gnupg and thus Libgcrypt. If you have any build problem using the current Libgcrypt replease please report here and we will try to help. Shalom-Salam, Werner -- Die Gedanken sind frei. Ausnahmen regelt ein Bundesgesetz. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 227 bytes Desc: not available URL: From alon.barlev at gmail.com Sat Oct 6 11:07:21 2018 From: alon.barlev at gmail.com (Alon Bar-Lev) Date: Sat, 6 Oct 2018 12:07:21 +0300 Subject: SYSROOT vs --with-xxx-prefix Message-ID: Hi, gnupg packages require xxx-config scripts instead of using pkg-config for gnupg project in-house dependencies, I am unsure what the reason is, but it makes integration with gnupg a bit harder for downstream. In recent years gnupg projects already use pkg-config for 3rd party dependencies so I guess it is not taboo these days. In any case, in order to configure the in-house dependencies there are three options: 1. Specify XXX_CONFIG with location of each individual config script. 2. Specify --with-xxx-prefix with prefix of bin/xxx-config script per each dependency. 3. Specify SYSROOT as global --with-xxx-prefix. When performing cross-compile all the dependencies should be re-configured, these that use pkg-config are simple, using SYSROOT for the in-house dependencies would have been nice. However, SYSROOT is not supported by all m4 macros of in-house dependencies. In addition, the SYSROOT searches config only at SYSROOT/bin and not SYSROOT/usr/bin making it harder to integrate with the sysroot concept of the cross compiler. Summary 1. Farfetched... can we support pkg-config? I can help to support this in parallel to the existing mechanism. 2. Can you please add SYSROOT support for all dependencies? (npth, ksba, assuan) 3. Can you please search in both SYSROOT/bin and SYSROOT/usr/bin for config script? Thanks, Alon From gniibe at fsij.org Mon Oct 8 02:39:57 2018 From: gniibe at fsij.org (NIIBE Yutaka) Date: Mon, 08 Oct 2018 09:39:57 +0900 Subject: SYSROOT vs --with-xxx-prefix In-Reply-To: References: Message-ID: <87in2djmoy.fsf@iwagami.gniibe.org> Hello, Alon Bar-Lev wrote: > 1. Farfetched... can we support pkg-config? I can help to support this > in parallel to the existing mechanism. Thank you. Somehow, we have started into this way (from libgpg-error). I put some effort providing gpg-error.pc (as well as the gpg-error-config script) in the master branch of libgpg-error. My plan is to extend it to other libraries. Please have a look. Please note that we need to keep the requirment of GnuPG build, as small as possible. GnuPG build should not require pkg-config. This is (still) important for older Unixen. -- From alon.barlev at gmail.com Mon Oct 8 06:36:20 2018 From: alon.barlev at gmail.com (Alon Bar-Lev) Date: Mon, 8 Oct 2018 07:36:20 +0300 Subject: SYSROOT vs --with-xxx-prefix In-Reply-To: <87in2djmoy.fsf@iwagami.gniibe.org> References: <87in2djmoy.fsf@iwagami.gniibe.org> Message-ID: On Mon, Oct 8, 2018 at 3:40 AM NIIBE Yutaka wrote: > > Hello, > > Alon Bar-Lev wrote: > > 1. Farfetched... can we support pkg-config? I can help to support this > > in parallel to the existing mechanism. > > Thank you. > > Somehow, we have started into this way (from libgpg-error). I put some > effort providing gpg-error.pc (as well as the gpg-error-config script) > in the master branch of libgpg-error. My plan is to extend it to other > libraries. Please have a look. OK, so while we are at this... the config script should also support SYSROOT and output directories relevant to SYSROOT. For example, when cross compling and acquiring CFLAGS or LDFLAGS outputting -I/usr/include and -L/usr/lib will not allow to build. Three tasks to help cross compile work: 1. *.m4: complete supporting SYSROOT to look for the *-config script within that prefix. 2. *.m4: support both SYSROOT/bin and SYSROOT/usr/bin when looking for *-config scripts. 3. *-config: add support of SYSROOT to output directories > Please note that we need to keep the requirment of GnuPG build, as small > as possible. GnuPG build should not require pkg-config. This is > (still) important for older Unixen. I do not think there is a problem to try pkg-config and fallback to *-config in autoconf, and in any case provide pkg-config for consumers that do not have this concern. Thanks! From wk at gnupg.org Mon Oct 8 15:47:12 2018 From: wk at gnupg.org (Werner Koch) Date: Mon, 08 Oct 2018 15:47:12 +0200 Subject: SYSROOT vs --with-xxx-prefix In-Reply-To: (Alon Bar-Lev's message of "Sat, 6 Oct 2018 12:07:21 +0300") References: Message-ID: <87lg78wnxb.fsf@wheatstone.g10code.de> On Sat, 6 Oct 2018 11:07, alon.barlev at gmail.com said: > 3. Specify SYSROOT as global --with-xxx-prefix. I actually liked that and iirc implemented it for one or two packages. > 1. Farfetched... can we support pkg-config? I can help to support this > in parallel to the existing mechanism. Gniibe alreadt started with this but pkg-config will only be the second choice. > 2. Can you please add SYSROOT support for all dependencies? (npth, ksba, assuan) Yes. Makes a lot of sense and avoid surprises. > 3. Can you please search in both SYSROOT/bin and SYSROOT/usr/bin for > config script? Yes. Would you be so kind and open a feature request? Salam-Shalom, Werner -- Die Gedanken sind frei. Ausnahmen regelt ein Bundesgesetz. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 227 bytes Desc: not available URL: From wk at gnupg.org Tue Oct 9 09:04:09 2018 From: wk at gnupg.org (Werner Koch) Date: Tue, 09 Oct 2018 09:04:09 +0200 Subject: [Announce] GnuPG Made Easy (GPGME) 1.12.0 released Message-ID: <87a7nnvbx2.fsf@wheatstone.g10code.de> Hello! We are pleased to announce version 1.12.0 of GPGME. GnuPG Made Easy (GPGME) is a C language library that allows to add support for cryptography to a program. It is designed to make access to public key crypto engines like gpg and gpgsm easier for applications. GPGME provides a high-level crypto API for encryption, decryption, signing, signature verification, and key management. GPGME comes with language bindings for Common Lisp, C++, QT, Python 2 and 3. See https://gnupg.org/software/gpgme for more. Noteworthy changes in version 1.12.0 ==================================== * Enhanced the JSON based interface tool gpgme-json to support Native Messaging as well as new Javascript code to support the browser site. See lang/js/README for details. * Major overhaul of the Python language bindings documentation. * Even for old versions of gpg a missing MDC will now lead to a decryption failure. * Added context flag "auto-key-locate" to control the behavior of GPGME_KEYLIST_MODE_LOCATE. * New data function to create a data object from an estream. * Add more interfaces to the C++ bindings. * Improved error codes on decryption failure. * Lots of minor fixes. * Interface changes relative to the 1.11.1 release: gpgme_data_new_from_estream NEW. gpgme_decrypt_result_t EXTENDED: New field legacy_cipher_nomdc. gpgme_set_ctx_flag EXTENDED: New flag 'ignore-mdc-error'. GPGME_AUDITLOG_DEFAULT NEW. GPGME_AUDITLOG_DIAG NEW. gpgme_set_ctx_flag EXTENDED: New flag 'auto-key-locate'. cpp: DecryptionResult::sessionKey NEW. cpp: DecryptionResult::symkeyAlgo NEW. cpp: DecryptionResult::isLegacyCipherNoMDC NEW. cpp: Data::rewind NEW. cpp: Context::setFlag NEW. cpp: Context::getFlag NEW. cpp: Context::createKeyEx NEW. Release-info: https://dev.gnupg.org/T4109 Download ======== You may download this library and its OpenPGP signature from: https://gnupg.org/ftp/gcrypt/gpgme/gpgme-1.12.0.tar.bz2 (1619k) https://gnupg.org/ftp/gcrypt/gpgme/gpgme-1.12.0.tar.bz2.sig or from ftp.gnupg.org. The SHA-1 checksum is 6f1828fcd7de4366ca063e57f35e4ab24bc91baf gpgme-1.12.0.tar.bz2 but you better check the integrity using the provided signature. See for details. Thanks ====== Maintenance and development of GnuPG is mostly financed by donations. The GnuPG project currently employs one full-time developer and two contractors. All work exclusively on GnuPG and closely related software like Libgcrypt and GPGME. We have to thank all the people who helped the GnuPG project, be it testing, coding, translating, suggesting, auditing, administering the servers, spreading the word, and answering questions on the mailing lists. Many thanks to our numerous financial supporters, both corporate and individuals. Without you it would not be possible to keep GnuPG in a good shape and address all the small and larger requests made by our users. Thanks. Happy hacking, Your GnuPG hackers p.s. This is an announcement only mailing list. Please send replies only to the gnupg-devel 'at' gnupg.org mailing list. p.p.s List of Release Signing Keys: To guarantee that a downloaded GnuPG version has not been tampered by malicious entities we provide signature files for all tarballs and binary versions. The keys are also signed by the long term keys of their respective owners. Current releases are signed by one or more of these four keys: rsa2048 2011-01-12 [expires: 2019-12-31] Key fingerprint = D869 2123 C406 5DEA 5E0F 3AB5 249B 39D2 4F25 E3B6 Werner Koch (dist sig) rsa2048 2014-10-29 [expires: 2019-12-31] Key fingerprint = 46CC 7308 65BB 5C78 EBAB ADCF 0437 6F3E E085 6959 David Shaw (GnuPG Release Signing Key) rsa2048 2014-10-29 [expires: 2020-10-30] Key fingerprint = 031E C253 6E58 0D8E A286 A9F2 2071 B08A 33BD 3F06 NIIBE Yutaka (GnuPG Release Key) rsa3072 2017-03-17 [expires: 2027-03-15] Key fingerprint = 5B80 C575 4298 F0CB 55D8 ED6A BCEF 7E29 4B09 2E28 Andre Heinecke (Release Signing Key) The keys are available at and in any recently released GnuPG tarball in the file g10/distsigkey.gpg . Note that this mail has been signed by a different key. -- Die Gedanken sind frei. Ausnahmen regelt ein Bundesgesetz. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 227 bytes Desc: not available URL: -------------- next part -------------- _______________________________________________ Gnupg-announce mailing list Gnupg-announce at gnupg.org http://lists.gnupg.org/mailman/listinfo/gnupg-announce From alon.barlev at gmail.com Tue Oct 9 22:53:33 2018 From: alon.barlev at gmail.com (Alon Bar-Lev) Date: Tue, 9 Oct 2018 23:53:33 +0300 Subject: SYSROOT vs --with-xxx-prefix In-Reply-To: <87lg78wnxb.fsf@wheatstone.g10code.de> References: <87lg78wnxb.fsf@wheatstone.g10code.de> Message-ID: On Mon, Oct 8, 2018 at 4:59 PM Werner Koch wrote: > > On Sat, 6 Oct 2018 11:07, alon.barlev at gmail.com said: > > > 3. Specify SYSROOT as global --with-xxx-prefix. > > I actually liked that and iirc implemented it for one or two packages. > > > 1. Farfetched... can we support pkg-config? I can help to support this > > in parallel to the existing mechanism. > > Gniibe alreadt started with this but pkg-config will only be the second > choice. > > > 2. Can you please add SYSROOT support for all dependencies? (npth, ksba, assuan) > > Yes. Makes a lot of sense and avoid surprises. > > > 3. Can you please search in both SYSROOT/bin and SYSROOT/usr/bin for > > config script? > > Yes. > > Would you be so kind and open a feature request? Hi, I was about to do that, but first checked out the master of libgpg-error and found out that you already support pkg-config and converted the config script to delegate calls to the pkg-config. There are some gaps that can be solved. Then the same can be replicated to all other projects. I created a patch[1] with the required fixes, can you please review? I added some comments to describe reasoning. Most of the changes in the gpg-error.m4 are indent changes, please ignore them. The logic is simple, if a prefix was not provided explicitly and the config script was not provided explicitly, try autodetection using pkg-config before fallback of config script detection. I checked this briefly and it works for native, multilib and cross compile. Regards, Alon [1] https://github.com/alonbl/libgpg-error/pull/1/files From gniibe at fsij.org Wed Oct 10 04:18:29 2018 From: gniibe at fsij.org (NIIBE Yutaka) Date: Wed, 10 Oct 2018 11:18:29 +0900 Subject: SYSROOT vs --with-xxx-prefix In-Reply-To: References: <87lg78wnxb.fsf@wheatstone.g10code.de> Message-ID: <87va6alf2i.fsf@iwagami.gniibe.org> Hello, Thanks for yor try. Please have a lock at gnupg/doc/HACKING [0], and submit your change to dev.gnupg.org or this maling list. Before your submission, I'd suggest to minimize your change (e.g. don't include indentation change). Alon Bar-Lev wrote: > libgpg-error and found out that you already support pkg-config and > converted the config script to delegate calls to the pkg-config. No, the config script (gpg-error-config) never delegates to pkg-config. It *does* handle the gpg-error.pc file by itself. We share the gpg-error.pc file between gpg-error-config and pkg-config, and I'd like to keep deviation by gpg-error-config as small as possible. Well, I repeat: GnuPG build process never depends on pkg-config. > There are some gaps that can be solved. Then the same can be > replicated to all other projects. Today, I push two changes: gpg-error-config: Fix the place of *.pc (for multilib). 9f71b28dcb38e1d5d9001692e2f64009396aaf9b gpg-error-config: Add PKG_CONFIG_SYSROOT_DIR support. 6167f3c461a4e53ccc5af620cdbfa28bfa9234f5 The first one is what you pointed out. The second one is supporting cross build. [0] https://dev.gnupg.org/source/gnupg/browse/master/doc/HACKING -- From alon.barlev at gmail.com Wed Oct 10 06:51:31 2018 From: alon.barlev at gmail.com (Alon Bar-Lev) Date: Wed, 10 Oct 2018 07:51:31 +0300 Subject: SYSROOT vs --with-xxx-prefix In-Reply-To: <87va6alf2i.fsf@iwagami.gniibe.org> References: <87lg78wnxb.fsf@wheatstone.g10code.de> <87va6alf2i.fsf@iwagami.gniibe.org> Message-ID: On Wed, Oct 10, 2018 at 5:18 AM NIIBE Yutaka wrote: > > Hello, > > Thanks for yor try. Please have a lock at gnupg/doc/HACKING [0], > and submit your change to dev.gnupg.org or this maling list. > Before your submission, I'd suggest to minimize your change > (e.g. don't include indentation change). > > Alon Bar-Lev wrote: > > libgpg-error and found out that you already support pkg-config and > > converted the config script to delegate calls to the pkg-config. > > No, the config script (gpg-error-config) never delegates to pkg-config. > It *does* handle the gpg-error.pc file by itself. > We share the gpg-error.pc file between gpg-error-config and pkg-config, > and I'd like to keep deviation by gpg-error-config as small as possible. Yes, I should have phrased it better, it is great you begin support other libraries to use pkg-config, this is a great change in spirit. > Well, I repeat: GnuPG build process never depends on pkg-config. This is not entirely true you do have projects that use pkg-config, just not via the pkg.m4 macro for some reason. Providing pkg-config fixes issues for all packages outside of the gnupg project - that's great. However, the entire problem is that the gnupg project packages continue to use these proprietary config script and try to search for them during build, this tends to break, for example: 1. configure script cannot find these in path when building cross compile, autoconf shoud be modified to explicitly look for these at SYSROOT/bin:SYSROOT/usr/bin 2. multilib cannot use a single script, you need to install these scripts as HOST-xxx-config and use AC_PATH_TOOL instead of AC_PATH_RPOG However, using pkg-config do not have these issues, so using it enables you to bypass these issues without fixing them. I provided a clean way to use pkg-config if available and if not fallback to current proprietary mechanism, it takes nothing from the existing process, if user do not explicitly ask for config script and pkg-config is not available on system the current process remains. Will you feel better if we have --enable-pkg-config in autoconf disabled by default, and can be enabled by downstream that support pkg-config? Logic: --enable-pkg-config - use pkg-config instead of proprietary mechanism, this is as simple as and for sure do not take anything from anyone if not explicitly enabled: if test x"${enable_pkg_config}" = xyes; then PKG_CHECK_MODULES([GPG_ERROR], [gpg-error >= $min_gpg_error_version], , [enable_pkg_config=no]) PKG_CHECK_VAR([GPG_ERROR_MT_CFLAGS], [gpg-error], [mtcflags]) PKG_CHECK_VAR([GPG_ERROR_MT_LIBS], [gpg-error], [mtlibs]) fi if test x"${enable_pkg_config}" = xno; then # current code fi > > There are some gaps that can be solved. Then the same can be > > replicated to all other projects. > > Today, I push two changes: > > gpg-error-config: Fix the place of *.pc (for multilib). > 9f71b28dcb38e1d5d9001692e2f64009396aaf9b > > gpg-error-config: Add PKG_CONFIG_SYSROOT_DIR support. > 6167f3c461a4e53ccc5af620cdbfa28bfa9234f5 > > The first one is what you pointed out. > The second one is supporting cross build. Thanks! But this is not enough as long as these config scripts are being used. Alon From alon.barlev at gmail.com Wed Oct 10 08:00:37 2018 From: alon.barlev at gmail.com (Alon Bar-Lev) Date: Wed, 10 Oct 2018 09:00:37 +0300 Subject: SYSROOT vs --with-xxx-prefix In-Reply-To: References: <87lg78wnxb.fsf@wheatstone.g10code.de> <87va6alf2i.fsf@iwagami.gniibe.org> Message-ID: On Wed, Oct 10, 2018 at 7:51 AM Alon Bar-Lev wrote: > > On Wed, Oct 10, 2018 at 5:18 AM NIIBE Yutaka wrote: > > > > Hello, > > > > Thanks for yor try. Please have a lock at gnupg/doc/HACKING [0], > > and submit your change to dev.gnupg.org or this maling list. > > Before your submission, I'd suggest to minimize your change > > (e.g. don't include indentation change). > > > > Alon Bar-Lev wrote: > > > libgpg-error and found out that you already support pkg-config and > > > converted the config script to delegate calls to the pkg-config. > > > > No, the config script (gpg-error-config) never delegates to pkg-config. > > It *does* handle the gpg-error.pc file by itself. > > We share the gpg-error.pc file between gpg-error-config and pkg-config, > > and I'd like to keep deviation by gpg-error-config as small as possible. > > Yes, I should have phrased it better, it is great you begin support > other libraries to use pkg-config, this is a great change in spirit. > > > Well, I repeat: GnuPG build process never depends on pkg-config. > > This is not entirely true you do have projects that use pkg-config, > just not via the pkg.m4 macro for some reason. > Providing pkg-config fixes issues for all packages outside of the > gnupg project - that's great. > However, the entire problem is that the gnupg project packages > continue to use these proprietary config script and try to search for > them during build, this tends to break, for example: > 1. configure script cannot find these in path when building cross > compile, autoconf shoud be modified to explicitly look for these at > SYSROOT/bin:SYSROOT/usr/bin > 2. multilib cannot use a single script, you need to install these > scripts as HOST-xxx-config and use AC_PATH_TOOL instead of > AC_PATH_RPOG > > However, using pkg-config do not have these issues, so using it > enables you to bypass these issues without fixing them. > I provided a clean way to use pkg-config if available and if not > fallback to current proprietary mechanism, it takes nothing from the > existing process, if user do not explicitly ask for config script and > pkg-config is not available on system the current process remains. > > Will you feel better if we have --enable-pkg-config in autoconf > disabled by default, and can be enabled by downstream that support > pkg-config? > Logic: --enable-pkg-config - use pkg-config instead of proprietary > mechanism, this is as simple as and for sure do not take anything from > anyone if not explicitly enabled: > > if test x"${enable_pkg_config}" = xyes; then > PKG_CHECK_MODULES([GPG_ERROR], [gpg-error >= > $min_gpg_error_version], , [enable_pkg_config=no]) > PKG_CHECK_VAR([GPG_ERROR_MT_CFLAGS], [gpg-error], [mtcflags]) > PKG_CHECK_VAR([GPG_ERROR_MT_LIBS], [gpg-error], [mtlibs]) > fi > if test x"${enable_pkg_config}" = xno; then > # current code > fi Patch is available[1]. BTW: I am unsure that you aware aware, but when using pkg.m4 and these PKG_CHECK_* macros, if the pkg-config utility is not installed, it simply execute the if-not-found clause, so this does not introduce hard dependency, it simply falls back to the current logic. [1] https://github.com/alonbl/libgpg-error/pull/1/files?utf8=%E2%9C%93&diff=unified&w=1 > > > > There are some gaps that can be solved. Then the same can be > > > replicated to all other projects. > > > > Today, I push two changes: > > > > gpg-error-config: Fix the place of *.pc (for multilib). > > 9f71b28dcb38e1d5d9001692e2f64009396aaf9b > > > > gpg-error-config: Add PKG_CONFIG_SYSROOT_DIR support. > > 6167f3c461a4e53ccc5af620cdbfa28bfa9234f5 > > > > The first one is what you pointed out. > > The second one is supporting cross build. > > Thanks! > But this is not enough as long as these config scripts are being used. > Alon From glv at posteo.net Wed Oct 10 16:43:46 2018 From: glv at posteo.net (Guillaume LE VAILLANT) Date: Wed, 10 Oct 2018 16:43:46 +0200 Subject: [PATCH] GPGME: several fixes for Common Lisp bindings Message-ID: <20181010164229.334810d0@yamatai> Hello, Here's a patch (in attachment) updating the Common Lisp bindings for GPGME to make them work with recent versions of CFFI. With this patch I was able to use op-encrypt, op-decrypt, op-sign, op-verify and op-export successfully with SBCL, CCL, ECL and ABCL. -------------- next part -------------- A non-text attachment was scrubbed... Name: gpgme-cl-bindings.patch Type: text/x-patch Size: 35544 bytes Desc: not available URL: From glv at posteo.net Wed Oct 10 19:37:41 2018 From: glv at posteo.net (Guillaume LE VAILLANT) Date: Wed, 10 Oct 2018 19:37:41 +0200 Subject: [PATCH] GPGME: several fixes for Common Lisp bindings In-Reply-To: <20181010164229.334810d0@yamatai> References: <20181010164229.334810d0@yamatai> Message-ID: <20181010193741.233e09a4@yamatai> I just saw that my patch is missing the following: commit 22f38f932f6340a63d97bd1dd09feff64c2f2167 tree 6ceb3e3f2fa7e470fa4aab3af1f6c977331de4bc parent bd784a6763113ffbcbec1bba40b737610d680b16 author Guillaume LE VAILLANT 1539192657 +0200 committer Guillaume LE VAILLANT 1539192657 +0200 gpgsig -----BEGIN PGP SIGNATURE----- iIQEABYKAC0WIQQkUwKxurH4Z/3KlryPP4Yfgut6mgUCW743Vw8cZ2x2QHBvc3Rl by5uZXQACgkQjz+GH4Lreprs0wD4tQrVIyVy6V8E65nj2oS5X30sIZz5Dv3fbExT 4JmyewEAyRMLnUPuty3QJ3m3FpzuC0Ro/i68Uo6xw8Cv2KRpCwg= =ihQf -----END PGP SIGNATURE----- Common Lisp bindings: add gpgme-grovel.lisp to Makefile Signed-off-by: Guillaume LE VAILLANT diff --git a/lang/cl/Makefile.am b/lang/cl/Makefile.am index 553926e2..dee07119 100644 --- a/lang/cl/Makefile.am +++ b/lang/cl/Makefile.am @@ -18,7 +18,7 @@ # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA # 02111-1307, USA -clfiles = gpgme.asd gpgme-package.lisp gpgme.lisp +clfiles = gpgme.asd gpgme-package.lisp gpgme-grovel.lisp gpgme.lisp # FIXME: Should be configurable. clfilesdir = $(datadir)/common-lisp/source/gpgme From glv at posteo.net Thu Oct 11 11:41:14 2018 From: glv at posteo.net (Guillaume LE VAILLANT) Date: Thu, 11 Oct 2018 11:41:14 +0200 Subject: [PATCH] GPGME: several fixes for Common Lisp bindings In-Reply-To: <20181010193741.233e09a4@yamatai> References: <20181010164229.334810d0@yamatai> <20181010193741.233e09a4@yamatai> Message-ID: <20181011113825.76f9752a@yamatai> GPGME Developer's Certificate of Origin. Version 1.0 ===================================================== By making a contribution to the GPGME project, I certify that: (a) The contribution was created in whole or in part by me and I have the right to submit it under the free software license indicated in the file; or (b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate free software license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same free software license (unless I am permitted to submit under a different license), as indicated in the file; or (c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it. (d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the free software license(s) involved. Signed-off-by: Guillaume LE VAILLANT -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 248 bytes Desc: Signature digitale OpenPGP URL: From alon.barlev at gmail.com Thu Oct 11 20:34:37 2018 From: alon.barlev at gmail.com (Alon Bar-Lev) Date: Thu, 11 Oct 2018 21:34:37 +0300 Subject: [PATCH Libgpg-error] gpg-error.m4: support pkg-config In-Reply-To: <20181011182720.11991-1-alon.barlev@gmail.com> References: <20181011182720.11991-1-alon.barlev@gmail.com> Message-ID: Hi, This patch is mostly about adding indent, please review this online[1] to see how simple it is. I must say that I am truly excited, I waited many years (at least 10) to be able to use gnupg packages in standard way and reduce the effort of downstream maintenance, I truly hope we can push this forward. Thanks! Alon [1] https://github.com/alonbl/libgpg-error/pull/1/files?utf8=%E2%9C%93&diff=unified&w=1 From alon.barlev at gmail.com Thu Oct 11 20:27:20 2018 From: alon.barlev at gmail.com (Alon Bar-Lev) Date: Thu, 11 Oct 2018 21:27:20 +0300 Subject: [PATCH Libgpg-error] gpg-error.m4: support pkg-config Message-ID: <20181011182720.11991-1-alon.barlev@gmail.com> * m4/pkg.m4: New. * src/gpg-error.m4: use pkg-config if enabled. -- libgpg-error installs pkg-config configuration file, it even become the source of truth for the gpg-error-config script. All gpg-error.m4 consumers can leverage pkg-config as well. In order to keep backward compatibility, the pkg-config variant will be used only if enable_pkg_config is set to yes. if pkg-config not found or pkg-config configuration is missing it will fallback to current implementation. Apart of being a standard/popular method, pkg-config has advantages over using executable config scripts: * Simplicity, built for the task. * Good integration into autoconf using the pkg.m4 macros, enables the detection of pkg-config, fallback if missing and handle the interaction, including override settings by environment. * Support for multilib configurations as configuration is installed per library setup, unlike a common script that is installed at bindir. * Cross-compile support, support sysroot and does not require to execute anything from sysroot. Signed-off-by: Alon Bar-Lev --- m4/pkg.m4 | 343 +++++++++++++++++++++++++++++++++++++++++++++++ src/gpg-error.m4 | 149 +++++++++++--------- 2 files changed, 426 insertions(+), 66 deletions(-) create mode 100644 m4/pkg.m4 diff --git a/src/gpg-error.m4 b/src/gpg-error.m4 index 0564219..d3a0ad7 100644 --- a/src/gpg-error.m4 +++ b/src/gpg-error.m4 @@ -26,6 +26,7 @@ dnl is added to the gpg_config_script_warn variable. dnl AC_DEFUN([AM_PATH_GPG_ERROR], [ AC_REQUIRE([AC_CANONICAL_HOST]) + AC_REQUIRE([PKG_PROG_PKG_CONFIG]) gpg_error_config_prefix="" dnl --with-libgpg-error-prefix=PFX is the preferred name for this option, dnl since that is consistent with how our three siblings use the directory/ @@ -40,64 +41,79 @@ AC_DEFUN([AM_PATH_GPG_ERROR], dnl but do not document this old, inconsistently-named option. AC_ARG_WITH(gpg-error-prefix,, [gpg_error_config_prefix="$withval"]) + min_gpg_error_version=ifelse([$1], ,0.0,$1) - if test x"${GPG_ERROR_CONFIG}" = x ; then - if test x"${gpg_error_config_prefix}" != x ; then - GPG_ERROR_CONFIG="${gpg_error_config_prefix}/bin/gpg-error-config" - else - case "${SYSROOT}" in - /*) - if test -x "${SYSROOT}/bin/gpg-error-config" ; then - GPG_ERROR_CONFIG="${SYSROOT}/bin/gpg-error-config" - fi - ;; - '') - ;; - *) - AC_MSG_WARN([Ignoring \$SYSROOT as it is not an absolute path.]) - ;; - esac - fi + gpg_error_use_config=yes + if test x"${enable_pkg_config}" = xyes; then + PKG_CHECK_MODULES( + [GPG_ERROR], + [gpg-error >= $min_gpg_error_version], + [ + gpg_error_use_config=no + PKG_CHECK_VAR([GPG_ERROR_MT_CFLAGS], [gpg-error], [mtcflags]) + PKG_CHECK_VAR([GPG_ERROR_MT_LIBS], [gpg-error], [mtlibs]) + ], + [:] + ) fi - AC_PATH_PROG(GPG_ERROR_CONFIG, gpg-error-config, no) - min_gpg_error_version=ifelse([$1], ,0.0,$1) - AC_MSG_CHECKING(for GPG Error - version >= $min_gpg_error_version) - ok=no - if test "$GPG_ERROR_CONFIG" != "no" \ - && test -f "$GPG_ERROR_CONFIG" ; then - req_major=`echo $min_gpg_error_version | \ - sed 's/\([[0-9]]*\)\.\([[0-9]]*\)/\1/'` - req_minor=`echo $min_gpg_error_version | \ - sed 's/\([[0-9]]*\)\.\([[0-9]]*\)/\2/'` - gpg_error_config_version=`$GPG_ERROR_CONFIG $gpg_error_config_args --version` - major=`echo $gpg_error_config_version | \ - sed 's/\([[0-9]]*\)\.\([[0-9]]*\).*/\1/'` - minor=`echo $gpg_error_config_version | \ - sed 's/\([[0-9]]*\)\.\([[0-9]]*\).*/\2/'` - if test "$major" -gt "$req_major"; then - ok=yes - else - if test "$major" -eq "$req_major"; then - if test "$minor" -ge "$req_minor"; then - ok=yes - fi - fi + if test x"${gpg_error_use_config}" = xyes; then + if test x"${GPG_ERROR_CONFIG}" = x ; then + if test x"${gpg_error_config_prefix}" != x ; then + GPG_ERROR_CONFIG="${gpg_error_config_prefix}/bin/gpg-error-config" + else + case "${SYSROOT}" in + /*) + if test -x "${SYSROOT}/bin/gpg-error-config" ; then + GPG_ERROR_CONFIG="${SYSROOT}/bin/gpg-error-config" + fi + ;; + '') + ;; + *) + AC_MSG_WARN([Ignoring \$SYSROOT as it is not an absolute path.]) + ;; + esac + fi fi - fi - if test $ok = yes; then - GPG_ERROR_CFLAGS=`$GPG_ERROR_CONFIG $gpg_error_config_args --cflags` - GPG_ERROR_LIBS=`$GPG_ERROR_CONFIG $gpg_error_config_args --libs` - GPG_ERROR_MT_CFLAGS=`$GPG_ERROR_CONFIG $gpg_error_config_args --variable=mtcflags 2>/dev/null` - GPG_ERROR_MT_CFLAGS="$GPG_ERROR_CFLAGS${GPG_ERROR_CFLAGS:+ }$GPG_ERROR_MT_CFLAGS" - GPG_ERROR_MT_LIBS=`$GPG_ERROR_CONFIG $gpg_error_config_args --variable=mtlibs 2>/dev/null` - GPG_ERROR_MT_LIBS="$GPG_ERROR_LIBS${GPG_ERROR_LIBS:+ }$GPG_ERROR_MT_LIBS" - AC_MSG_RESULT([yes ($gpg_error_config_version)]) - ifelse([$2], , :, [$2]) - gpg_error_config_host=`$GPG_ERROR_CONFIG $gpg_error_config_args --variable=host 2>/dev/null || echo none` - if test x"$gpg_error_config_host" != xnone ; then - if test x"$gpg_error_config_host" != x"$host" ; then - AC_MSG_WARN([[ + + AC_PATH_PROG(GPG_ERROR_CONFIG, gpg-error-config, no) + AC_MSG_CHECKING(for GPG Error - version >= $min_gpg_error_version) + ok=no + if test "$GPG_ERROR_CONFIG" != "no" \ + && test -f "$GPG_ERROR_CONFIG" ; then + req_major=`echo $min_gpg_error_version | \ + sed 's/\([[0-9]]*\)\.\([[0-9]]*\)/\1/'` + req_minor=`echo $min_gpg_error_version | \ + sed 's/\([[0-9]]*\)\.\([[0-9]]*\)/\2/'` + gpg_error_config_version=`$GPG_ERROR_CONFIG $gpg_error_config_args --version` + major=`echo $gpg_error_config_version | \ + sed 's/\([[0-9]]*\)\.\([[0-9]]*\).*/\1/'` + minor=`echo $gpg_error_config_version | \ + sed 's/\([[0-9]]*\)\.\([[0-9]]*\).*/\2/'` + if test "$major" -gt "$req_major"; then + ok=yes + else + if test "$major" -eq "$req_major"; then + if test "$minor" -ge "$req_minor"; then + ok=yes + fi + fi + fi + fi + if test $ok = yes; then + GPG_ERROR_CFLAGS=`$GPG_ERROR_CONFIG $gpg_error_config_args --cflags` + GPG_ERROR_LIBS=`$GPG_ERROR_CONFIG $gpg_error_config_args --libs` + GPG_ERROR_MT_CFLAGS=`$GPG_ERROR_CONFIG $gpg_error_config_args --variable=mtcflags 2>/dev/null` + GPG_ERROR_MT_CFLAGS="$GPG_ERROR_CFLAGS${GPG_ERROR_CFLAGS:+ }$GPG_ERROR_MT_CFLAGS" + GPG_ERROR_MT_LIBS=`$GPG_ERROR_CONFIG $gpg_error_config_args --variable=mtlibs 2>/dev/null` + GPG_ERROR_MT_LIBS="$GPG_ERROR_LIBS${GPG_ERROR_LIBS:+ }$GPG_ERROR_MT_LIBS" + AC_MSG_RESULT([yes ($gpg_error_config_version)]) + ifelse([$2], , :, [$2]) + gpg_error_config_host=`$GPG_ERROR_CONFIG $gpg_error_config_args --variable=host 2>/dev/null || echo none` + if test x"$gpg_error_config_host" != xnone ; then + if test x"$gpg_error_config_host" != x"$host" ; then + AC_MSG_WARN([[ *** *** The config script $GPG_ERROR_CONFIG was *** built for $gpg_error_config_host and thus may not match the @@ -105,19 +121,20 @@ AC_DEFUN([AM_PATH_GPG_ERROR], *** You may want to use the configure option --with-libgpg-error-prefix *** to specify a matching config script or use \$SYSROOT. ***]]) - gpg_config_script_warn="$gpg_config_script_warn libgpg-error" + gpg_config_script_warn="$gpg_config_script_warn libgpg-error" + fi fi + else + GPG_ERROR_CFLAGS="" + GPG_ERROR_LIBS="" + GPG_ERROR_MT_CFLAGS="" + GPG_ERROR_MT_LIBS="" + AC_MSG_RESULT(no) + ifelse([$3], , :, [$3]) fi - else - GPG_ERROR_CFLAGS="" - GPG_ERROR_LIBS="" - GPG_ERROR_MT_CFLAGS="" - GPG_ERROR_MT_LIBS="" - AC_MSG_RESULT(no) - ifelse([$3], , :, [$3]) + AC_SUBST(GPG_ERROR_CFLAGS) + AC_SUBST(GPG_ERROR_LIBS) + AC_SUBST(GPG_ERROR_MT_CFLAGS) + AC_SUBST(GPG_ERROR_MT_LIBS) fi - AC_SUBST(GPG_ERROR_CFLAGS) - AC_SUBST(GPG_ERROR_LIBS) - AC_SUBST(GPG_ERROR_MT_CFLAGS) - AC_SUBST(GPG_ERROR_MT_LIBS) ]) diff --git a/m4/pkg.m4 b/m4/pkg.m4 new file mode 100644 index 0000000..ef61d61 --- /dev/null +++ b/m4/pkg.m4 @@ -0,0 +1,343 @@ +# pkg.m4 - Macros to locate and utilise pkg-config. -*- Autoconf -*- +# serial 11 (pkg-config-0.29.1) + +dnl Copyright ? 2004 Scott James Remnant . +dnl Copyright ? 2012-2015 Dan Nicholson +dnl +dnl This program is free software; you can redistribute it and/or modify +dnl it under the terms of the GNU General Public License as published by +dnl the Free Software Foundation; either version 2 of the License, or +dnl (at your option) any later version. +dnl +dnl This program is distributed in the hope that it will be useful, but +dnl WITHOUT ANY WARRANTY; without even the implied warranty of +dnl MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +dnl General Public License for more details. +dnl +dnl You should have received a copy of the GNU General Public License +dnl along with this program; if not, write to the Free Software +dnl Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA +dnl 02111-1307, USA. +dnl +dnl As a special exception to the GNU General Public License, if you +dnl distribute this file as part of a program that contains a +dnl configuration script generated by Autoconf, you may include it under +dnl the same distribution terms that you use for the rest of that +dnl program. + +dnl PKG_PREREQ(MIN-VERSION) +dnl ----------------------- +dnl Since: 0.29 +dnl +dnl Verify that the version of the pkg-config macros are at least +dnl MIN-VERSION. Unlike PKG_PROG_PKG_CONFIG, which checks the user's +dnl installed version of pkg-config, this checks the developer's version +dnl of pkg.m4 when generating configure. +dnl +dnl To ensure that this macro is defined, also add: +dnl m4_ifndef([PKG_PREREQ], +dnl [m4_fatal([must install pkg-config 0.29 or later before running autoconf/autogen])]) +dnl +dnl See the "Since" comment for each macro you use to see what version +dnl of the macros you require. +m4_defun([PKG_PREREQ], +[m4_define([PKG_MACROS_VERSION], [0.29.1]) +m4_if(m4_version_compare(PKG_MACROS_VERSION, [$1]), -1, + [m4_fatal([pkg.m4 version $1 or higher is required but ]PKG_MACROS_VERSION[ found])]) +])dnl PKG_PREREQ + +dnl PKG_PROG_PKG_CONFIG([MIN-VERSION]) +dnl ---------------------------------- +dnl Since: 0.16 +dnl +dnl Search for the pkg-config tool and set the PKG_CONFIG variable to +dnl first found in the path. Checks that the version of pkg-config found +dnl is at least MIN-VERSION. If MIN-VERSION is not specified, 0.9.0 is +dnl used since that's the first version where most current features of +dnl pkg-config existed. +AC_DEFUN([PKG_PROG_PKG_CONFIG], +[m4_pattern_forbid([^_?PKG_[A-Z_]+$]) +m4_pattern_allow([^PKG_CONFIG(_(PATH|LIBDIR|SYSROOT_DIR|ALLOW_SYSTEM_(CFLAGS|LIBS)))?$]) +m4_pattern_allow([^PKG_CONFIG_(DISABLE_UNINSTALLED|TOP_BUILD_DIR|DEBUG_SPEW)$]) +AC_ARG_VAR([PKG_CONFIG], [path to pkg-config utility]) +AC_ARG_VAR([PKG_CONFIG_PATH], [directories to add to pkg-config's search path]) +AC_ARG_VAR([PKG_CONFIG_LIBDIR], [path overriding pkg-config's built-in search path]) + +if test "x$ac_cv_env_PKG_CONFIG_set" != "xset"; then + AC_PATH_TOOL([PKG_CONFIG], [pkg-config]) +fi +if test -n "$PKG_CONFIG"; then + _pkg_min_version=m4_default([$1], [0.9.0]) + AC_MSG_CHECKING([pkg-config is at least version $_pkg_min_version]) + if $PKG_CONFIG --atleast-pkgconfig-version $_pkg_min_version; then + AC_MSG_RESULT([yes]) + else + AC_MSG_RESULT([no]) + PKG_CONFIG="" + fi +fi[]dnl +])dnl PKG_PROG_PKG_CONFIG + +dnl PKG_CHECK_EXISTS(MODULES, [ACTION-IF-FOUND], [ACTION-IF-NOT-FOUND]) +dnl ------------------------------------------------------------------- +dnl Since: 0.18 +dnl +dnl Check to see whether a particular set of modules exists. Similar to +dnl PKG_CHECK_MODULES(), but does not set variables or print errors. +dnl +dnl Please remember that m4 expands AC_REQUIRE([PKG_PROG_PKG_CONFIG]) +dnl only at the first occurence in configure.ac, so if the first place +dnl it's called might be skipped (such as if it is within an "if", you +dnl have to call PKG_CHECK_EXISTS manually +AC_DEFUN([PKG_CHECK_EXISTS], +[AC_REQUIRE([PKG_PROG_PKG_CONFIG])dnl +if test -n "$PKG_CONFIG" && \ + AC_RUN_LOG([$PKG_CONFIG --exists --print-errors "$1"]); then + m4_default([$2], [:]) +m4_ifvaln([$3], [else + $3])dnl +fi]) + +dnl _PKG_CONFIG([VARIABLE], [COMMAND], [MODULES]) +dnl --------------------------------------------- +dnl Internal wrapper calling pkg-config via PKG_CONFIG and setting +dnl pkg_failed based on the result. +m4_define([_PKG_CONFIG], +[if test -n "$$1"; then + pkg_cv_[]$1="$$1" + elif test -n "$PKG_CONFIG"; then + PKG_CHECK_EXISTS([$3], + [pkg_cv_[]$1=`$PKG_CONFIG --[]$2 "$3" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes ], + [pkg_failed=yes]) + else + pkg_failed=untried +fi[]dnl +])dnl _PKG_CONFIG + +dnl _PKG_SHORT_ERRORS_SUPPORTED +dnl --------------------------- +dnl Internal check to see if pkg-config supports short errors. +AC_DEFUN([_PKG_SHORT_ERRORS_SUPPORTED], +[AC_REQUIRE([PKG_PROG_PKG_CONFIG]) +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi[]dnl +])dnl _PKG_SHORT_ERRORS_SUPPORTED + + +dnl PKG_CHECK_MODULES(VARIABLE-PREFIX, MODULES, [ACTION-IF-FOUND], +dnl [ACTION-IF-NOT-FOUND]) +dnl -------------------------------------------------------------- +dnl Since: 0.4.0 +dnl +dnl Note that if there is a possibility the first call to +dnl PKG_CHECK_MODULES might not happen, you should be sure to include an +dnl explicit call to PKG_PROG_PKG_CONFIG in your configure.ac +AC_DEFUN([PKG_CHECK_MODULES], +[AC_REQUIRE([PKG_PROG_PKG_CONFIG])dnl +AC_ARG_VAR([$1][_CFLAGS], [C compiler flags for $1, overriding pkg-config])dnl +AC_ARG_VAR([$1][_LIBS], [linker flags for $1, overriding pkg-config])dnl + +pkg_failed=no +AC_MSG_CHECKING([for $1]) + +_PKG_CONFIG([$1][_CFLAGS], [cflags], [$2]) +_PKG_CONFIG([$1][_LIBS], [libs], [$2]) + +m4_define([_PKG_TEXT], [Alternatively, you may set the environment variables $1[]_CFLAGS +and $1[]_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details.]) + +if test $pkg_failed = yes; then + AC_MSG_RESULT([no]) + _PKG_SHORT_ERRORS_SUPPORTED + if test $_pkg_short_errors_supported = yes; then + $1[]_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "$2" 2>&1` + else + $1[]_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "$2" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$$1[]_PKG_ERRORS" >&AS_MESSAGE_LOG_FD + + m4_default([$4], [AC_MSG_ERROR( +[Package requirements ($2) were not met: + +$$1_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +_PKG_TEXT])[]dnl + ]) +elif test $pkg_failed = untried; then + AC_MSG_RESULT([no]) + m4_default([$4], [AC_MSG_FAILURE( +[The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +_PKG_TEXT + +To get pkg-config, see .])[]dnl + ]) +else + $1[]_CFLAGS=$pkg_cv_[]$1[]_CFLAGS + $1[]_LIBS=$pkg_cv_[]$1[]_LIBS + AC_MSG_RESULT([yes]) + $3 +fi[]dnl +])dnl PKG_CHECK_MODULES + + +dnl PKG_CHECK_MODULES_STATIC(VARIABLE-PREFIX, MODULES, [ACTION-IF-FOUND], +dnl [ACTION-IF-NOT-FOUND]) +dnl --------------------------------------------------------------------- +dnl Since: 0.29 +dnl +dnl Checks for existence of MODULES and gathers its build flags with +dnl static libraries enabled. Sets VARIABLE-PREFIX_CFLAGS from --cflags +dnl and VARIABLE-PREFIX_LIBS from --libs. +dnl +dnl Note that if there is a possibility the first call to +dnl PKG_CHECK_MODULES_STATIC might not happen, you should be sure to +dnl include an explicit call to PKG_PROG_PKG_CONFIG in your +dnl configure.ac. +AC_DEFUN([PKG_CHECK_MODULES_STATIC], +[AC_REQUIRE([PKG_PROG_PKG_CONFIG])dnl +_save_PKG_CONFIG=$PKG_CONFIG +PKG_CONFIG="$PKG_CONFIG --static" +PKG_CHECK_MODULES($@) +PKG_CONFIG=$_save_PKG_CONFIG[]dnl +])dnl PKG_CHECK_MODULES_STATIC + + +dnl PKG_INSTALLDIR([DIRECTORY]) +dnl ------------------------- +dnl Since: 0.27 +dnl +dnl Substitutes the variable pkgconfigdir as the location where a module +dnl should install pkg-config .pc files. By default the directory is +dnl $libdir/pkgconfig, but the default can be changed by passing +dnl DIRECTORY. The user can override through the --with-pkgconfigdir +dnl parameter. +AC_DEFUN([PKG_INSTALLDIR], +[m4_pushdef([pkg_default], [m4_default([$1], ['${libdir}/pkgconfig'])]) +m4_pushdef([pkg_description], + [pkg-config installation directory @<:@]pkg_default[@:>@]) +AC_ARG_WITH([pkgconfigdir], + [AS_HELP_STRING([--with-pkgconfigdir], pkg_description)],, + [with_pkgconfigdir=]pkg_default) +AC_SUBST([pkgconfigdir], [$with_pkgconfigdir]) +m4_popdef([pkg_default]) +m4_popdef([pkg_description]) +])dnl PKG_INSTALLDIR + + +dnl PKG_NOARCH_INSTALLDIR([DIRECTORY]) +dnl -------------------------------- +dnl Since: 0.27 +dnl +dnl Substitutes the variable noarch_pkgconfigdir as the location where a +dnl module should install arch-independent pkg-config .pc files. By +dnl default the directory is $datadir/pkgconfig, but the default can be +dnl changed by passing DIRECTORY. The user can override through the +dnl --with-noarch-pkgconfigdir parameter. +AC_DEFUN([PKG_NOARCH_INSTALLDIR], +[m4_pushdef([pkg_default], [m4_default([$1], ['${datadir}/pkgconfig'])]) +m4_pushdef([pkg_description], + [pkg-config arch-independent installation directory @<:@]pkg_default[@:>@]) +AC_ARG_WITH([noarch-pkgconfigdir], + [AS_HELP_STRING([--with-noarch-pkgconfigdir], pkg_description)],, + [with_noarch_pkgconfigdir=]pkg_default) +AC_SUBST([noarch_pkgconfigdir], [$with_noarch_pkgconfigdir]) +m4_popdef([pkg_default]) +m4_popdef([pkg_description]) +])dnl PKG_NOARCH_INSTALLDIR + + +dnl PKG_CHECK_VAR(VARIABLE, MODULE, CONFIG-VARIABLE, +dnl [ACTION-IF-FOUND], [ACTION-IF-NOT-FOUND]) +dnl ------------------------------------------- +dnl Since: 0.28 +dnl +dnl Retrieves the value of the pkg-config variable for the given module. +AC_DEFUN([PKG_CHECK_VAR], +[AC_REQUIRE([PKG_PROG_PKG_CONFIG])dnl +AC_ARG_VAR([$1], [value of $3 for $2, overriding pkg-config])dnl + +_PKG_CONFIG([$1], [variable="][$3]["], [$2]) +AS_VAR_COPY([$1], [pkg_cv_][$1]) + +AS_VAR_IF([$1], [""], [$5], [$4])dnl +])dnl PKG_CHECK_VAR + +dnl PKG_WITH_MODULES(VARIABLE-PREFIX, MODULES, +dnl [ACTION-IF-FOUND],[ACTION-IF-NOT-FOUND], +dnl [DESCRIPTION], [DEFAULT]) +dnl ------------------------------------------ +dnl +dnl Prepare a "--with-" configure option using the lowercase +dnl [VARIABLE-PREFIX] name, merging the behaviour of AC_ARG_WITH and +dnl PKG_CHECK_MODULES in a single macro. +AC_DEFUN([PKG_WITH_MODULES], +[ +m4_pushdef([with_arg], m4_tolower([$1])) + +m4_pushdef([description], + [m4_default([$5], [build with ]with_arg[ support])]) + +m4_pushdef([def_arg], [m4_default([$6], [auto])]) +m4_pushdef([def_action_if_found], [AS_TR_SH([with_]with_arg)=yes]) +m4_pushdef([def_action_if_not_found], [AS_TR_SH([with_]with_arg)=no]) + +m4_case(def_arg, + [yes],[m4_pushdef([with_without], [--without-]with_arg)], + [m4_pushdef([with_without],[--with-]with_arg)]) + +AC_ARG_WITH(with_arg, + AS_HELP_STRING(with_without, description[ @<:@default=]def_arg[@:>@]),, + [AS_TR_SH([with_]with_arg)=def_arg]) + +AS_CASE([$AS_TR_SH([with_]with_arg)], + [yes],[PKG_CHECK_MODULES([$1],[$2],$3,$4)], + [auto],[PKG_CHECK_MODULES([$1],[$2], + [m4_n([def_action_if_found]) $3], + [m4_n([def_action_if_not_found]) $4])]) + +m4_popdef([with_arg]) +m4_popdef([description]) +m4_popdef([def_arg]) + +])dnl PKG_WITH_MODULES + +dnl PKG_HAVE_WITH_MODULES(VARIABLE-PREFIX, MODULES, +dnl [DESCRIPTION], [DEFAULT]) +dnl ----------------------------------------------- +dnl +dnl Convenience macro to trigger AM_CONDITIONAL after PKG_WITH_MODULES +dnl check._[VARIABLE-PREFIX] is exported as make variable. +AC_DEFUN([PKG_HAVE_WITH_MODULES], +[ +PKG_WITH_MODULES([$1],[$2],,,[$3],[$4]) + +AM_CONDITIONAL([HAVE_][$1], + [test "$AS_TR_SH([with_]m4_tolower([$1]))" = "yes"]) +])dnl PKG_HAVE_WITH_MODULES + +dnl PKG_HAVE_DEFINE_WITH_MODULES(VARIABLE-PREFIX, MODULES, +dnl [DESCRIPTION], [DEFAULT]) +dnl ------------------------------------------------------ +dnl +dnl Convenience macro to run AM_CONDITIONAL and AC_DEFINE after +dnl PKG_WITH_MODULES check. HAVE_[VARIABLE-PREFIX] is exported as make +dnl and preprocessor variable. +AC_DEFUN([PKG_HAVE_DEFINE_WITH_MODULES], +[ +PKG_HAVE_WITH_MODULES([$1],[$2],[$3],[$4]) + +AS_IF([test "$AS_TR_SH([with_]m4_tolower([$1]))" = "yes"], + [AC_DEFINE([HAVE_][$1], 1, [Enable ]m4_tolower([$1])[ support])]) +])dnl PKG_HAVE_DEFINE_WITH_MODULES -- 2.18.1 From wk at gnupg.org Fri Oct 12 08:31:10 2018 From: wk at gnupg.org (Werner Koch) Date: Fri, 12 Oct 2018 08:31:10 +0200 Subject: [PATCH Libgpg-error] gpg-error.m4: support pkg-config In-Reply-To: <20181011182720.11991-1-alon.barlev@gmail.com> (Alon Bar-Lev's message of "Thu, 11 Oct 2018 21:27:20 +0300") References: <20181011182720.11991-1-alon.barlev@gmail.com> Message-ID: <871s8vptg1.fsf@wheatstone.g10code.de> On Thu, 11 Oct 2018 20:27, alon.barlev at gmail.com said: > * Simplicity, built for the task. I cannot say that 145,517 source lines of code is a good indication of simplicity (Debian's 0.29). Compare that to make which has 30,919 SLOC and thus 21% of pkg-config. I know that pkg-config it comes with glib included but for a bootstrapping tool this is quite excessive. In particular when looking at the 45,000 SLOC of libgpg-error. > * Good integration into autoconf using the pkg.m4 macros, enables the > detection of pkg-config, fallback if missing and handle the > interaction, including override settings by environment. And still not way to indicate an API break. Pretty inflexible way to add new configure options due to using its own syntax instead of utilizing the standard glue scripting language on Unix. Remember that it was build for large projects like GNOME where creating new libs is so common that virtually nobody takes care of ABI/API maintenance. For such an environment pkg-config (or the older larger foo-config) stuff makes a lot of sense. For libraries with a maintained API and ABI a simpler, more portable but also harder to initially create dedicated config file is a cleaner approach. > * Support for multilib configurations as configuration is installed per > library setup, unlike a common script that is installed at bindir. Multilib is a questionable feature as it introduced a third way to build packages (native, cross, multilib). The major problem, here is that it was been developed by the linux distros without integrating this in the standard GNU toolchain or getting it properly upstream. > * Cross-compile support, support sysroot and does not require to execute > anything from sysroot. It seems to have improved over the years. I remember well, how we needed to patch around pkg-config when cross building larger systems for Windows. Please don't let us have such discussions in commit logs. Salam-Shalom, Werner -- Die Gedanken sind frei. Ausnahmen regelt ein Bundesgesetz. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 227 bytes Desc: not available URL: From gnupg-devel at spodhuis.org Fri Oct 12 17:18:56 2018 From: gnupg-devel at spodhuis.org (Phil Pennock) Date: Fri, 12 Oct 2018 11:18:56 -0400 Subject: [PATCH Libgpg-error] gpg-error.m4: support pkg-config In-Reply-To: <871s8vptg1.fsf@wheatstone.g10code.de> References: <20181011182720.11991-1-alon.barlev@gmail.com> <871s8vptg1.fsf@wheatstone.g10code.de> Message-ID: <20181012151856.GA93352@osmium.pennocktech.home.arpa> On 2018-10-12 at 08:31 +0200, Werner Koch wrote: > On Thu, 11 Oct 2018 20:27, alon.barlev at gmail.com said: > > * Simplicity, built for the task. > > I cannot say that 145,517 source lines of code is a good indication of > simplicity (Debian's 0.29). Compare that to make which has 30,919 SLOC > and thus 21% of pkg-config. I know that pkg-config it comes with glib > included but for a bootstrapping tool this is quite excessive. In > particular when looking at the 45,000 SLOC of libgpg-error. FreeBSD reimplemented the pkg-config API in a much simpler code-base, back in 2011. AFAIK this is the tooling used by FreeBSD Ports for all pkg-config stuff, so is pretty well reality-tested by now. Port devel/pkgconf with repo https://github.com/pkgconf/pkgconf % wc -l libpkgconf/**/*(.) cli/**/*(.) | tail -n 1 9152 total One implementation is big, but that doesn't mean folks have to use that one. If GnuPG can invoke external tooling, they can use pkgconf for bootstrapping. -Phil From dkg at fifthhorseman.net Fri Oct 12 16:51:40 2018 From: dkg at fifthhorseman.net (Daniel Kahn Gillmor) Date: Fri, 12 Oct 2018 10:51:40 -0400 Subject: Bug#869605: libgpg-error: Add support for arm64ilp32 In-Reply-To: <20170724175848.8955.99124.reportbug@cheddar.halon.org.uk> References: <20170724175848.8955.99124.reportbug@cheddar.halon.org.uk> Message-ID: <875zy7nrpf.fsf@fifthhorseman.net> Hi Wookey-- On Mon 2017-07-24 18:58:48 +0100, Wookey wrote: > This small patch adds support for the arm64ilp32 architecture. I've just pushed this upstream at https://dev.gnupg.org/rEa3f4e8838036a14e87cca811e40c9f670f152fcd sorry for the delay, and thanks for contributing another per-architecture header to libgpg-error! --dkg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 227 bytes Desc: not available URL: From alon.barlev at gmail.com Sat Oct 13 00:07:29 2018 From: alon.barlev at gmail.com (Alon Bar-Lev) Date: Sat, 13 Oct 2018 01:07:29 +0300 Subject: [PATCH Libgpg-error] gpg-error.m4: support pkg-config In-Reply-To: <871s8vptg1.fsf@wheatstone.g10code.de> References: <20181011182720.11991-1-alon.barlev@gmail.com> <871s8vptg1.fsf@wheatstone.g10code.de> Message-ID: On Fri, Oct 12, 2018 at 9:35 AM Werner Koch wrote: > > On Thu, 11 Oct 2018 20:27, alon.barlev at gmail.com said: Hi, > > * Simplicity, built for the task. > > I cannot say that 145,517 source lines of code is a good indication of > simplicity (Debian's 0.29). Compare that to make which has 30,919 SLOC > and thus 21% of pkg-config. I know that pkg-config it comes with glib > included but for a bootstrapping tool this is quite excessive. In > particular when looking at the 45,000 SLOC of libgpg-error. I am sure you know the following, but I must address you counting line of codes... This is odd... feel free to skip the next story. << __STORY__ scripts were used to build programs, then realized that building everything every time is a waste of time and resources, and scripts became complex to address these concerns. make was introduced to manage a set of simple rules to avoid re-build when possible, then realized that for large project it is difficult to write proper make files, make files got complex and in most cases implemented using wrong logic. autoconf was introduced to generate files from templates based on logic as it was difficult to add functional logic into make files. automake was introduced to provide a simple method to generate make files that actually work with less error prune syntax, supporting dependency management, distribution, tests, conditionals and more. When shared library were introduced, most people got it wrong as each architecture had different technology (if any) and tweaks (RPATH, SONAME), support test and post-install etc... make files become complex once again. libtool was introduced to simplify library (static and shared) creation, addressing all concerns. CFLAGS, LDFLAGS detection were not that simple as threading and other compiler settings should have considered, it required additional logic of detection which was not always aligned with the library best practice. These concerns could have been address using metadata or scripts, at first there was no standard for metadata, so these programs that required special settings, developed their own unique solutions. pkg-config evolved as metadata usage. There are advantages of using metadata based information, mainly the ability to provide tool selection to manage the metadata, having consistent behavior among packages, not running anything from sysroot - all is important to downstream maintainer and anyone who try to build. __STORY__ The gnupg projects already use them all, make, autoconf, automake, libtool and pkg-config, so let's at least count them all and understand why we use and reuse components that are shared between projects. Don't get me wrong, I truly respect the gnupg project and maintained its packages for many years, however, after reached to more or less stable feature-set and structure, it may be the time to address some of the issues that downstream maintainer experience to make it easier to use the packages is required use cases. When I saw libgpg-error master publishes pkg-config metadata, I was very happy, as it does show some new openness, as I know your point of view. You also stated that you want pkg-config to be second class option, so I introduced the enable_pkg_config flag with default no, to keep backward compatibility. Using the pkg-config resolves issues of multilib and cross-compile without need to modify the existing mechanism. Some statistics: 1. Per component a non trivial specific m4 script is maintained. 2. The m4 script required by dependencies so it is distributed to other components (aclocal auto install is not used), this create additional maintenance effort. 3. Size of each: $ n=0; for f in ./gpgme/src/gpgme.m4 ./libassuan/src/libassuan.m4 ./libgcrypt/src/libgcrypt.m4 ./libgpg-error/src/gpg-error.m4 ./libksba/src/ksba.m4; do l=$(cat $f | grep -v '^dnl' | wc -l); echo $l $f; n=$(($n+$l)); done; echo $n total 265 ./gpgme/src/gpgme.m4 127 ./libassuan/src/libassuan.m4 128 ./libgcrypt/src/libgcrypt.m4 128 ./libgpg-error/src/gpg-error.m4 113 ./libksba/src/ksba.m4 761 total 4. Per each component a config script is being maintained. 5. The script is different per project. 6. In libgpg-error master the script is a wrapper on top of pkg-config metadata which implies that the pkg-config metadata is formally maintained. 7. Size of scripts: $ find . -name '*-config.in' | sort | xargs wc -l 139 ./libksba/src/ksba-config.in 134 ./libassuan/src/libassuan-config.in 189 ./libgcrypt/src/libgcrypt-config.in 103 ./libgpg-error/src/gpg-error-config.in 211 ./gpgme/src/gpgme-config.in 776 total 8. pkg-config and pkg.m4 are already used by the following projects: $ find . -name 'pkg.m4' | sort ./gnupg/m4/pkg.m4 ./gpgme/m4/pkg.m4 ./pinentry/m4/pkg.m4 9. Size of the pkg.m4 is smaller than the usage of component m4, and reused for all cases of dependency detection: $ cat m4/pkg.m4 | grep -v '^dnl' | wc -l 212 9. pkg-conf is an re-implementation of pkg-config: $ find pkgconf-1.3.7/ -name '*.c' | xargs wc -l | grep total 6421 total 10. pkg-config is bloated because it has embedded glib: $ find pkg-config-0.29.2/ -name '*.c' | xargs wc -l | grep total 123507 total $ find pkg-config-0.29.2/ -name '*.c' | grep -v /glib | xargs wc -l | grep total 3386 total > > * Good integration into autoconf using the pkg.m4 macros, enables the > > detection of pkg-config, fallback if missing and handle the > > interaction, including override settings by environment. > > And still not way to indicate an API break. Pretty inflexible way to > add new configure options due to using its own syntax instead of > utilizing the standard glue scripting language on Unix. I am unsure I follow, I will appreciate if you provide an example from the existing script. I do not see a difference between pkg-config and script: a. it can detect a package based on component version expression b. if ABI version is required retrieve a specific variable out of the metadata. c. if a severe ABI breakage is happening project can modify the name of the metadata for side-by-side or even install multiple metadata each with different setting What is missing? > Remember that it was build for large projects like GNOME where creating > new libs is so common that virtually nobody takes care of ABI/API > maintenance. For such an environment pkg-config (or the older larger > foo-config) stuff makes a lot of sense. For libraries with a maintained > API and ABI a simpler, more portable but also harder to initially create > dedicated config file is a cleaner approach. I do not understand this argument, I will appreciate an example. > > * Support for multilib configurations as configuration is installed per > > library setup, unlike a common script that is installed at bindir. > > Multilib is a questionable feature as it introduced a third way to build > packages (native, cross, multilib). The major problem, here is that it > was been developed by the linux distros without integrating this in the > standard GNU toolchain or getting it properly upstream. On the contrary... I think. Each multilib configuration has a different ${HOST} which complies with toolchain standards. The autoconf AC_*_TOOL are designed to search for a tool at convention of ${HOST}-${TOOL}, which is aligned to the config script approach. I already raised this issue in this recent thread, using multilib with config script should: 1. install the tool as ${HOST}-${TOOL}, symlink this into ${TOOL} on native if it makes sense. 2. use AC_*_TOOL to search for the tool 3. for cross-compile, install these config scripts outside of the sysroot, as it makes no sense to execute anything from sysroot of other architecture 4. for cross-compile, the tool need to be familiar with sysroot concept to output paths relative to sysroot This is actually, the only point of why config script are "better" supported, however, it require effort to do this right, and the gnupg project should be fixed to follow the pattern. However, using metadata instead of config scripts make it easier to mange the settings as: 1. the tool is the tool which reads the metadata 2. nothing is running from the sysroot 3. the tool is familiar with sysroot Once the gnupg projects provide pkg-config metadata, it is probably easier to support these concerns properly within the pkg-config scope, and providing a pkg-config alternative (enable_pkg_config=yes) while preserving the current behavior for backward compatibility with past use cases. > > * Cross-compile support, support sysroot and does not require to execute > > anything from sysroot. > > It seems to have improved over the years. I remember well, how we > needed to patch around pkg-config when cross building larger systems for > Windows. Correct, it has been improved, it is not the best tool, but it is sufficient in most of the cases. If you find a specific issue, I will be happy to try and sort it out. > Please don't let us have such discussions in commit logs. I only expected to trigger discussion, I did not expected you just merge it as is :) If you want to proceed I will happily remove these comments. I hope you will be in favor of supporting pkg-config as 2nd citizen in gnupg projects. Thanks for the discussion, Alon From wk at gnupg.org Mon Oct 15 09:45:04 2018 From: wk at gnupg.org (Werner Koch) Date: Mon, 15 Oct 2018 09:45:04 +0200 Subject: [PATCH] GPGME: several fixes for Common Lisp bindings In-Reply-To: <20181011113825.76f9752a@yamatai> (Guillaume LE VAILLANT's message of "Thu, 11 Oct 2018 11:41:14 +0200") References: <20181010164229.334810d0@yamatai> <20181010193741.233e09a4@yamatai> <20181011113825.76f9752a@yamatai> Message-ID: <87d0sbmz5r.fsf@wheatstone.g10code.de> Hi! Thanks for providing the DCO and the cl patches. I applied them. Shalom-Salam, Werner p.s. Next time please use git format-pacth and provide a proper commit log as described in doc/HACKING. -- Die Gedanken sind frei. Ausnahmen regelt ein Bundesgesetz. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 227 bytes Desc: not available URL: From marcelf at selfnet.de Tue Oct 16 22:44:55 2018 From: marcelf at selfnet.de (Marcel Fest) Date: Tue, 16 Oct 2018 22:44:55 +0200 Subject: hkp4py hkp|hkps bindings for python. In-Reply-To: <20180922200007.eyrvhxoc6jmfiatn@adversary.org> References: <7c28519a-6ea7-b705-f749-6ad0a6e74df7@selfnet.de> <20180919172929.ktlixlmu5ju7ksgl@adversary.org> <96557dcc-cc5a-9bd3-9efa-df7a1ef5db4b@selfnet.de> <20180919201329.bmdjkq74tyxnxdjy@adversary.org> <20180919204926.ygrly3n2vtvnes3o@adversary.org> <6e33e016-0802-6be2-83f1-b11b67a21f73@selfnet.de> <20180922200007.eyrvhxoc6jmfiatn@adversary.org> Message-ID: Nice after releasing 1.12 everything is working fine. The python KeyServer library hkp4py is now supporting hkps keyservers. https://github.com/Selfnet/hkp4py Please checkout. Best Regards Marcel > Not that it's Arch's fault, it's not. It's that the key_import > function was added after GPGME 1.11.1 was released. The same is true > of the three export methods added around the same time. > > Whereas I'm using the latest version in the master branch almost all > the time (for obvious reasons). Anyway, the key_import, key_export, > key_export_minimal and key_export_secret functions will be available > to all when GPGME 1.12.0 is released very soon. With the additional > key_blob response you added, I'll be able to include some more > detailed examples in the HOWTO for that release. > > > Regards, > Ben > > _______________________________________________ > Gnupg-devel mailing list > Gnupg-devel at gnupg.org > http://lists.gnupg.org/mailman/listinfo/gnupg-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From wk at gnupg.org Wed Oct 17 07:17:09 2018 From: wk at gnupg.org (Werner Koch) Date: Wed, 17 Oct 2018 07:17:09 +0200 Subject: [Announce] GPA 0.10.0 released Message-ID: <87r2gpjgoa.fsf@wheatstone.g10code.de> Hello! We are pleased to announce GPA version 0.10.0. GPA is a graphical frontend for the GNU Privacy Guard (GnuPG). GPA can be used for most operations supported by GnuPG using either the OpenPGP or the S/MIME protocol. A smartcard manager and a generic user interface server are included as well. Find its homepage at https://gnupg.org/software/gpa . GPA is Free Software (meaning that it respects your freedom). It can be freely used, modified and distributed under the terms of the GNU General Public License. Noteworthy changes in version 0.10.0 (2018-10-16) ================================================ * Added key manager context menu items to copy the key fingerprint and the secret key to the clipboard. * Added "Details" buttons to many error popups to show raw diagnostic output from gpg. * Changed the "Retrieve Key" dialog to first try the Web Key Directory if a mail address is given. Only if this lookup fails the keyservers are searched. * Added a user-ID notebook page to show per user-ID info. * Made location of locale dir under Windows more flexible. * Fixed crash on filename conversion error. [#2185] * Fixed listing of key algos in the subkey windows. [#3405] * Removed lazy loading of the secret keyring. [#3748] Release-info: https://dev.gnupg.org/T4186 Download ======== You can find the source code here: https://gnupg.org/ftp/gcrypt/gpa/gpa-0.10.0.tar.bz2 (745k) https://gnupg.org/ftp/gcrypt/gpa/gpa-0.10.0.tar.bz2.sig and soon on all ftp.gnupg.org mirrors. A binary version for Windows will be part of the next Gpg4win release. The SHA1 checksum for this release is: 61475989acd12de8b7daacd906200e8b4f519c5a gpa-0.10.0.tar.bz2 Support ======= Please consult the archive of the gnupg-users mailing list and the release info given above before reporting a bug. We suggest to send bug reports for a fresh release to that list in favor of filing a bug at . If you are a developer and you may need a certain feature for your project, please do not hesitate to bring it to the gnupg-devel mailing list for discussion. Thanks ====== Maintenance and development of GnuPG is mostly financed by donations. The GnuPG project currently employs one full-time developer and two contractors. They work exclusively on GnuPG and closely related software like Libgcrypt and GPGME. We have to thank all the people who helped the GnuPG project, be it testing, coding, translating, suggesting, auditing, administering the servers, spreading the word, and answering questions on the mailing lists. Many thanks to our numerous financial supporters, both corporate and individuals. Without you it would not be possible to keep GnuPG in a good shape and address all the small and larger requests made by our users. Thanks. Happy hacking, Your GnuPG hackers p.s. This is an announcement only mailing list. Please send replies only to the gnupg-users 'at' gnupg.org mailing list. p.p.s List of Release Signing Keys: To guarantee that a downloaded GnuPG version has not been tampered by malicious entities we provide signature files for all tarballs and binary versions. The keys are also signed by the long term keys of their respective owners. Current releases are signed by one or more of these four keys: rsa2048 2011-01-12 [expires: 2019-12-31] Key fingerprint = D869 2123 C406 5DEA 5E0F 3AB5 249B 39D2 4F25 E3B6 Werner Koch (dist sig) rsa2048 2014-10-29 [expires: 2019-12-31] Key fingerprint = 46CC 7308 65BB 5C78 EBAB ADCF 0437 6F3E E085 6959 David Shaw (GnuPG Release Signing Key) rsa2048 2014-10-29 [expires: 2020-10-30] Key fingerprint = 031E C253 6E58 0D8E A286 A9F2 2071 B08A 33BD 3F06 NIIBE Yutaka (GnuPG Release Key) rsa3072 2017-03-17 [expires: 2027-03-15] Key fingerprint = 5B80 C575 4298 F0CB 55D8 ED6A BCEF 7E29 4B09 2E28 Andre Heinecke (Release Signing Key) The keys are available at and in any recently released GnuPG tarball in the file g10/distsigkey.gpg . Note that this mail has been signed by a different key. -- Die Gedanken sind frei. Ausnahmen regelt ein Bundesgesetz. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 227 bytes Desc: not available URL: -------------- next part -------------- _______________________________________________ Gnupg-announce mailing list Gnupg-announce at gnupg.org http://lists.gnupg.org/mailman/listinfo/gnupg-announce From wk at gnupg.org Wed Oct 17 11:27:41 2018 From: wk at gnupg.org (Werner Koch) Date: Wed, 17 Oct 2018 11:27:41 +0200 Subject: [PATCH Libgpg-error] gpg-error.m4: support pkg-config In-Reply-To: (Alon Bar-Lev's message of "Sat, 13 Oct 2018 01:07:29 +0300") References: <20181011182720.11991-1-alon.barlev@gmail.com> <871s8vptg1.fsf@wheatstone.g10code.de> Message-ID: <87efcokjn6.fsf@wheatstone.g10code.de> On Sat, 13 Oct 2018 00:07, alon.barlev at gmail.com said: > make was introduced to manage a set of simple rules to avoid re-build > when possible, then realized that for large project it is difficult to make was also devised in 1976 as a simple form of dependency tracker to make sure that changed source file won't go unnoticed. > autoconf was introduced to generate files from templates based on > logic as it was difficult to add functional logic into make files. That more describes imake. autoconf impelements the GNU strategy to first test a system for features and the use only standard style macros to make use of system specific features. This avoided the often deep nested ifdef chhains you still see in some software. > automake was introduced to provide a simple method to generate make > files that actually work with less error prune syntax, supporting Right. And to make sure that the required targets are always availabale (e.g. make distcheck). > These concerns could have been address using metadata or scripts, at > first there was no standard for metadata, so these programs that This is why autoconf runs tests. The meta data provided by libraries are actually not tests but hints on how they were configured. > selection to manage the metadata, having consistent behavior among > packages, not running anything from sysroot - all is important to Why should one not run something from sysroot? POSIX shell scripts are well suited to be run on all platforms, be it on the build system or or final host (after installation or shared with an emulator). > The gnupg projects already use them all, make, autoconf, automake, > libtool and pkg-config, so let's at least count them all and pkg-config only becuase some external packages provide only pkg-config stuff. > When I saw libgpg-error master publishes pkg-config metadata, I was > very happy, as it does show some new openness, as I know your point of > view. You also stated that you want pkg-config to be second class > option, so I introduced the enable_pkg_config flag with default no, to That is the whole poing with avoiding a second build system. People will soon start to use that alternative system and as maintainers we run in all kind of problems because it is assumed tha this is a supported way of using a library. > keep backward compatibility. Using the pkg-config resolves issues of > multilib and cross-compile without need to modify the existing I still can't see why this is the case. SYSROOT support is in our libraries since 2014 and makes it really easy to use a cross-build library: The foo-config script is run as $SYSROOT/bin/foo-config where the make install of the library has stored it. > 4. Per each component a config script is being maintained. Which is as easy as writing a pc file if not easier. > 5. The script is different per project. Sure, it describes the configuraion. > 6. In libgpg-error master the script is a wrapper on top of pkg-config > metadata which implies that the pkg-config metadata is formally > maintained. That is a technical detail on how the "second class citizen" foo.pc is implemented. > 8. pkg-config and pkg.m4 are already used by the following projects: > $ find . -name 'pkg.m4' | sort > ./gnupg/m4/pkg.m4 > ./gpgme/m4/pkg.m4 > ./pinentry/m4/pkg.m4 As well as dozens of other m4 files to test for system features. > the existing script. I do not see a difference between pkg-config and > script: > a. it can detect a package based on component version expression But that is all what you get. A script is much more powerful than a static description language and the script interpreter is a standard Unix tool on _all_ Unix platforms. In contrast pgk-config requires to build and install an extra tool before you can start. > c. if a severe ABI breakage is happening project can modify the name > of the metadata for side-by-side or even install multiple metadata > each with different setting Changing the project name to declare an ABI break. There are better ways. In fact pkg-config could also support this by not only tracking a version but also an ABI counter. AFAIK, it does not do that. >> foo-config) stuff makes a lot of sense. For libraries with a maintained >> API and ABI a simpler, more portable but also harder to initially create >> dedicated config file is a cleaner approach. > > I do not understand this argument, I will appreciate an example. If you look at really old foo-config scripts (they were introduced by Gtk+) you will see a lot of checks done by those scripts to cope with the fact that developers did not know how to properly design and maintain libraries. The foo-config scripts tried to detect that and warned the developer/user. Given the complexity of those scripts it was natural to unify that and we finally got to pkg-config. Large projects like GNOME or KDE put a lot of code in libraries and view them similar as programs and change the as they like. Those libraries are not re-usable but other software unless this other software's authors want to track the development of the umbrella project of a used library. In contrast other libraries are written in a way to make sure that other software can rely on the interface of such a library. For those libraries it is a negligible effort to maintain even a complicated foo-config script compared to what work is needed for all the other interface maintenance. > 3. for cross-compile, install these config scripts outside of the > sysroot, as it makes no sense to execute anything from sysroot of > other architecture As explained above I take another view here. Running config scripts is portable accorsoo platforms. > 4. for cross-compile, the tool need to be familiar with sysroot > concept to output paths relative to sysroot Agreed and implemented. Where do you see the bug in the SYSROOT support of our foo-configs? > I only expected to trigger discussion, I did not expected you just > merge it as is :) Worked. Salam-Shalom, Werner -- Die Gedanken sind frei. Ausnahmen regelt ein Bundesgesetz. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 227 bytes Desc: not available URL: From astieger at suse.com Wed Oct 17 21:56:52 2018 From: astieger at suse.com (Andreas Stieger) Date: Wed, 17 Oct 2018 21:56:52 +0200 Subject: [PATCH] gpa: needs gpgme 1.11.1 Message-ID: Hello, It would seem that gpa needs gpgme 1.11.1 to build. See attached patch. Andreas -- Andreas Stieger Head of Product Security SUSE Linux GmbH, GF: Felix Imend?rffer, Jane Smithard, Graham Norton, HRB 21284 (AG N?rnberg) -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Note-the-build-time-dependency-on-gpgme-1.11.1.patch Type: text/x-patch Size: 815 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From wk at gnupg.org Thu Oct 18 11:31:58 2018 From: wk at gnupg.org (Werner Koch) Date: Thu, 18 Oct 2018 11:31:58 +0200 Subject: [PATCH] gpa: needs gpgme 1.11.1 In-Reply-To: (Andreas Stieger's message of "Wed, 17 Oct 2018 21:56:52 +0200") References: Message-ID: <87tvljha7l.fsf@wheatstone.g10code.de> On Wed, 17 Oct 2018 21:56, astieger at suse.com said: > Hello, > > It would seem that gpa needs gpgme 1.11.1 to build. See attached patch. I wonder why this should be the case. > 9e11986 references GPGME_AUDITLOG_DIAG (new in gpgme 1.11.1) The code is only build if gpgme is recent enough: --8<---------------cut here---------------start------------->8--- #if GPGME_VERSION_NUMBER >= 0x010c00 /* >= 1.12 */ gpgme_data_t diag; char *info, *buffer, *result; gpg_error_t err; gpgme_engine_info_t engine; gpgme_protocol_t proto; int need_gpgme_free; if (!context) return NULL; if (gpgme_data_new (&diag)) return NULL; /* Ooops. */ context->inhibit_gpgme_events++; err = gpgme_op_getauditlog (context->ctx, diag, GPGME_AUDITLOG_DIAG); --8<---------------cut here---------------end--------------->8--- I wonder what went wrong here. > f160e92 references GPGME_KEYLIST_MODE_LOCATE (new in 1.11.0) You are right. A simple fix is attached. Salam-Shalom, Werner -- Die Gedanken sind frei. Ausnahmen regelt ein Bundesgesetz. -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Replace-use-of-the-GPGME_KEYLIST_MODE_LOCATE-alias.patch Type: text/x-diff Size: 1129 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 227 bytes Desc: not available URL: From astieger at suse.com Thu Oct 18 11:49:00 2018 From: astieger at suse.com (Andreas Stieger) Date: Thu, 18 Oct 2018 11:49:00 +0200 Subject: [PATCH] gpa: needs gpgme 1.11.1 In-Reply-To: <87tvljha7l.fsf@wheatstone.g10code.de> References: <87tvljha7l.fsf@wheatstone.g10code.de> Message-ID: On 10/18/18 11:31 AM, Werner Koch wrote: >> 9e11986 references GPGME_AUDITLOG_DIAG (new in gpgme 1.11.1) > The code is only build if gpgme is recent enough: > > --8<---------------cut here---------------start------------->8--- > #if GPGME_VERSION_NUMBER >= 0x010c00 /* >= 1.12 */ Right. For GPGME_AUDITLOG_DIAG I was merely looking for new symbols in gpgme. The actual build failure with 1.10.0 exactly was GPGME_KEYLIST_MODE_LOCATE. >> f160e92 references GPGME_KEYLIST_MODE_LOCATE (new in 1.11.0) > You are right. A simple fix is attached. Looks good. Thanks, Andreas -- Andreas Stieger Head of Product Security SUSE Linux GmbH, GF: Felix Imend?rffer, Jane Smithard, Graham Norton, HRB 21284 (AG N?rnberg) From alon.barlev at gmail.com Sat Oct 20 10:59:25 2018 From: alon.barlev at gmail.com (Alon Bar-Lev) Date: Sat, 20 Oct 2018 11:59:25 +0300 Subject: [PATCH Libgpg-error] gpg-error.m4: support pkg-config In-Reply-To: <87efcokjn6.fsf@wheatstone.g10code.de> References: <20181011182720.11991-1-alon.barlev@gmail.com> <871s8vptg1.fsf@wheatstone.g10code.de> <87efcokjn6.fsf@wheatstone.g10code.de> Message-ID: Hi Werner, First of all I must say that I fully agree with you that pkg-config is not the idle implementation, yes, I can think of many improvements and suggestions. However, the question... is it good enough. The fact that you plan to publish pkg-config metadata as "second class citizen", suggests that we already understand that we support the lowest common denominator, which is pkg-config capabilities. At the next ABI break (which is as you pointed one of the missing features of pkg-config. Well, apart of the obvious provide a new metadata name which you do not like) you will have to consider both script based and pkg-config implications on your consumers. I am still waiting you to address the actual patch... Provided gnupg libraries already publish pkg-config metadata, this trivial patch provides the option to select either mode, while keeping current approach as the default. Everybody wins... you keep upstream build pkg-config free whenever possible, while providing downstream and users to use pkg-config if there is a benefit of doing so. For your question about the current config scripts and cross compile, I already provided some missing bits... Here they are again: 1. Executables and/or scripts within SYSROOT is to be used only by target machine, when building nothing should be executed out of SYSROOT. [[pkg-config solves this by reading metadata out of the SYSROOT]] 2. The tools should be installed as ${HOST}-${TOOL}, so that multiple modes can be supported. This applies also for non-cross compile, such as multilib. [[pkg-config solves this by installing the metadata under the libdir]] 3. Tools should be SYSROOT aware, and output paths based on SYSROOT. [pkg-config solves this by adding PKG_CONFIG_SYSROOT_DIR as prefix to absolute paths]] 4. In autoconf you should use AC_*_TOOL instead of AC_*_PROG to locate these configuration scripts, this will search for a tool prefixed with ${HOST}-, it will find the right script per the target host, and it is the standard autoconf practice for build tools. [[pkg-config solves this by reading metadata from SYSROOT:libdir]] 5. The standard method would be to install ${HOST}-${TOOL} at SYSROOT/*/bin per (2) to serve the target system, while installing a ${HOST}-${TOOL} at build /*/bin with SYSROOT consideration, default where SYSROOT was when building the package. This will allow autoconf (4) to find it without any tweaks, per autoconf best practices. 5. Instead of (5), you can tweak the AC_*_TOOL (4) will search the SYSROOT/*/bin for the tools. However, you must still support the ${HOST}- prefix (2). This option still violate (1) the fact that nothing from SYSROOT should be executed during cross-compile, and should be avoided. Supporting pkg-config as an option solves these issues without adding more logic into scripts/autoconf/automake/users with this simple patch. You can decide in future to solve this properly also using the config scripts. Even if you decide to solve this in config scripts, and based on the fact that the config scripts already read pkg-config metadata... Probably the simplest option is to merge all scripts into a single script as the logic is the same, then provide a pkg-config compatible interface for this script, in other words, create a mini-pkg-config, so that autoconf might fallback to this script when pkg-config is not available. Regardless of the discussion of how to improve the config scripts, can you please consider to add the pkg-config as an optional method? Regards, Alon On Wed, Oct 17, 2018 at 12:30 PM Werner Koch wrote: > On Sat, 13 Oct 2018 00:07, alon.barlev at gmail.com said: > > > make was introduced to manage a set of simple rules to avoid re-build > > when possible, then realized that for large project it is difficult to > > make was also devised in 1976 as a simple form of dependency tracker to > make sure that changed source file won't go unnoticed. > > > autoconf was introduced to generate files from templates based on > > logic as it was difficult to add functional logic into make files. > > That more describes imake. autoconf impelements the GNU strategy to > first test a system for features and the use only standard style macros > to make use of system specific features. This avoided the often deep > nested ifdef chhains you still see in some software. > > > automake was introduced to provide a simple method to generate make > > files that actually work with less error prune syntax, supporting > > Right. And to make sure that the required targets are always availabale > (e.g. make distcheck). > > > These concerns could have been address using metadata or scripts, at > > first there was no standard for metadata, so these programs that > > This is why autoconf runs tests. The meta data provided by libraries > are actually not tests but hints on how they were configured. > > > selection to manage the metadata, having consistent behavior among > > packages, not running anything from sysroot - all is important to > > Why should one not run something from sysroot? POSIX shell scripts are > well suited to be run on all platforms, be it on the build system or or > final host (after installation or shared with an emulator). > > > The gnupg projects already use them all, make, autoconf, automake, > > libtool and pkg-config, so let's at least count them all and > > pkg-config only becuase some external packages provide only pkg-config > stuff. > > > When I saw libgpg-error master publishes pkg-config metadata, I was > > very happy, as it does show some new openness, as I know your point of > > view. You also stated that you want pkg-config to be second class > > option, so I introduced the enable_pkg_config flag with default no, to > > That is the whole poing with avoiding a second build system. People > will soon start to use that alternative system and as maintainers we run > in all kind of problems because it is assumed tha this is a supported > way of using a library. > > > keep backward compatibility. Using the pkg-config resolves issues of > > multilib and cross-compile without need to modify the existing > > I still can't see why this is the case. SYSROOT support is in our > libraries since 2014 and makes it really easy to use a cross-build > library: The foo-config script is run as $SYSROOT/bin/foo-config where > the make install of the library has stored it. > > > 4. Per each component a config script is being maintained. > > Which is as easy as writing a pc file if not easier. > > > 5. The script is different per project. > > Sure, it describes the configuraion. > > > 6. In libgpg-error master the script is a wrapper on top of pkg-config > > metadata which implies that the pkg-config metadata is formally > > maintained. > > That is a technical detail on how the "second class citizen" foo.pc is > implemented. > > > 8. pkg-config and pkg.m4 are already used by the following projects: > > $ find . -name 'pkg.m4' | sort > > ./gnupg/m4/pkg.m4 > > ./gpgme/m4/pkg.m4 > > ./pinentry/m4/pkg.m4 > > As well as dozens of other m4 files to test for system features. > > > the existing script. I do not see a difference between pkg-config and > > script: > > a. it can detect a package based on component version expression > > But that is all what you get. A script is much more powerful than a > static description language and the script interpreter is a standard > Unix tool on _all_ Unix platforms. In contrast pgk-config requires to > build and install an extra tool before you can start. > > > c. if a severe ABI breakage is happening project can modify the name > > of the metadata for side-by-side or even install multiple metadata > > each with different setting > > Changing the project name to declare an ABI break. There are better > ways. In fact pkg-config could also support this by not only tracking a > version but also an ABI counter. AFAIK, it does not do that. > > >> foo-config) stuff makes a lot of sense. For libraries with a maintained > >> API and ABI a simpler, more portable but also harder to initially create > >> dedicated config file is a cleaner approach. > > > > I do not understand this argument, I will appreciate an example. > > If you look at really old foo-config scripts (they were introduced by > Gtk+) you will see a lot of checks done by those scripts to cope with > the fact that developers did not know how to properly design and > maintain libraries. The foo-config scripts tried to detect that and > warned the developer/user. Given the complexity of those scripts it was > natural to unify that and we finally got to pkg-config. Large projects > like GNOME or KDE put a lot of code in libraries and view them similar > as programs and change the as they like. Those libraries are not > re-usable but other software unless this other software's authors want > to track the development of the umbrella project of a used library. In > contrast other libraries are written in a way to make sure that other > software can rely on the interface of such a library. For those > libraries it is a negligible effort to maintain even a complicated > foo-config script compared to what work is needed for all the other > interface maintenance. > > > 3. for cross-compile, install these config scripts outside of the > > sysroot, as it makes no sense to execute anything from sysroot of > > other architecture > > As explained above I take another view here. Running config scripts is > portable accorsoo platforms. > > > 4. for cross-compile, the tool need to be familiar with sysroot > > concept to output paths relative to sysroot > > Agreed and implemented. Where do you see the bug in the SYSROOT support > of our foo-configs? > > > I only expected to trigger discussion, I did not expected you just > > merge it as is :) > > Worked. > > > Salam-Shalom, > > Werner > > -- > Die Gedanken sind frei. Ausnahmen regelt ein Bundesgesetz. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wk at gnupg.org Fri Oct 26 20:16:21 2018 From: wk at gnupg.org (Werner Koch) Date: Fri, 26 Oct 2018 20:16:21 +0200 Subject: [Announce] Libgcrypt 1.8.4 released Message-ID: <87r2gclgju.fsf@wheatstone.g10code.de> Hi! The GnuPG Project is pleased to announce the availability of Libgcrypt versions 1.8.4. This is a maintenance release to fix a few minor bugs. Libgcrypt is a general purpose library of cryptographic building blocks. It is originally based on code used by GnuPG. It does not provide any implementation of OpenPGP or other protocols. Thorough understanding of applied cryptography is required to use Libgcrypt. Noteworthy changes in version 1.8.4 =================================== * Bug fixes: - Fix infinite loop due to applications using fork the wrong way. [#3491] - Fix possible leak of a few bits of secret primes to pageable memory. [#3848] - Fix possible hang in the RNG (1.8.3 only). [#4034] - Several minor fixes. [#4102,#4208,#4209,#4210,#4211,#4212] * Performance: - On Linux always make use of getrandom if possible and then use its /dev/urandom behaviour. [#3894] Download ======== Source code is hosted at the GnuPG FTP server and its mirrors as listed at . On the primary server the source tarball and its digital signature are: https://gnupg.org/ftp/gcrypt/libgcrypt/libgcrypt-1.8.4.tar.bz2 https://gnupg.org/ftp/gcrypt/libgcrypt/libgcrypt-1.8.4.tar.bz2.sig or gzip compressed: https://gnupg.org/ftp/gcrypt/libgcrypt/libgcrypt-1.8.4.tar.gz https://gnupg.org/ftp/gcrypt/libgcrypt/libgcrypt-1.8.4.tar.gz.sig In order to check that the version of Libgcrypt you downloaded is an original and unmodified file please follow the instructions found at . In short, you may use one of the following methods: - Check the supplied OpenPGP signature. For example to check the signature of the file libgcrypt-1.8.4.tar.bz2 you would use this command: gpg --verify libgcrypt-1.8.4.tar.bz2.sig libgcrypt-1.8.4.tar.bz2 This checks whether the signature file matches the source file. You should see a message indicating that the signature is good and made by one or more of the release signing keys. Make sure that this is a valid key, either by matching the shown fingerprint against a trustworthy list of valid release signing keys or by checking that the key has been signed by trustworthy other keys. See the end of this mail for information on the signing keys. - If you are not able to use an existing version of GnuPG, you have to verify the SHA-1 checksum. On Unix systems the command to do this is either "sha1sum" or "shasum". Assuming you downloaded the file libgcrypt-1.8.4.tar.bz2, you run the command like this: sha1sum libgcrypt-1.8.4.tar.bz2 and check that the output matches the first line from the this list: 4a8ef9db6922f3a31992aca5640b4198a69b58fc libgcrypt-1.8.4.tar.bz2 211855f39f3bc3c4a4f444d4c09d743dfc5cb427 libgcrypt-1.8.4.tar.gz You should also verify that the checksums above are authentic by matching them with copies of this announcement. Those copies can be found at other mailing lists, web sites, and search engines. Copying ======= Libgcrypt is distributed under the terms of the GNU Lesser General Public License (LGPLv2.1+). The helper programs as well as the documentation are distributed under the terms of the GNU General Public License (GPLv2+). The file LICENSES has notices about contributions that require that these additional notices are distributed. Support ======= In case of build problems specific to this release please first check https://dev.gnupg.org/T4234 for updated information. For help on developing with Libgcrypt you should read the included manual and optional ask on the gcrypt-devel mailing list [1]. A listing with commercial support offers for Libgcrypt and related software is available at the GnuPG web site [2]. If you are a developer and you may need a certain feature for your project, please do not hesitate to bring it to the gcrypt-devel mailing list for discussion. Thanks ====== Maintenance and development of GnuPG is mostly financed by donations. The GnuPG project currently employs one full-time developer and two contractors. They all work exclusively on GnuPG and closely related software like Libgcrypt, GPGME, and GPA. We have to thank all the people who helped the GnuPG project, be it testing, coding, translating, suggesting, auditing, administering the servers, spreading the word, and answering questions on the mailing lists. Thanks to Tomas Mraz for pointing out several smaller flaws. Many thanks to our numerous financial supporters, both corporate and individuals. Without you it would not be possible to keep GnuPG in a good shape and address all the small and larger requests made by our users. Thanks. Happy hacking, Your GnuPG hackers p.s. This is an announcement only mailing list. Please send replies only to the gnupg-users'at'gnupg.org mailing list. p.p.s List of Release Signing Keys: To guarantee that a downloaded GnuPG version has not been tampered by malicious entities we provide signature files for all tarballs and binary versions. The keys are also signed by the long term keys of their respective owners. Current releases are signed by one or more of these four keys: rsa2048 2011-01-12 [expires: 2019-12-31] Key fingerprint = D869 2123 C406 5DEA 5E0F 3AB5 249B 39D2 4F25 E3B6 Werner Koch (dist sig) rsa2048 2014-10-29 [expires: 2019-12-31] Key fingerprint = 46CC 7308 65BB 5C78 EBAB ADCF 0437 6F3E E085 6959 David Shaw (GnuPG Release Signing Key) rsa2048 2014-10-29 [expires: 2020-10-30] Key fingerprint = 031E C253 6E58 0D8E A286 A9F2 2071 B08A 33BD 3F06 NIIBE Yutaka (GnuPG Release Key) rsa3072 2017-03-17 [expires: 2027-03-15] Key fingerprint = 5B80 C575 4298 F0CB 55D8 ED6A BCEF 7E29 4B09 2E28 Andre Heinecke (Release Signing Key) The keys are available at and in any recently released GnuPG tarball in the file g10/distsigkey.gpg . Note that this mail has been signed by a different key. -- Die Gedanken sind frei. Ausnahmen regelt ein Bundesgesetz. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 227 bytes Desc: not available URL: -------------- next part -------------- _______________________________________________ Gnupg-announce mailing list Gnupg-announce at gnupg.org http://lists.gnupg.org/mailman/listinfo/gnupg-announce From jussi.kivilinna at iki.fi Sat Oct 27 23:05:50 2018 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sun, 28 Oct 2018 00:05:50 +0300 Subject: [PATCH 2/8] g10/decrypt-data: use iobuf_read for higher performance In-Reply-To: <154067434467.18498.15345072060569568612.stgit@localhost.localdomain> References: <154067434467.18498.15345072060569568612.stgit@localhost.localdomain> Message-ID: <154067434984.18498.10460060246369566821.stgit@localhost.localdomain> * g10/decrypt-data.c (fill_buffer): Use iobuf_read instead of iobuf_get for reading data. -- This patch reduces iobuf_read per byte processing overhead and speeds up decryption. Benchmark results below, tested on Intel Core i7-4790K (turbo off). Encrypted 2 GiB through pipe to ramfs file using AES128. Decrypt ramfs file out through pipe to /dev/null. before patch-set ---------------- gpg process no-armor: user time pipe transfer rate encrypt-aead: 1.02 1.0 GB/s decrypt-aead: 10.8 185 MB/s encrypt-cfb: 4.8 342 MB/s decrypt-cfb: 12.7 157 MB/s gpg process armor: user time pipe transfer rate encrypt-aead: 13.8 140 MB/s decrypt-aead: 30.6 68 MB/s encrypt-cfb: 17.4 114 MB/s decrypt-cfb: 32.6 64 MB/s after (decrypt opt) ------------------- gpg process no-armor: user time pipe transfer rate decrypt-aead: 7.3 263 MB/s decrypt-cfb: 9.3 211 MB/s gpg process armor: user time pipe transfer rate decrypt-aead: 27.0 77 MB/s decrypt-cfb: 29.0 72 MB/s Note: decryption results are much slower than encryption because of extra SHA1 & RIPEMD160 hashing. GnuPG-bug-id: 3786 Signed-off-by: Jussi Kivilinna --- 0 files changed diff --git a/g10/decrypt-data.c b/g10/decrypt-data.c index d1d72a30f..493471961 100644 --- a/g10/decrypt-data.c +++ b/g10/decrypt-data.c @@ -551,31 +551,42 @@ fill_buffer (decode_filter_ctx_t dfx, iobuf_t stream, byte *buffer, size_t nbytes, size_t offset) { size_t nread = offset; - int c; + size_t curr; + int ret; if (dfx->partial) { - for (; nread < nbytes; nread++ ) + while (nread < nbytes) { - if ((c = iobuf_get (stream)) == -1) + curr = nbytes - nread; + + ret = iobuf_read (stream, &buffer[nread], curr); + if (ret == -1) { dfx->eof_seen = 1; /* Normal EOF. */ break; } - buffer[nread] = c; + + nread += ret; } } else { - for (; nread < nbytes && dfx->length; nread++, dfx->length--) + while (nread < nbytes && dfx->length) { - c = iobuf_get (stream); - if (c == -1) + curr = nbytes - nread; + if (curr > dfx->length) + curr = dfx->length; + + ret = iobuf_read (stream, &buffer[nread], curr); + if (ret == -1) { dfx->eof_seen = 3; /* Premature EOF. */ break; } - buffer[nread] = c; + + nread += ret; + dfx->length -= ret; } if (!dfx->length) dfx->eof_seen = 1; /* Normal EOF. */ From jussi.kivilinna at iki.fi Sat Oct 27 23:06:21 2018 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sun, 28 Oct 2018 00:06:21 +0300 Subject: [PATCH 8/8] g10/armor: optimize radix64 to binary conversion In-Reply-To: <154067434467.18498.15345072060569568612.stgit@localhost.localdomain> References: <154067434467.18498.15345072060569568612.stgit@localhost.localdomain> Message-ID: <154067438088.18498.7179260928839603567.stgit@localhost.localdomain> * g10/armor.c (asctobin): Larger look-up table for fast path. (initialize): Update 'asctobin' initialization. (radix64_read): Add fast path for radix64 to binary conversion. -- This patch adds fast path for radix64 to binary conversion in armored decryption. Benchmark results below, tested on Intel Core i7-4790K (turbo off). Encrypted 2 GiB through pipe to ramfs file using AES128. Decrypt ramfs file out through pipe to /dev/null. before patch-set ---------------- gpg process armor: user time pipe transfer rate encrypt-aead: 13.8 140 MB/s decrypt-aead: 30.6 68 MB/s encrypt-cfb: 17.4 114 MB/s decrypt-cfb: 32.6 64 MB/s after (decrypt+iobuf+crc+radix64 opt) ------------------------------------- gpg process armor: user time pipe transfer rate decrypt-aead: 9.8 200 MB/s decrypt-cfb: 11.9 168 MB/s Signed-off-by: Jussi Kivilinna --- 0 files changed diff --git a/g10/armor.c b/g10/armor.c index 95293d91c..972766503 100644 --- a/g10/armor.c +++ b/g10/armor.c @@ -40,7 +40,7 @@ static const byte bintoasc[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZ" "abcdefghijklmnopqrstuvwxyz" "0123456789+/"; -static byte asctobin[256]; /* runtime initialized */ +static u32 asctobin[4][256]; /* runtime initialized */ static int is_initialized; @@ -171,11 +171,16 @@ initialize(void) u32 i; const byte *s; - /* build the helptable for radix64 to bin conversion */ - for(i=0; i < 256; i++ ) - asctobin[i] = 255; /* used to detect invalid characters */ + /* Build the helptable for radix64 to bin conversion. Value 0xffffffff is + used to detect invalid characters. */ + memset (asctobin, 0xff, sizeof(asctobin)); for(s=bintoasc,i=0; *s; s++,i++ ) - asctobin[*s] = i; + { + asctobin[0][*s] = i << (0 * 6); + asctobin[1][*s] = i << (1 * 6); + asctobin[2][*s] = i << (2 * 6); + asctobin[3][*s] = i << (3 * 6); + } is_initialized=1; } @@ -802,11 +807,13 @@ radix64_read( armor_filter_context_t *afx, IOBUF a, size_t *retn, byte *buf, size_t size ) { byte val; - int c, c2; + int c; + u32 binc; int checkcrc=0; int rc = 0; size_t n = 0; - int idx, onlypad=0; + int idx, onlypad=0; + int skip_fast = 0; idx = afx->idx; val = afx->radbuf[0]; @@ -827,6 +834,122 @@ radix64_read( armor_filter_context_t *afx, IOBUF a, size_t *retn, } again: + binc = asctobin[0][c]; + + if( binc != 0xffffffffUL ) + { + if( idx == 0 && skip_fast == 0 + && afx->buffer_pos + (16 - 1) < afx->buffer_len + && n + 12 < size) + { + /* Fast path for radix64 to binary conversion. */ + u32 b0,b1,b2,b3; + + /* Speculatively load 15 more input bytes. */ + b0 = binc << (3 * 6); + b0 |= asctobin[2][afx->buffer[afx->buffer_pos + 0]]; + b0 |= asctobin[1][afx->buffer[afx->buffer_pos + 1]]; + b0 |= asctobin[0][afx->buffer[afx->buffer_pos + 2]]; + b1 = asctobin[3][afx->buffer[afx->buffer_pos + 3]]; + b1 |= asctobin[2][afx->buffer[afx->buffer_pos + 4]]; + b1 |= asctobin[1][afx->buffer[afx->buffer_pos + 5]]; + b1 |= asctobin[0][afx->buffer[afx->buffer_pos + 6]]; + b2 = asctobin[3][afx->buffer[afx->buffer_pos + 7]]; + b2 |= asctobin[2][afx->buffer[afx->buffer_pos + 8]]; + b2 |= asctobin[1][afx->buffer[afx->buffer_pos + 9]]; + b2 |= asctobin[0][afx->buffer[afx->buffer_pos + 10]]; + b3 = asctobin[3][afx->buffer[afx->buffer_pos + 11]]; + b3 |= asctobin[2][afx->buffer[afx->buffer_pos + 12]]; + b3 |= asctobin[1][afx->buffer[afx->buffer_pos + 13]]; + b3 |= asctobin[0][afx->buffer[afx->buffer_pos + 14]]; + + /* Check if any of the input bytes were invalid. */ + if( (b0 | b1 | b2 | b3) != 0xffffffffUL ) + { + /* All 16 bytes are valid. */ + buf[n + 0] = b0 >> (2 * 8); + buf[n + 1] = b0 >> (1 * 8); + buf[n + 2] = b0 >> (0 * 8); + buf[n + 3] = b1 >> (2 * 8); + buf[n + 4] = b1 >> (1 * 8); + buf[n + 5] = b1 >> (0 * 8); + buf[n + 6] = b2 >> (2 * 8); + buf[n + 7] = b2 >> (1 * 8); + buf[n + 8] = b2 >> (0 * 8); + buf[n + 9] = b3 >> (2 * 8); + buf[n + 10] = b3 >> (1 * 8); + buf[n + 11] = b3 >> (0 * 8); + afx->buffer_pos += 16 - 1; + n += 12; + continue; + } + else if( b0 == 0xffffffffUL ) + { + /* byte[1..3] have invalid character(s). Switch to slow + path. */ + skip_fast = 1; + } + else if( b1 == 0xffffffffUL ) + { + /* byte[4..7] have invalid character(s), first 4 bytes are + valid. */ + buf[n + 0] = b0 >> (2 * 8); + buf[n + 1] = b0 >> (1 * 8); + buf[n + 2] = b0 >> (0 * 8); + afx->buffer_pos += 4 - 1; + n += 3; + skip_fast = 1; + continue; + } + else if( b2 == 0xffffffffUL ) + { + /* byte[8..11] have invalid character(s), first 8 bytes are + valid. */ + buf[n + 0] = b0 >> (2 * 8); + buf[n + 1] = b0 >> (1 * 8); + buf[n + 2] = b0 >> (0 * 8); + buf[n + 3] = b1 >> (2 * 8); + buf[n + 4] = b1 >> (1 * 8); + buf[n + 5] = b1 >> (0 * 8); + afx->buffer_pos += 8 - 1; + n += 6; + skip_fast = 1; + continue; + } + else /*if( b3 == 0xffffffffUL )*/ + { + /* byte[12..15] have invalid character(s), first 12 bytes + are valid. */ + buf[n + 0] = b0 >> (2 * 8); + buf[n + 1] = b0 >> (1 * 8); + buf[n + 2] = b0 >> (0 * 8); + buf[n + 3] = b1 >> (2 * 8); + buf[n + 4] = b1 >> (1 * 8); + buf[n + 5] = b1 >> (0 * 8); + buf[n + 6] = b2 >> (2 * 8); + buf[n + 7] = b2 >> (1 * 8); + buf[n + 8] = b2 >> (0 * 8); + afx->buffer_pos += 12 - 1; + n += 9; + skip_fast = 1; + continue; + } + } + + switch(idx) + { + case 0: val = binc << 2; break; + case 1: val |= (binc>>4)&3; buf[n++]=val;val=(binc<<4)&0xf0;break; + case 2: val |= (binc>>2)&15; buf[n++]=val;val=(binc<<6)&0xc0;break; + case 3: val |= binc&0x3f; buf[n++] = val; break; + } + idx = (idx+1) % 4; + + continue; + } + + skip_fast = 0; + if( c == '\n' || c == ' ' || c == '\r' || c == '\t' ) continue; else if( c == '=' ) { /* pad character: stop */ @@ -857,10 +980,10 @@ radix64_read( armor_filter_context_t *afx, IOBUF a, size_t *retn, if (afx->buffer_pos + 6 < afx->buffer_len && afx->buffer[afx->buffer_pos + 0] == '3' && afx->buffer[afx->buffer_pos + 1] == 'D' - && asctobin[afx->buffer[afx->buffer_pos + 2]] != 255 - && asctobin[afx->buffer[afx->buffer_pos + 3]] != 255 - && asctobin[afx->buffer[afx->buffer_pos + 4]] != 255 - && asctobin[afx->buffer[afx->buffer_pos + 5]] != 255 + && asctobin[0][afx->buffer[afx->buffer_pos + 2]] != 0xffffffffUL + && asctobin[0][afx->buffer[afx->buffer_pos + 3]] != 0xffffffffUL + && asctobin[0][afx->buffer[afx->buffer_pos + 4]] != 0xffffffffUL + && asctobin[0][afx->buffer[afx->buffer_pos + 5]] != 0xffffffffUL && afx->buffer[afx->buffer_pos + 6] == '\n') { afx->buffer_pos += 2; @@ -875,17 +998,10 @@ radix64_read( armor_filter_context_t *afx, IOBUF a, size_t *retn, checkcrc++; break; } - else if( (c = asctobin[(c2=c)]) == 255 ) { - log_error(_("invalid radix64 character %02X skipped\n"), c2); + else { + log_error(_("invalid radix64 character %02X skipped\n"), c); continue; } - switch(idx) { - case 0: val = c << 2; break; - case 1: val |= (c>>4)&3; buf[n++]=val;val=(c<<4)&0xf0;break; - case 2: val |= (c>>2)&15; buf[n++]=val;val=(c<<6)&0xc0;break; - case 3: val |= c&0x3f; buf[n++] = val; break; - } - idx = (idx+1) % 4; } afx->idx = idx; @@ -924,13 +1040,13 @@ radix64_read( armor_filter_context_t *afx, IOBUF a, size_t *retn, u32 mycrc = 0; idx = 0; do { - if( (c = asctobin[c]) == 255 ) + if( (binc = asctobin[0][c]) == 0xffffffffUL ) break; switch(idx) { - case 0: val = c << 2; break; - case 1: val |= (c>>4)&3; mycrc |= val << 16;val=(c<<4)&0xf0;break; - case 2: val |= (c>>2)&15; mycrc |= val << 8;val=(c<<6)&0xc0;break; - case 3: val |= c&0x3f; mycrc |= val; break; + case 0: val = binc << 2; break; + case 1: val |= (binc>>4)&3; mycrc |= val << 16;val=(binc<<4)&0xf0;break; + case 2: val |= (binc>>2)&15; mycrc |= val << 8;val=(binc<<6)&0xc0;break; + case 3: val |= binc&0x3f; mycrc |= val; break; } for(;;) { if( afx->buffer_pos < afx->buffer_len ) From jussi.kivilinna at iki.fi Sat Oct 27 23:05:44 2018 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sun, 28 Oct 2018 00:05:44 +0300 Subject: [PATCH 1/8] g10/decrypt-data: use fill_buffer in more places Message-ID: <154067434467.18498.15345072060569568612.stgit@localhost.localdomain> * g10/decrypt-data.c (mdc_decode_filter, decode_filter): Use fill_buffer. -- Signed-off-by: Jussi Kivilinna --- 0 files changed diff --git a/g10/decrypt-data.c b/g10/decrypt-data.c index 3951fa794..d1d72a30f 100644 --- a/g10/decrypt-data.c +++ b/g10/decrypt-data.c @@ -873,7 +873,6 @@ mdc_decode_filter (void *opaque, int control, IOBUF a, decode_filter_ctx_t dfx = opaque; size_t n, size = *ret_len; int rc = 0; - int c; /* Note: We need to distinguish between a partial and a fixed length packet. The first is the usual case as created by GPG. However @@ -894,25 +893,7 @@ mdc_decode_filter (void *opaque, int control, IOBUF a, log_assert (size > 44); /* Our code requires at least this size. */ /* Get at least 22 bytes and put it ahead in the buffer. */ - if (dfx->partial) - { - for (n=22; n < 44; n++) - { - if ( (c = iobuf_get(a)) == -1 ) - break; - buf[n] = c; - } - } - else - { - for (n=22; n < 44 && dfx->length; n++, dfx->length--) - { - c = iobuf_get (a); - if (c == -1) - break; /* Premature EOF. */ - buf[n] = c; - } - } + n = fill_buffer (dfx, a, buf, 44, 22); if (n == 44) { /* We have enough stuff - flush the holdback buffer. */ @@ -923,37 +904,11 @@ mdc_decode_filter (void *opaque, int control, IOBUF a, } else { - memcpy (buf, dfx->holdback, 22); } + /* Fill up the buffer. */ - if (dfx->partial) - { - for (; n < size; n++ ) - { - if ( (c = iobuf_get(a)) == -1 ) - { - dfx->eof_seen = 1; /* Normal EOF. */ - break; - } - buf[n] = c; - } - } - else - { - for (; n < size && dfx->length; n++, dfx->length--) - { - c = iobuf_get(a); - if (c == -1) - { - dfx->eof_seen = 3; /* Premature EOF. */ - break; - } - buf[n] = c; - } - if (!dfx->length) - dfx->eof_seen = 1; /* Normal EOF. */ - } + n = fill_buffer (dfx, a, buf, size, n); /* Move the trailing 22 bytes back to the holdback buffer. We have at least 44 bytes thus a memmove is not needed. */ @@ -1008,7 +963,7 @@ decode_filter( void *opaque, int control, IOBUF a, byte *buf, size_t *ret_len) decode_filter_ctx_t fc = opaque; size_t size = *ret_len; size_t n; - int c, rc = 0; + int rc = 0; if ( control == IOBUFCTRL_UNDERFLOW && fc->eof_seen ) @@ -1020,34 +975,7 @@ decode_filter( void *opaque, int control, IOBUF a, byte *buf, size_t *ret_len) { log_assert (a); - if (fc->partial) - { - for (n=0; n < size; n++ ) - { - c = iobuf_get(a); - if (c == -1) - { - fc->eof_seen = 1; /* Normal EOF. */ - break; - } - buf[n] = c; - } - } - else - { - for (n=0; n < size && fc->length; n++, fc->length--) - { - c = iobuf_get(a); - if (c == -1) - { - fc->eof_seen = 3; /* Premature EOF. */ - break; - } - buf[n] = c; - } - if (!fc->length) - fc->eof_seen = 1; /* Normal EOF. */ - } + n = fill_buffer (fc, a, buf, size, 0); if (n) { if (fc->cipher_hd) From jussi.kivilinna at iki.fi Sat Oct 27 23:06:15 2018 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sun, 28 Oct 2018 00:06:15 +0300 Subject: [PATCH 7/8] g10/armor: optimize binary to radix64 conversion In-Reply-To: <154067434467.18498.15345072060569568612.stgit@localhost.localdomain> References: <154067434467.18498.15345072060569568612.stgit@localhost.localdomain> Message-ID: <154067437571.18498.6931951523550513172.stgit@localhost.localdomain> * g10/armor.c (bintoasc): Change to read-only. (initialize): Use const pointer for 'bintoasc'. (armor_output_buf_as_radix64): New function for faster binary to radix64 conversion. (armor_filter): Use new conversion function. -- This patch adds faster binary to radix64 conversion to speed up armored encryption. Benchmark results below, tested on Intel Core i7-4790K (turbo off). Encrypted 2 GiB through pipe to ramfs file using AES128. Decrypt ramfs file out through pipe to /dev/null. before patch-set ---------------- gpg process armor: user time pipe transfer rate encrypt-aead: 13.8 140 MB/s decrypt-aead: 30.6 68 MB/s encrypt-cfb: 17.4 114 MB/s decrypt-cfb: 32.6 64 MB/s after (decrypt+iobuf+crc+radix64 opt) ------------------------------------- gpg process armor: user time pipe transfer rate encrypt-aead: 2.7 523 MB/s encrypt-cfb: 6.7 264 MB/s Signed-off-by: Jussi Kivilinna --- 0 files changed diff --git a/g10/armor.c b/g10/armor.c index 9da135a2b..95293d91c 100644 --- a/g10/armor.c +++ b/g10/armor.c @@ -37,9 +37,9 @@ #define MAX_LINELEN 20000 -static byte bintoasc[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZ" - "abcdefghijklmnopqrstuvwxyz" - "0123456789+/"; +static const byte bintoasc[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZ" + "abcdefghijklmnopqrstuvwxyz" + "0123456789+/"; static byte asctobin[256]; /* runtime initialized */ static int is_initialized; @@ -169,7 +169,8 @@ static void initialize(void) { u32 i; - byte *s; + const byte *s; + /* build the helptable for radix64 to bin conversion */ for(i=0; i < 256; i++ ) asctobin[i] = 255; /* used to detect invalid characters */ @@ -1003,6 +1004,121 @@ radix64_read( armor_filter_context_t *afx, IOBUF a, size_t *retn, return rc; } +static void +armor_output_buf_as_radix64 (armor_filter_context_t *afx, IOBUF a, + byte *buf, size_t size) +{ + byte radbuf[sizeof (afx->radbuf)]; + byte outbuf[64 + sizeof (afx->eol)]; + unsigned int eollen = strlen (afx->eol); + u32 in, in2; + int idx, idx2; + int i; + + idx = afx->idx; + idx2 = afx->idx2; + memcpy (radbuf, afx->radbuf, sizeof (afx->radbuf)); + + if (size && (idx || idx2)) + { + /* preload eol to outbuf buffer */ + memcpy (outbuf + 4, afx->eol, sizeof (afx->eol)); + + for (; size && (idx || idx2); buf++, size--) + { + radbuf[idx++] = *buf; + if (idx > 2) + { + idx = 0; + in = (u32)radbuf[0] << (2 * 8); + in |= (u32)radbuf[1] << (1 * 8); + in |= (u32)radbuf[2] << (0 * 8); + outbuf[0] = bintoasc[(in >> 18) & 077]; + outbuf[1] = bintoasc[(in >> 12) & 077]; + outbuf[2] = bintoasc[(in >> 6) & 077]; + outbuf[3] = bintoasc[(in >> 0) & 077]; + if (++idx2 >= (64/4)) + { /* pgp doesn't like 72 here */ + idx2=0; + iobuf_write (a, outbuf, 4 + eollen); + } + else + { + iobuf_write (a, outbuf, 4); + } + } + } + } + + if (size >= (64/4)*3) + { + /* preload eol to outbuf buffer */ + memcpy (outbuf + 64, afx->eol, sizeof(afx->eol)); + + do + { + /* idx and idx2 == 0 */ + + for (i = 0; i < (64/8); i++) + { + in = (u32)buf[0] << (2 * 8); + in |= (u32)buf[1] << (1 * 8); + in |= (u32)buf[2] << (0 * 8); + in2 = (u32)buf[3] << (2 * 8); + in2 |= (u32)buf[4] << (1 * 8); + in2 |= (u32)buf[5] << (0 * 8); + outbuf[i*8+0] = bintoasc[(in >> 18) & 077]; + outbuf[i*8+1] = bintoasc[(in >> 12) & 077]; + outbuf[i*8+2] = bintoasc[(in >> 6) & 077]; + outbuf[i*8+3] = bintoasc[(in >> 0) & 077]; + outbuf[i*8+4] = bintoasc[(in2 >> 18) & 077]; + outbuf[i*8+5] = bintoasc[(in2 >> 12) & 077]; + outbuf[i*8+6] = bintoasc[(in2 >> 6) & 077]; + outbuf[i*8+7] = bintoasc[(in2 >> 0) & 077]; + buf+=6; + size-=6; + } + + /* pgp doesn't like 72 here */ + iobuf_write (a, outbuf, 64 + eollen); + } + while (size >= (64/4)*3); + + /* restore eol for tail handling */ + if (size) + memcpy (outbuf + 4, afx->eol, sizeof (afx->eol)); + } + + for (; size; buf++, size--) + { + radbuf[idx++] = *buf; + if (idx > 2) + { + idx = 0; + in = (u32)radbuf[0] << (2 * 8); + in |= (u32)radbuf[1] << (1 * 8); + in |= (u32)radbuf[2] << (0 * 8); + outbuf[0] = bintoasc[(in >> 18) & 077]; + outbuf[1] = bintoasc[(in >> 12) & 077]; + outbuf[2] = bintoasc[(in >> 6) & 077]; + outbuf[3] = bintoasc[(in >> 0) & 077]; + if (++idx2 >= (64/4)) + { /* pgp doesn't like 72 here */ + idx2=0; + iobuf_write (a, outbuf, 4 + eollen); + } + else + { + iobuf_write (a, outbuf, 4); + } + } + } + + memcpy (afx->radbuf, radbuf, sizeof (afx->radbuf)); + afx->idx = idx; + afx->idx2 = idx2; +} + /**************** * This filter is used to handle the armor stuff */ @@ -1012,7 +1128,7 @@ armor_filter( void *opaque, int control, { size_t size = *ret_len; armor_filter_context_t *afx = opaque; - int rc=0, i, c; + int rc=0, c; byte radbuf[3]; int idx, idx2; size_t n=0; @@ -1196,37 +1312,11 @@ armor_filter( void *opaque, int control, afx->idx2 = 0; gcry_md_reset (afx->crc_md); } - idx = afx->idx; - idx2 = afx->idx2; - for(i=0; i < idx; i++ ) - radbuf[i] = afx->radbuf[i]; - - if( size ) - gcry_md_write (afx->crc_md, buf, size); - - for( ; size; buf++, size-- ) { - radbuf[idx++] = *buf; - if( idx > 2 ) { - idx = 0; - c = bintoasc[(*radbuf >> 2) & 077]; - iobuf_put(a, c); - c = bintoasc[(((*radbuf<<4)&060)|((radbuf[1] >> 4)&017))&077]; - iobuf_put(a, c); - c = bintoasc[(((radbuf[1]<<2)&074)|((radbuf[2]>>6)&03))&077]; - iobuf_put(a, c); - c = bintoasc[radbuf[2]&077]; - iobuf_put(a, c); - if( ++idx2 >= (64/4) ) - { /* pgp doesn't like 72 here */ - iobuf_writestr(a,afx->eol); - idx2=0; - } - } - } - for(i=0; i < idx; i++ ) - afx->radbuf[i] = radbuf[i]; - afx->idx = idx; - afx->idx2 = idx2; + + if( size ) { + gcry_md_write (afx->crc_md, buf, size); + armor_output_buf_as_radix64 (afx, a, buf, size); + } } else if( control == IOBUFCTRL_INIT ) { From jussi.kivilinna at iki.fi Sat Oct 27 23:06:10 2018 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sun, 28 Oct 2018 00:06:10 +0300 Subject: [PATCH 6/8] g10/armor: use libgcrypt's CRC24 implementation In-Reply-To: <154067434467.18498.15345072060569568612.stgit@localhost.localdomain> References: <154067434467.18498.15345072060569568612.stgit@localhost.localdomain> Message-ID: <154067437052.18498.8640610463986827148.stgit@localhost.localdomain> * g10/armor.c (CRCINIT, CRCPOLY, CRCUPDATE, crc_table): Remove. (new_armor_context): Open libgcrypt CRC24 context. (release_armor_context): Close CRC24 context. (initialize): Remove CRC table generation. (get_afx_crc): New. (check_input, fake_packet, radix64_read, armor_filter): Update to use CRC24 context. * g10/filter.h (armor_filter_context_t): Replace crc intermediate value with libgcrypt md context pointer. -- This patch changes armor filter to use optimized CRC24 implementation from libgcrypt to speed up encryption and decryption. Benchmark results below, tested on Intel Core i7-4790K (turbo off). Encrypted 2 GiB through pipe to ramfs file using AES128. Decrypt ramfs file out through pipe to /dev/null. before patch-set ---------------- gpg process armor: user time pipe transfer rate encrypt-aead: 13.8 140 MB/s decrypt-aead: 30.6 68 MB/s encrypt-cfb: 17.4 114 MB/s decrypt-cfb: 32.6 64 MB/s after (decrypt+iobuf+crc opt) ----------------------------- gpg process armor: user time pipe transfer rate encrypt-aead: 8.7 211 MB/s decrypt-aead: 17.6 116 MB/s encrypt-cfb: 12.6 153 MB/s decrypt-cfb: 19.6 105 MB/s Signed-off-by: Jussi Kivilinna --- 0 files changed diff --git a/g10/armor.c b/g10/armor.c index 714282f45..9da135a2b 100644 --- a/g10/armor.c +++ b/g10/armor.c @@ -37,13 +37,6 @@ #define MAX_LINELEN 20000 -#define CRCINIT 0xB704CE -#define CRCPOLY 0X864CFB -#define CRCUPDATE(a,c) do { \ - a = ((a) << 8) ^ crc_table[((a)&0xff >> 16) ^ (c)]; \ - a &= 0x00ffffff; \ - } while(0) -static u32 crc_table[256]; static byte bintoasc[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZ" "abcdefghijklmnopqrstuvwxyz" "0123456789+/"; @@ -121,9 +114,22 @@ armor_filter_context_t * new_armor_context (void) { armor_filter_context_t *afx; + gpg_error_t err; afx = xcalloc (1, sizeof *afx); - afx->refcount = 1; + if (afx) + { + err = gcry_md_open (&afx->crc_md, GCRY_MD_CRC24_RFC2440, 0); + if (err != 0) + { + log_error ("gcry_md_open failed for GCRY_MD_CRC24_RFC2440: %s", + gpg_strerror (err)); + xfree (afx); + return NULL; + } + + afx->refcount = 1; + } return afx; } @@ -138,6 +144,7 @@ release_armor_context (armor_filter_context_t *afx) log_assert (afx->refcount); if ( --afx->refcount ) return; + gcry_md_close (afx->crc_md); xfree (afx); } @@ -161,25 +168,8 @@ push_armor_filter (armor_filter_context_t *afx, iobuf_t iobuf) static void initialize(void) { - int i, j; - u32 t; + u32 i; byte *s; - - /* init the crc lookup table */ - crc_table[0] = 0; - for(i=j=0; j < 128; j++ ) { - t = crc_table[j]; - if( t & 0x00800000 ) { - t <<= 1; - crc_table[i++] = t ^ CRCPOLY; - crc_table[i++] = t; - } - else { - t <<= 1; - crc_table[i++] = t; - crc_table[i++] = t ^ CRCPOLY; - } - } /* build the helptable for radix64 to bin conversion */ for(i=0; i < 256; i++ ) asctobin[i] = 255; /* used to detect invalid characters */ @@ -190,6 +180,24 @@ initialize(void) } +static inline u32 +get_afx_crc (armor_filter_context_t *afx) +{ + const byte *crc_buf; + u32 crc; + + crc_buf = gcry_md_read (afx->crc_md, GCRY_MD_CRC24_RFC2440); + + crc = crc_buf[0]; + crc <<= 8; + crc |= crc_buf[1]; + crc <<= 8; + crc |= crc_buf[2]; + + return crc; +} + + /* * Check whether this is an armored file. See also * parse-packet.c for details on this code. @@ -592,7 +600,7 @@ check_input( armor_filter_context_t *afx, IOBUF a ) afx->faked = 1; else { afx->inp_checked = 1; - afx->crc = CRCINIT; + gcry_md_reset (afx->crc_md); afx->idx = 0; afx->radbuf[0] = 0; } @@ -768,7 +776,7 @@ fake_packet( armor_filter_context_t *afx, IOBUF a, } } afx->inp_checked = 1; - afx->crc = CRCINIT; + gcry_md_reset (afx->crc_md); afx->idx = 0; afx->radbuf[0] = 0; } @@ -797,10 +805,8 @@ radix64_read( armor_filter_context_t *afx, IOBUF a, size_t *retn, int checkcrc=0; int rc = 0; size_t n = 0; - int idx, i, onlypad=0; - u32 crc; + int idx, onlypad=0; - crc = afx->crc; idx = afx->idx; val = afx->radbuf[0]; for( n=0; n < size; ) { @@ -881,14 +887,14 @@ radix64_read( armor_filter_context_t *afx, IOBUF a, size_t *retn, idx = (idx+1) % 4; } - for(i=0; i < n; i++ ) - crc = (crc << 8) ^ crc_table[((crc >> 16)&0xff) ^ buf[i]]; - crc &= 0x00ffffff; - afx->crc = crc; afx->idx = idx; afx->radbuf[0] = val; + if( n ) + gcry_md_write (afx->crc_md, buf, n); + if( checkcrc ) { + gcry_md_final (afx->crc_md); afx->any_data = 1; afx->inp_checked=0; afx->faked = 0; @@ -957,10 +963,10 @@ radix64_read( armor_filter_context_t *afx, IOBUF a, size_t *retn, log_info(_("malformed CRC\n")); rc = invalid_crc(); } - else if( mycrc != afx->crc ) { - log_info (_("CRC error; %06lX - %06lX\n"), - (ulong)afx->crc, (ulong)mycrc); - rc = invalid_crc(); + else if( mycrc != get_afx_crc (afx) ) { + log_info (_("CRC error; %06lX - %06lX\n"), + (ulong)get_afx_crc (afx), (ulong)mycrc); + rc = invalid_crc(); } else { rc = 0; @@ -1188,18 +1194,15 @@ armor_filter( void *opaque, int control, afx->status++; afx->idx = 0; afx->idx2 = 0; - afx->crc = CRCINIT; - + gcry_md_reset (afx->crc_md); } - crc = afx->crc; idx = afx->idx; idx2 = afx->idx2; for(i=0; i < idx; i++ ) radbuf[i] = afx->radbuf[i]; - for(i=0; i < size; i++ ) - crc = (crc << 8) ^ crc_table[((crc >> 16)&0xff) ^ buf[i]]; - crc &= 0x00ffffff; + if( size ) + gcry_md_write (afx->crc_md, buf, size); for( ; size; buf++, size-- ) { radbuf[idx++] = *buf; @@ -1224,7 +1227,6 @@ armor_filter( void *opaque, int control, afx->radbuf[i] = radbuf[i]; afx->idx = idx; afx->idx2 = idx2; - afx->crc = crc; } else if( control == IOBUFCTRL_INIT ) { @@ -1250,7 +1252,8 @@ armor_filter( void *opaque, int control, if( afx->cancel ) ; else if( afx->status ) { /* pad, write cecksum, and bottom line */ - crc = afx->crc; + gcry_md_final (afx->crc_md); + crc = get_afx_crc (afx); idx = afx->idx; idx2 = afx->idx2; if( idx ) { diff --git a/g10/filter.h b/g10/filter.h index fa1f5a24f..b2ef3828f 100644 --- a/g10/filter.h +++ b/g10/filter.h @@ -61,7 +61,7 @@ typedef struct { byte radbuf[4]; int idx, idx2; - u32 crc; + gcry_md_hd_t crc_md; int status; /* an internal state flag */ int cancel; From jussi.kivilinna at iki.fi Sat Oct 27 23:06:00 2018 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sun, 28 Oct 2018 00:06:00 +0300 Subject: [PATCH 4/8] g10/armor: remove unused unarmor_pump code In-Reply-To: <154067434467.18498.15345072060569568612.stgit@localhost.localdomain> References: <154067434467.18498.15345072060569568612.stgit@localhost.localdomain> Message-ID: <154067436018.18498.1905531138008509294.stgit@localhost.localdomain> * g10/armor.c (unarmor_state_e, unarmor_pump_s, unarmor_pump_new) (unarmor_pump_release, unarmor_pump): Remove. * g10/filter.h (UnarmorPump, unarmor_pump_new, unarmor_pump_release) (unarmor_pump): Remove. -- Signed-off-by: Jussi Kivilinna --- 0 files changed diff --git a/g10/armor.c b/g10/armor.c index 1b02a231f..714282f45 100644 --- a/g10/armor.c +++ b/g10/armor.c @@ -1350,221 +1350,3 @@ make_radix64_string( const byte *data, size_t len ) *p = 0; return buffer; } - - -/*********************************************** - * For the pipemode command we can't use the armor filter for various - * reasons, so we use this new unarmor_pump stuff to remove the armor - */ - -enum unarmor_state_e { - STA_init = 0, - STA_bypass, - STA_wait_newline, - STA_wait_dash, - STA_first_dash, - STA_compare_header, - STA_found_header_wait_newline, - STA_skip_header_lines, - STA_skip_header_lines_non_ws, - STA_read_data, - STA_wait_crc, - STA_read_crc, - STA_ready -}; - -struct unarmor_pump_s { - enum unarmor_state_e state; - byte val; - int checkcrc; - int pos; /* counts from 0..3 */ - u32 crc; - u32 mycrc; /* the one store in the data */ -}; - - - -UnarmorPump -unarmor_pump_new (void) -{ - UnarmorPump x; - - if( !is_initialized ) - initialize(); - x = xmalloc_clear (sizeof *x); - return x; -} - -void -unarmor_pump_release (UnarmorPump x) -{ - xfree (x); -} - -/* - * Get the next character from the ascii armor taken from the IOBUF - * created earlier by unarmor_pump_new(). - * Return: c = Character - * 256 = ignore this value - * -1 = End of current armor - * -2 = Premature EOF (not used) - * -3 = Invalid armor - */ -int -unarmor_pump (UnarmorPump x, int c) -{ - int rval = 256; /* default is to ignore the return value */ - - switch (x->state) { - case STA_init: - { - byte tmp[2]; - tmp[0] = c; - tmp[1] = 0; - if ( is_armored (tmp) ) - x->state = c == '-'? STA_first_dash : STA_wait_newline; - else { - x->state = STA_bypass; - return c; - } - } - break; - case STA_bypass: - return c; /* return here to avoid crc calculation */ - case STA_wait_newline: - if (c == '\n') - x->state = STA_wait_dash; - break; - case STA_wait_dash: - x->state = c == '-'? STA_first_dash : STA_wait_newline; - break; - case STA_first_dash: /* just need for initialization */ - x->pos = 0; - x->state = STA_compare_header; /* fall through */ - case STA_compare_header: - if ( "-----BEGIN PGP SIGNATURE-----"[++x->pos] == c ) { - if ( x->pos == 28 ) - x->state = STA_found_header_wait_newline; - } - else - x->state = c == '\n'? STA_wait_dash : STA_wait_newline; - break; - case STA_found_header_wait_newline: - /* to make CR,LF issues easier we simply allow for white space - behind the 5 dashes */ - if ( c == '\n' ) - x->state = STA_skip_header_lines; - else if ( c != '\r' && c != ' ' && c != '\t' ) - x->state = STA_wait_dash; /* garbage after the header line */ - break; - case STA_skip_header_lines: - /* i.e. wait for one empty line */ - if ( c == '\n' ) { - x->state = STA_read_data; - x->crc = CRCINIT; - x->val = 0; - x->pos = 0; - } - else if ( c != '\r' && c != ' ' && c != '\t' ) - x->state = STA_skip_header_lines_non_ws; - break; - case STA_skip_header_lines_non_ws: - /* like above but we already encountered non white space */ - if ( c == '\n' ) - x->state = STA_skip_header_lines; - break; - case STA_read_data: - /* fixme: we don't check for the trailing dash lines but rely - * on the armor stop characters */ - if( c == '\n' || c == ' ' || c == '\r' || c == '\t' ) - break; /* skip all kind of white space */ - - if( c == '=' ) { /* pad character: stop */ - if( x->pos == 1 ) /* in this case val has some value */ - rval = x->val; - x->state = STA_wait_crc; - break; - } - - { - int c2; - if( (c = asctobin[(c2=c)]) == 255 ) { - log_error(_("invalid radix64 character %02X skipped\n"), c2); - break; - } - } - - switch(x->pos) { - case 0: - x->val = c << 2; - break; - case 1: - x->val |= (c>>4)&3; - rval = x->val; - x->val = (c<<4)&0xf0; - break; - case 2: - x->val |= (c>>2)&15; - rval = x->val; - x->val = (c<<6)&0xc0; - break; - case 3: - x->val |= c&0x3f; - rval = x->val; - break; - } - x->pos = (x->pos+1) % 4; - break; - case STA_wait_crc: - if( c == '\n' || c == ' ' || c == '\r' || c == '\t' || c == '=' ) - break; /* skip ws and pad characters */ - /* assume that we are at the next line */ - x->state = STA_read_crc; - x->pos = 0; - x->mycrc = 0; /* fall through */ - case STA_read_crc: - if( (c = asctobin[c]) == 255 ) { - rval = -1; /* ready */ - if( x->crc != x->mycrc ) { - log_info (_("CRC error; %06lX - %06lX\n"), - (ulong)x->crc, (ulong)x->mycrc); - if ( invalid_crc() ) - rval = -3; - } - x->state = STA_ready; /* not sure whether this is correct */ - break; - } - - switch(x->pos) { - case 0: - x->val = c << 2; - break; - case 1: - x->val |= (c>>4)&3; - x->mycrc |= x->val << 16; - x->val = (c<<4)&0xf0; - break; - case 2: - x->val |= (c>>2)&15; - x->mycrc |= x->val << 8; - x->val = (c<<6)&0xc0; - break; - case 3: - x->val |= c&0x3f; - x->mycrc |= x->val; - break; - } - x->pos = (x->pos+1) % 4; - break; - case STA_ready: - rval = -1; - break; - } - - if ( !(rval & ~255) ) { /* compute the CRC */ - x->crc = (x->crc << 8) ^ crc_table[((x->crc >> 16)&0xff) ^ rval]; - x->crc &= 0x00ffffff; - } - - return rval; -} diff --git a/g10/filter.h b/g10/filter.h index 6daf273fa..fa1f5a24f 100644 --- a/g10/filter.h +++ b/g10/filter.h @@ -69,8 +69,6 @@ typedef struct { int pending_lf; /* used together with faked */ } armor_filter_context_t; -struct unarmor_pump_s; -typedef struct unarmor_pump_s *UnarmorPump; struct compress_filter_context_s { @@ -172,9 +170,6 @@ armor_filter_context_t *new_armor_context (void); void release_armor_context (armor_filter_context_t *afx); int push_armor_filter (armor_filter_context_t *afx, iobuf_t iobuf); int use_armor_filter( iobuf_t a ); -UnarmorPump unarmor_pump_new (void); -void unarmor_pump_release (UnarmorPump x); -int unarmor_pump (UnarmorPump x, int c); /*-- compress.c --*/ gpg_error_t push_compress_filter (iobuf_t out, compress_filter_context_t *zfx, From jussi.kivilinna at iki.fi Sat Oct 27 23:06:05 2018 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sun, 28 Oct 2018 00:06:05 +0300 Subject: [PATCH 5/8] common/iobuf: optimize iobuf_read_line In-Reply-To: <154067434467.18498.15345072060569568612.stgit@localhost.localdomain> References: <154067434467.18498.15345072060569568612.stgit@localhost.localdomain> Message-ID: <154067436534.18498.11599770807421210793.stgit@localhost.localdomain> * common/iobuf.c (iobuf_read_line): Add fast path for finding '\n' character in buffer. -- This patch reduce per byte overhead in iobuf_read_line by avoiding using iobuf_get when possible and use memchr to find '\n'. This speeds armored decryption. Benchmark results below, tested on Intel Core i7-4790K (turbo off). Encrypted 2 GiB through pipe to ramfs file using AES128. Decrypt ramfs file out through pipe to /dev/null. before patch-set ---------------- gpg process armor: user time pipe transfer rate encrypt-aead: 13.8 140 MB/s decrypt-aead: 30.6 68 MB/s encrypt-cfb: 17.4 114 MB/s decrypt-cfb: 32.6 64 MB/s after (decrypt+iobuf opt) ------------------------- gpg process armor: user time pipe transfer rate decrypt-aead: 22.5 92 MB/s decrypt-cfb: 24.4 85 MB/s Signed-off-by: Jussi Kivilinna --- 0 files changed diff --git a/common/iobuf.c b/common/iobuf.c index 18a458e0a..5eeba8fe6 100644 --- a/common/iobuf.c +++ b/common/iobuf.c @@ -2610,12 +2610,50 @@ iobuf_read_line (iobuf_t a, byte ** addr_of_buffer, } p = buffer; - while ((c = iobuf_get (a)) != -1) + while (1) { - *p++ = c; - nbytes++; - if (c == '\n') - break; + if (!a->nofast && a->d.start < a->d.len && nbytes < length - 1) + /* Fast path for finding '\n' by using standard C library's optimized + memchr. */ + { + unsigned size = a->d.len - a->d.start; + byte *newline_pos; + + if (size > length - 1 - nbytes) + size = length - 1 - nbytes; + + newline_pos = memchr (a->d.buf + a->d.start, '\n', size); + if (newline_pos) + { + /* Found newline, copy buffer and return. */ + size = (newline_pos - (a->d.buf + a->d.start)) + 1; + memcpy (p, a->d.buf + a->d.start, size); + p += size; + nbytes += size; + a->d.start += size; + a->nbytes += size; + break; + } + else + { + /* No newline, copy buffer and continue. */ + memcpy (p, a->d.buf + a->d.start, size); + p += size; + nbytes += size; + a->d.start += size; + a->nbytes += size; + } + } + else + { + c = iobuf_readbyte (a); + if (c == -1) + break; + *p++ = c; + nbytes++; + if (c == '\n') + break; + } if (nbytes == length - 1) /* We don't have enough space to add a \n and a \0. Increase From jussi.kivilinna at iki.fi Sat Oct 27 23:05:55 2018 From: jussi.kivilinna at iki.fi (Jussi Kivilinna) Date: Sun, 28 Oct 2018 00:05:55 +0300 Subject: [PATCH 3/8] g10/armor: fix eof checks in radix64_read In-Reply-To: <154067434467.18498.15345072060569568612.stgit@localhost.localdomain> References: <154067434467.18498.15345072060569568612.stgit@localhost.localdomain> Message-ID: <154067435501.18498.15551952250668041977.stgit@localhost.localdomain> * g10/armor.c (radix64_read): Check EOF with '!afx->buffer_len' instead of 'c == -1', as 'c' is never set to this value. -- Signed-off-by: Jussi Kivilinna --- 0 files changed diff --git a/g10/armor.c b/g10/armor.c index 98b870ab9..1b02a231f 100644 --- a/g10/armor.c +++ b/g10/armor.c @@ -793,7 +793,7 @@ radix64_read( armor_filter_context_t *afx, IOBUF a, size_t *retn, byte *buf, size_t size ) { byte val; - int c=0, c2; /*init c because gcc is not clever enough for the continue*/ + int c, c2; int checkcrc=0; int rc = 0; size_t n = 0; @@ -911,7 +911,7 @@ radix64_read( armor_filter_context_t *afx, IOBUF a, size_t *retn, continue; break; } - if( c == -1 ) + if( !afx->buffer_len ) log_error(_("premature eof (no CRC)\n")); else { u32 mycrc = 0; @@ -945,7 +945,7 @@ radix64_read( armor_filter_context_t *afx, IOBUF a, size_t *retn, if( !afx->buffer_len ) break; /* eof */ } while( ++idx < 4 ); - if( c == -1 ) { + if( !afx->buffer_len ) { log_info(_("premature eof (in CRC)\n")); rc = invalid_crc(); } From wk at gnupg.org Mon Oct 29 14:45:43 2018 From: wk at gnupg.org (Werner Koch) Date: Mon, 29 Oct 2018 14:45:43 +0100 Subject: [PATCH 1/8] g10/decrypt-data: use fill_buffer in more places In-Reply-To: <154067434467.18498.15345072060569568612.stgit@localhost.localdomain> (Jussi Kivilinna's message of "Sun, 28 Oct 2018 00:05:44 +0300") References: <154067434467.18498.15345072060569568612.stgit@localhost.localdomain> Message-ID: <87efc8j27s.fsf@wheatstone.g10code.de> Hi! All patches look fine to me. After applying we should however run extensive tests against another implementation to see whether we have any breakage. With the current code this has been with the help of the www.rnpgp.com folks. Salam-Shalom, Werner -- Die Gedanken sind frei. Ausnahmen regelt ein Bundesgesetz. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 227 bytes Desc: not available URL: From dkg at fifthhorseman.net Mon Oct 29 18:22:02 2018 From: dkg at fifthhorseman.net (Daniel Kahn Gillmor) Date: Mon, 29 Oct 2018 13:22:02 -0400 Subject: [Announce] Libgcrypt 1.8.4 released In-Reply-To: <87r2gclgju.fsf@wheatstone.g10code.de> References: <87r2gclgju.fsf@wheatstone.g10code.de> Message-ID: <87r2g8is79.fsf@fifthhorseman.net> On Fri 2018-10-26 20:16:21 +0200, Werner Koch wrote: > The GnuPG Project is pleased to announce the availability of Libgcrypt > versions 1.8.4. thanks for this release, Werner! > * Performance: > > - On Linux always make use of getrandom if possible and then use > its /dev/urandom behaviour. [#3894] This characterization is unfortunate, since the getrandom() default behavior is *not* the /dev/urandom behavior. In particular, the getrandom() default behavior blocks until the kernel's internal pool has been fully initialized, while /dev/urandom never blocks. This is one of the main arguments for using getrandom() instead of /dev/urandom in the first place. I appreciate that the actual change landed in libgcrypt! This is a real improvement for users of GNU/Linux systems. I just don't want people to think that the change will cause them to possibly use an uninitialized PRNG like /dev/urandom, because that is not the case with the change that we made here. --dkg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 227 bytes Desc: not available URL: From sergi at calcurco.cat Tue Oct 30 10:35:14 2018 From: sergi at calcurco.cat (=?UTF-8?Q?Sergi_Blanch-Torn=c3=a9?=) Date: Tue, 30 Oct 2018 10:35:14 +0100 Subject: [Announce] Libgcrypt 1.8.4 released In-Reply-To: <87r2gclgju.fsf@wheatstone.g10code.de> References: <87r2gclgju.fsf@wheatstone.g10code.de> Message-ID: Hi, I miss the tag labelling the version. I thought it would be the commit in master that mentions the merge release (f1fe14 Fri Oct 26 20:04:44 2018 +0200) but after the "./autogen.sh", "libgcrypt-config --version" says '1.9.0'. Thanks /Sergi. On 10/26/18 20:16, Werner Koch wrote: > Hi! > > The GnuPG Project is pleased to announce the availability of Libgcrypt > versions 1.8.4. This is a maintenance release to fix a few minor bugs. > > Libgcrypt is a general purpose library of cryptographic building blocks. > It is originally based on code used by GnuPG. It does not provide any > implementation of OpenPGP or other protocols. Thorough understanding of > applied cryptography is required to use Libgcrypt. > > > Noteworthy changes in version 1.8.4 > =================================== > > * Bug fixes: > > - Fix infinite loop due to applications using fork the wrong > way. [#3491] > > - Fix possible leak of a few bits of secret primes to pageable > memory. [#3848] > > - Fix possible hang in the RNG (1.8.3 only). [#4034] > > - Several minor fixes. [#4102,#4208,#4209,#4210,#4211,#4212] > > * Performance: > > - On Linux always make use of getrandom if possible and then use > its /dev/urandom behaviour. [#3894] > > > Download > ======== > > Source code is hosted at the GnuPG FTP server and its mirrors as listed > at . On the primary server > the source tarball and its digital signature are: > > https://gnupg.org/ftp/gcrypt/libgcrypt/libgcrypt-1.8.4.tar.bz2 > https://gnupg.org/ftp/gcrypt/libgcrypt/libgcrypt-1.8.4.tar.bz2.sig > > or gzip compressed: > > https://gnupg.org/ftp/gcrypt/libgcrypt/libgcrypt-1.8.4.tar.gz > https://gnupg.org/ftp/gcrypt/libgcrypt/libgcrypt-1.8.4.tar.gz.sig > > In order to check that the version of Libgcrypt you downloaded is an > original and unmodified file please follow the instructions found at > . In short, you may > use one of the following methods: > > - Check the supplied OpenPGP signature. For example to check the > signature of the file libgcrypt-1.8.4.tar.bz2 you would use this > command: > > gpg --verify libgcrypt-1.8.4.tar.bz2.sig libgcrypt-1.8.4.tar.bz2 > > This checks whether the signature file matches the source file. > You should see a message indicating that the signature is good and > made by one or more of the release signing keys. Make sure that > this is a valid key, either by matching the shown fingerprint > against a trustworthy list of valid release signing keys or by > checking that the key has been signed by trustworthy other keys. > See the end of this mail for information on the signing keys. > > - If you are not able to use an existing version of GnuPG, you have > to verify the SHA-1 checksum. On Unix systems the command to do > this is either "sha1sum" or "shasum". Assuming you downloaded the > file libgcrypt-1.8.4.tar.bz2, you run the command like this: > > sha1sum libgcrypt-1.8.4.tar.bz2 > > and check that the output matches the first line from the > this list: > > 4a8ef9db6922f3a31992aca5640b4198a69b58fc libgcrypt-1.8.4.tar.bz2 > 211855f39f3bc3c4a4f444d4c09d743dfc5cb427 libgcrypt-1.8.4.tar.gz > > You should also verify that the checksums above are authentic by > matching them with copies of this announcement. Those copies can be > found at other mailing lists, web sites, and search engines. > > > Copying > ======= > > Libgcrypt is distributed under the terms of the GNU Lesser General > Public License (LGPLv2.1+). The helper programs as well as the > documentation are distributed under the terms of the GNU General Public > License (GPLv2+). The file LICENSES has notices about contributions > that require that these additional notices are distributed. > > > Support > ======= > > In case of build problems specific to this release please first check > https://dev.gnupg.org/T4234 for updated information. > > For help on developing with Libgcrypt you should read the included > manual and optional ask on the gcrypt-devel mailing list [1]. A > listing with commercial support offers for Libgcrypt and related > software is available at the GnuPG web site [2]. > > If you are a developer and you may need a certain feature for your > project, please do not hesitate to bring it to the gcrypt-devel > mailing list for discussion. > > > Thanks > ====== > > Maintenance and development of GnuPG is mostly financed by donations. > The GnuPG project currently employs one full-time developer and two > contractors. They all work exclusively on GnuPG and closely related > software like Libgcrypt, GPGME, and GPA. > > We have to thank all the people who helped the GnuPG project, be it > testing, coding, translating, suggesting, auditing, administering the > servers, spreading the word, and answering questions on the mailing > lists. Thanks to Tomas Mraz for pointing out several smaller flaws. > > Many thanks to our numerous financial supporters, both corporate and > individuals. Without you it would not be possible to keep GnuPG in a > good shape and address all the small and larger requests made by our > users. Thanks. > > > Happy hacking, > > Your GnuPG hackers > > > > p.s. > This is an announcement only mailing list. Please send replies only to > the gnupg-users'at'gnupg.org mailing list. > > p.p.s > List of Release Signing Keys: > > To guarantee that a downloaded GnuPG version has not been tampered by > malicious entities we provide signature files for all tarballs and > binary versions. The keys are also signed by the long term keys of > their respective owners. Current releases are signed by one or more > of these four keys: > > rsa2048 2011-01-12 [expires: 2019-12-31] > Key fingerprint = D869 2123 C406 5DEA 5E0F 3AB5 249B 39D2 4F25 E3B6 > Werner Koch (dist sig) > > rsa2048 2014-10-29 [expires: 2019-12-31] > Key fingerprint = 46CC 7308 65BB 5C78 EBAB ADCF 0437 6F3E E085 6959 > David Shaw (GnuPG Release Signing Key) > > rsa2048 2014-10-29 [expires: 2020-10-30] > Key fingerprint = 031E C253 6E58 0D8E A286 A9F2 2071 B08A 33BD 3F06 > NIIBE Yutaka (GnuPG Release Key) > > rsa3072 2017-03-17 [expires: 2027-03-15] > Key fingerprint = 5B80 C575 4298 F0CB 55D8 ED6A BCEF 7E29 4B09 2E28 > Andre Heinecke (Release Signing Key) > > The keys are available at and > in any recently released GnuPG tarball in the file g10/distsigkey.gpg . > Note that this mail has been signed by a different key. > > > > _______________________________________________ > Gnupg-announce mailing list > Gnupg-announce at gnupg.org > http://lists.gnupg.org/mailman/listinfo/gnupg-announce > > > > _______________________________________________ > Gnupg-users mailing list > Gnupg-users at gnupg.org > http://lists.gnupg.org/mailman/listinfo/gnupg-users > -- Sergi Blanch-Torn? C797 490A C93E DB00 0615 680A FC1C 1CB6 8079 85E6 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 659 bytes Desc: OpenPGP digital signature URL: From dkg at fifthhorseman.net Tue Oct 30 14:29:03 2018 From: dkg at fifthhorseman.net (Daniel Kahn Gillmor) Date: Tue, 30 Oct 2018 09:29:03 -0400 Subject: [Announce] Libgcrypt 1.8.4 released In-Reply-To: References: <87r2gclgju.fsf@wheatstone.g10code.de> Message-ID: <87muqvftr4.fsf@fifthhorseman.net> On Tue 2018-10-30 10:35:14 +0100, Sergi Blanch-Torn? wrote: > I miss the tag labelling the version. I thought it would be the commit > in master that mentions the merge release (f1fe14 Fri Oct 26 20:04:44 > 2018 +0200) but after the "./autogen.sh", "libgcrypt-config --version" > says '1.9.0'. It looks to me that libgcrypt-1.8.4 should be tagged at 93775172713c00c363187b5d6a88895b04ac7c8e, which is on the branch named LIBGCRYPT-1.8-BRANCH. i think f1fe14 is on the master branch, and is not what you're looking for. I agree that the tag should have been published to https://dev.gnupg.org/source/libgcrypt.git at the same time as the new version was published and announced, though -- some step of the release process probably isn't as automated as it should be. --dkg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 227 bytes Desc: not available URL: From sergi at calcurco.cat Tue Oct 30 15:21:56 2018 From: sergi at calcurco.cat (=?UTF-8?Q?Sergi_Blanch-Torn=c3=a9?=) Date: Tue, 30 Oct 2018 15:21:56 +0100 Subject: [Announce] Libgcrypt 1.8.4 released In-Reply-To: <87muqvftr4.fsf@fifthhorseman.net> References: <87r2gclgju.fsf@wheatstone.g10code.de> <87muqvftr4.fsf@fifthhorseman.net> Message-ID: <97dcee6f-1115-fdf9-03dc-c1297879f18b@calcurco.cat> Hi, Yes Daniel, I went to master instead of the 1.8 branch. By the way, the commit you mention gives me version '1.8.4-beta14'. Perhaps there is something else apart of the tagging. /Sergi. On 10/30/18 14:29, Daniel Kahn Gillmor wrote: > On Tue 2018-10-30 10:35:14 +0100, Sergi Blanch-Torn? wrote: >> I miss the tag labelling the version. I thought it would be the commit >> in master that mentions the merge release (f1fe14 Fri Oct 26 20:04:44 >> 2018 +0200) but after the "./autogen.sh", "libgcrypt-config --version" >> says '1.9.0'. > > It looks to me that libgcrypt-1.8.4 should be tagged at > 93775172713c00c363187b5d6a88895b04ac7c8e, which is on the branch named > LIBGCRYPT-1.8-BRANCH. > > i think f1fe14 is on the master branch, and is not what you're looking > for. > > I agree that the tag should have been published to > https://dev.gnupg.org/source/libgcrypt.git at the same time as the new > version was published and announced, though -- some step of the release > process probably isn't as automated as it should be. > > --dkg > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 659 bytes Desc: OpenPGP digital signature URL: From wk at gnupg.org Tue Oct 30 15:33:03 2018 From: wk at gnupg.org (Werner Koch) Date: Tue, 30 Oct 2018 15:33:03 +0100 Subject: [Announce] Libgcrypt 1.8.4 released In-Reply-To: <87r2g8is79.fsf@fifthhorseman.net> (Daniel Kahn Gillmor's message of "Mon, 29 Oct 2018 13:22:02 -0400") References: <87r2gclgju.fsf@wheatstone.g10code.de> <87r2g8is79.fsf@fifthhorseman.net> Message-ID: <87y3afec80.fsf@wheatstone.g10code.de> On Mon, 29 Oct 2018 18:22, dkg at fifthhorseman.net said: > This characterization is unfortunate, since the getrandom() default > behavior is *not* the /dev/urandom behavior. In particular, the I used to explain that getrandom uses the urandom pool and not the random pool. They are separate and have different properties. See Stephan M?ller's paper he wrote for the BSI. > getrandom() default behavior blocks until the kernel's internal pool has > been fully initialized, while /dev/urandom never blocks. This is one of That is detail and could even be viewed as a bug fix for the open("/dev/urandom") behaviour. Which in the early days of Linux was also different. > think that the change will cause them to possibly use an uninitialized > PRNG like /dev/urandom, because that is not the case with the change Those who are using Libgcrypt in the early boot phase should be aware of the problem with modern Linux kernels. Again, see the paper. Shalom-Salam, Werner -- Die Gedanken sind frei. Ausnahmen regelt ein Bundesgesetz. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 227 bytes Desc: not available URL: From wk at gnupg.org Tue Oct 30 15:35:20 2018 From: wk at gnupg.org (Werner Koch) Date: Tue, 30 Oct 2018 15:35:20 +0100 Subject: [Announce] Libgcrypt 1.8.4 released In-Reply-To: <87muqvftr4.fsf@fifthhorseman.net> (Daniel Kahn Gillmor's message of "Tue, 30 Oct 2018 09:29:03 -0400") References: <87r2gclgju.fsf@wheatstone.g10code.de> <87muqvftr4.fsf@fifthhorseman.net> Message-ID: <87pnvrec47.fsf@wheatstone.g10code.de> On Tue, 30 Oct 2018 14:29, dkg at fifthhorseman.net said: > version was published and announced, though -- some step of the release > process probably isn't as automated as it should be. Right. This can't be automated because it requires to to find the token plug it in and enter the PIN. This is mostly driven by a make target but the pushing is done manually to allow for a final check. Shalom-Salam, Werner -- Die Gedanken sind frei. Ausnahmen regelt ein Bundesgesetz. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 227 bytes Desc: not available URL: From dkg at fifthhorseman.net Tue Oct 30 17:29:18 2018 From: dkg at fifthhorseman.net (Daniel Kahn Gillmor) Date: Tue, 30 Oct 2018 12:29:18 -0400 Subject: [Announce] Libgcrypt 1.8.4 released In-Reply-To: <87y3afec80.fsf@wheatstone.g10code.de> References: <87r2gclgju.fsf@wheatstone.g10code.de> <87r2g8is79.fsf@fifthhorseman.net> <87y3afec80.fsf@wheatstone.g10code.de> Message-ID: <8736snflep.fsf@fifthhorseman.net> On Tue 2018-10-30 15:33:03 +0100, Werner Koch wrote: > On Mon, 29 Oct 2018 18:22, dkg at fifthhorseman.net said: > >> This characterization is unfortunate, since the getrandom() default >> behavior is *not* the /dev/urandom behavior. In particular, the > > I used to explain that getrandom uses the urandom pool and not the > random pool. They are separate and have different properties. See > Stephan M?ller's paper he wrote for the BSI. right, but the release notes say it uses the /dev/urandom *behavior*, not the urandom *pool*. the /dev/urandom behavior is still: a) whether the urandom pool is initialized or not, use it. but what getrandom() does is: b) wait for the urandom pool to be initialized, and then use it. >> getrandom() default behavior blocks until the kernel's internal pool has >> been fully initialized, while /dev/urandom never blocks. This is one of > > That is detail and could even be viewed as a bug fix for the > open("/dev/urandom") behaviour. Which in the early days of Linux was > also different. that is certainly detail, but it is the relevant detail here :) > Those who are using Libgcrypt in the early boot phase should be aware of > the problem with modern Linux kernels. Again, see the paper. I'm assuming that "the early boot phase" means "until the crng is initialized". On minimalist virtual machines that use a modern system supervisor that has relatively short boot times, it can be a surprisingly long time after system boot that the crng is initialized. I suspect that there are users of libgcrypt who don't know that their code is being run in this sense of "in the early boot phase". At any rate, the change in gcrypt 1.8.4 actually *helps* those users, rather than harms them, because it removes unnecessary blocking while avoiding exposing the user to use of an unintialized RNG. so as long as they're running a modern kernel, they don't need to read the paper :) thanks for making the change! --dkg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 227 bytes Desc: not available URL: From dkg at fifthhorseman.net Tue Oct 30 17:31:10 2018 From: dkg at fifthhorseman.net (Daniel Kahn Gillmor) Date: Tue, 30 Oct 2018 12:31:10 -0400 Subject: [Announce] Libgcrypt 1.8.4 released In-Reply-To: <97dcee6f-1115-fdf9-03dc-c1297879f18b@calcurco.cat> References: <87r2gclgju.fsf@wheatstone.g10code.de> <87muqvftr4.fsf@fifthhorseman.net> <97dcee6f-1115-fdf9-03dc-c1297879f18b@calcurco.cat> Message-ID: <87zhuve6r5.fsf@fifthhorseman.net> On Tue 2018-10-30 15:21:56 +0100, Sergi Blanch-Torn? wrote: > Yes Daniel, I went to master instead of the 1.8 branch. By the way, the > commit you mention gives me version '1.8.4-beta14'. Perhaps there is > something else apart of the tagging. until the tag is published (it has been made and published now), the auto-versioning done by configure.ac will assume that you're working on a beta release. Please try again after a "git remote update"! --dkg From dkg at fifthhorseman.net Tue Oct 30 17:32:35 2018 From: dkg at fifthhorseman.net (Daniel Kahn Gillmor) Date: Tue, 30 Oct 2018 12:32:35 -0400 Subject: [Announce] Libgcrypt 1.8.4 released In-Reply-To: <87pnvrec47.fsf@wheatstone.g10code.de> References: <87r2gclgju.fsf@wheatstone.g10code.de> <87muqvftr4.fsf@fifthhorseman.net> <87pnvrec47.fsf@wheatstone.g10code.de> Message-ID: <87wopze6os.fsf@fifthhorseman.net> On Tue 2018-10-30 15:35:20 +0100, Werner Koch wrote: > On Tue, 30 Oct 2018 14:29, dkg at fifthhorseman.net said: > >> version was published and announced, though -- some step of the release >> process probably isn't as automated as it should be. > > Right. This can't be automated because it requires to to find the token > plug it in and enter the PIN. This is mostly driven by a make target > but the pushing is done manually to allow for a final check. it sounds like the tag can't be created automatically. that makes sense, thanks for explaining it! But perhaps whatever tools do an automated push/publication of the release could refuse to run if no signed tag is present and available to be pushed? --dkg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 227 bytes Desc: not available URL: From alon.barlev at gmail.com Tue Oct 30 19:54:41 2018 From: alon.barlev at gmail.com (Alon Bar-Lev) Date: Tue, 30 Oct 2018 20:54:41 +0200 Subject: [PATCH Libgpg-error] gpg-error.m4: support pkg-config In-Reply-To: References: <20181011182720.11991-1-alon.barlev@gmail.com> <871s8vptg1.fsf@wheatstone.g10code.de> <87efcokjn6.fsf@wheatstone.g10code.de> Message-ID: Hello Werner, I will appreciate if we can keep this thread alive to help us improve the usability of the build system. Thanks! On Sat, Oct 20, 2018 at 11:59 AM Alon Bar-Lev wrote: > > Hi Werner, > > First of all I must say that I fully agree with you that pkg-config is not the idle implementation, yes, I can think of many improvements and suggestions. However, the question... is it good enough. > > The fact that you plan to publish pkg-config metadata as "second class citizen", suggests that we already understand that we support the lowest common denominator, which is pkg-config capabilities. At the next ABI break (which is as you pointed one of the missing features of pkg-config. Well, apart of the obvious provide a new metadata name which you do not like) you will have to consider both script based and pkg-config implications on your consumers. > > I am still waiting you to address the actual patch... Provided gnupg libraries already publish pkg-config metadata, this trivial patch provides the option to select either mode, while keeping current approach as the default. Everybody wins... you keep upstream build pkg-config free whenever possible, while providing downstream and users to use pkg-config if there is a benefit of doing so. > > For your question about the current config scripts and cross compile, I already provided some missing bits... Here they are again: > > 1. Executables and/or scripts within SYSROOT is to be used only by target machine, when building nothing should be executed out of SYSROOT. [[pkg-config solves this by reading metadata out of the SYSROOT]] > > 2. The tools should be installed as ${HOST}-${TOOL}, so that multiple modes can be supported. This applies also for non-cross compile, such as multilib. [[pkg-config solves this by installing the metadata under the libdir]] > > 3. Tools should be SYSROOT aware, and output paths based on SYSROOT. [pkg-config solves this by adding PKG_CONFIG_SYSROOT_DIR as prefix to absolute paths]] > > 4. In autoconf you should use AC_*_TOOL instead of AC_*_PROG to locate these configuration scripts, this will search for a tool prefixed with ${HOST}-, it will find the right script per the target host, and it is the standard autoconf practice for build tools. [[pkg-config solves this by reading metadata from SYSROOT:libdir]] > > 5. The standard method would be to install ${HOST}-${TOOL} at SYSROOT/*/bin per (2) to serve the target system, while installing a ${HOST}-${TOOL} at build /*/bin with SYSROOT consideration, default where SYSROOT was when building the package. This will allow autoconf (4) to find it without any tweaks, per autoconf best practices. > > 5. Instead of (5), you can tweak the AC_*_TOOL (4) will search the SYSROOT/*/bin for the tools. However, you must still support the ${HOST}- prefix (2). This option still violate (1) the fact that nothing from SYSROOT should be executed during cross-compile, and should be avoided. > > Supporting pkg-config as an option solves these issues without adding more logic into scripts/autoconf/automake/users with this simple patch. You can decide in future to solve this properly also using the config scripts. > > Even if you decide to solve this in config scripts, and based on the fact that the config scripts already read pkg-config metadata... Probably the simplest option is to merge all scripts into a single script as the logic is the same, then provide a pkg-config compatible interface for this script, in other words, create a mini-pkg-config, so that autoconf might fallback to this script when pkg-config is not available. > > Regardless of the discussion of how to improve the config scripts, can you please consider to add the pkg-config as an optional method? > > Regards, > Alon > > > On Wed, Oct 17, 2018 at 12:30 PM Werner Koch wrote: >> >> On Sat, 13 Oct 2018 00:07, alon.barlev at gmail.com said: >> >> > make was introduced to manage a set of simple rules to avoid re-build >> > when possible, then realized that for large project it is difficult to >> >> make was also devised in 1976 as a simple form of dependency tracker to >> make sure that changed source file won't go unnoticed. >> >> > autoconf was introduced to generate files from templates based on >> > logic as it was difficult to add functional logic into make files. >> >> That more describes imake. autoconf impelements the GNU strategy to >> first test a system for features and the use only standard style macros >> to make use of system specific features. This avoided the often deep >> nested ifdef chhains you still see in some software. >> >> > automake was introduced to provide a simple method to generate make >> > files that actually work with less error prune syntax, supporting >> >> Right. And to make sure that the required targets are always availabale >> (e.g. make distcheck). >> >> > These concerns could have been address using metadata or scripts, at >> > first there was no standard for metadata, so these programs that >> >> This is why autoconf runs tests. The meta data provided by libraries >> are actually not tests but hints on how they were configured. >> >> > selection to manage the metadata, having consistent behavior among >> > packages, not running anything from sysroot - all is important to >> >> Why should one not run something from sysroot? POSIX shell scripts are >> well suited to be run on all platforms, be it on the build system or or >> final host (after installation or shared with an emulator). >> >> > The gnupg projects already use them all, make, autoconf, automake, >> > libtool and pkg-config, so let's at least count them all and >> >> pkg-config only becuase some external packages provide only pkg-config >> stuff. >> >> > When I saw libgpg-error master publishes pkg-config metadata, I was >> > very happy, as it does show some new openness, as I know your point of >> > view. You also stated that you want pkg-config to be second class >> > option, so I introduced the enable_pkg_config flag with default no, to >> >> That is the whole poing with avoiding a second build system. People >> will soon start to use that alternative system and as maintainers we run >> in all kind of problems because it is assumed tha this is a supported >> way of using a library. >> >> > keep backward compatibility. Using the pkg-config resolves issues of >> > multilib and cross-compile without need to modify the existing >> >> I still can't see why this is the case. SYSROOT support is in our >> libraries since 2014 and makes it really easy to use a cross-build >> library: The foo-config script is run as $SYSROOT/bin/foo-config where >> the make install of the library has stored it. >> >> > 4. Per each component a config script is being maintained. >> >> Which is as easy as writing a pc file if not easier. >> >> > 5. The script is different per project. >> >> Sure, it describes the configuraion. >> >> > 6. In libgpg-error master the script is a wrapper on top of pkg-config >> > metadata which implies that the pkg-config metadata is formally >> > maintained. >> >> That is a technical detail on how the "second class citizen" foo.pc is >> implemented. >> >> > 8. pkg-config and pkg.m4 are already used by the following projects: >> > $ find . -name 'pkg.m4' | sort >> > ./gnupg/m4/pkg.m4 >> > ./gpgme/m4/pkg.m4 >> > ./pinentry/m4/pkg.m4 >> >> As well as dozens of other m4 files to test for system features. >> >> > the existing script. I do not see a difference between pkg-config and >> > script: >> > a. it can detect a package based on component version expression >> >> But that is all what you get. A script is much more powerful than a >> static description language and the script interpreter is a standard >> Unix tool on _all_ Unix platforms. In contrast pgk-config requires to >> build and install an extra tool before you can start. >> >> > c. if a severe ABI breakage is happening project can modify the name >> > of the metadata for side-by-side or even install multiple metadata >> > each with different setting >> >> Changing the project name to declare an ABI break. There are better >> ways. In fact pkg-config could also support this by not only tracking a >> version but also an ABI counter. AFAIK, it does not do that. >> >> >> foo-config) stuff makes a lot of sense. For libraries with a maintained >> >> API and ABI a simpler, more portable but also harder to initially create >> >> dedicated config file is a cleaner approach. >> > >> > I do not understand this argument, I will appreciate an example. >> >> If you look at really old foo-config scripts (they were introduced by >> Gtk+) you will see a lot of checks done by those scripts to cope with >> the fact that developers did not know how to properly design and >> maintain libraries. The foo-config scripts tried to detect that and >> warned the developer/user. Given the complexity of those scripts it was >> natural to unify that and we finally got to pkg-config. Large projects >> like GNOME or KDE put a lot of code in libraries and view them similar >> as programs and change the as they like. Those libraries are not >> re-usable but other software unless this other software's authors want >> to track the development of the umbrella project of a used library. In >> contrast other libraries are written in a way to make sure that other >> software can rely on the interface of such a library. For those >> libraries it is a negligible effort to maintain even a complicated >> foo-config script compared to what work is needed for all the other >> interface maintenance. >> >> > 3. for cross-compile, install these config scripts outside of the >> > sysroot, as it makes no sense to execute anything from sysroot of >> > other architecture >> >> As explained above I take another view here. Running config scripts is >> portable accorsoo platforms. >> >> > 4. for cross-compile, the tool need to be familiar with sysroot >> > concept to output paths relative to sysroot >> >> Agreed and implemented. Where do you see the bug in the SYSROOT support >> of our foo-configs? >> >> > I only expected to trigger discussion, I did not expected you just >> > merge it as is :) >> >> Worked. >> >> >> Salam-Shalom, >> >> Werner >> >> -- >> Die Gedanken sind frei. Ausnahmen regelt ein Bundesgesetz. From wk at gnupg.org Tue Oct 30 22:22:49 2018 From: wk at gnupg.org (Werner Koch) Date: Tue, 30 Oct 2018 22:22:49 +0100 Subject: [Announce] Libgcrypt 1.8.4 released In-Reply-To: <8736snflep.fsf@fifthhorseman.net> (Daniel Kahn Gillmor's message of "Tue, 30 Oct 2018 12:29:18 -0400") References: <87r2gclgju.fsf@wheatstone.g10code.de> <87r2g8is79.fsf@fifthhorseman.net> <87y3afec80.fsf@wheatstone.g10code.de> <8736snflep.fsf@fifthhorseman.net> Message-ID: <878t2fdt92.fsf@wheatstone.g10code.de> On Tue, 30 Oct 2018 17:29, dkg at fifthhorseman.net said: > right, but the release notes say it uses the /dev/urandom *behavior*, > not the urandom *pool*. the /dev/urandom behavior is still: Come on, why this nitpicking for release notes virtually nobody reads. The use of the /dev/urandom (blocking or not) is the real change because it changes the security properties we assume in certain use cases of GnuPG (gpg4vs-nfd project). This is not a technical but a security policy question. > I'm assuming that "the early boot phase" means "until the crng is > initialized". On minimalist virtual machines that use a modern system > supervisor that has relatively short boot times, it can be a Even with a full failing /dev/urandom we still have sufficient entropy From RDRAND and the JitterRNG. In fact the JitterRNG is the only measurable and thus valid entropy source we have on Windows. And we use it on Linux as well for some fraction of the overall entropy fed into Libgcrypt's pool. > rather than harms them, because it removes unnecessary blocking while > avoiding exposing the user to use of an unintialized RNG. so as long as There won't be an uninitialized RNG just due to a failing /dev/random in 1.8. Shalom-Salam, Werner -- Die Gedanken sind frei. Ausnahmen regelt ein Bundesgesetz. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 227 bytes Desc: not available URL: From gniibe at fsij.org Wed Oct 31 00:30:48 2018 From: gniibe at fsij.org (NIIBE Yutaka) Date: Wed, 31 Oct 2018 08:30:48 +0900 Subject: [PATCH Libgpg-error] gpg-error.m4: support pkg-config In-Reply-To: References: <20181011182720.11991-1-alon.barlev@gmail.com> <871s8vptg1.fsf@wheatstone.g10code.de> <87efcokjn6.fsf@wheatstone.g10code.de> Message-ID: <8736snt3kn.fsf@fsij.org> Alon Bar-Lev wrote: > I will appreciate if we can keep this thread alive to help us improve > the usability of the build system. Not directly related to your purpose of introducing pkg-config, but since it is strongly related, let me explain my work these days. FWIW, I pushed my effort to offer .pc files by GnuPG libraries (npth, libgpg-error, libgcrypt, libassuan, libksba, and ntbtls), and to enable gpgrt-config handle .pc file. Now, GnuPG master can be cross built with no SYSROOT nor --with-*-prefix (given libraries of master installed), for major cases like Debian multiarch (e.g., i686-linux-gnu on x86_64-linux-gnu), for Windows, and possibly for Gentoo/Fedora/Arch -style multilib. I mean, just configure with --host=* (and --libdir=*) option. My focus for this change is for GnuPG build. There is no change for GnuPG requirements. >From my viewpoint, it will be a kind of side effect when those libraries can be used by pkg-config. -- From alon.barlev at gmail.com Wed Oct 31 01:57:40 2018 From: alon.barlev at gmail.com (Alon Bar-Lev) Date: Wed, 31 Oct 2018 02:57:40 +0200 Subject: [PATCH Libgpg-error] gpg-error.m4: support pkg-config In-Reply-To: <8736snt3kn.fsf@fsij.org> References: <20181011182720.11991-1-alon.barlev@gmail.com> <871s8vptg1.fsf@wheatstone.g10code.de> <87efcokjn6.fsf@wheatstone.g10code.de> <8736snt3kn.fsf@fsij.org> Message-ID: On Wed, Oct 31, 2018 at 1:30 AM NIIBE Yutaka wrote: > > Alon Bar-Lev wrote: > > I will appreciate if we can keep this thread alive to help us improve > > the usability of the build system. > > Not directly related to your purpose of introducing pkg-config, but > since it is strongly related, let me explain my work these days. > > FWIW, I pushed my effort to offer .pc files by GnuPG libraries (npth, > libgpg-error, libgcrypt, libassuan, libksba, and ntbtls), and > to enable gpgrt-config handle .pc file. > > Now, GnuPG master can be cross built with no SYSROOT nor --with-*-prefix > (given libraries of master installed), for major cases like Debian > multiarch (e.g., i686-linux-gnu on x86_64-linux-gnu), for Windows, and > possibly for Gentoo/Fedora/Arch -style multilib. I mean, just configure > with --host=* (and --libdir=*) option. > > My focus for this change is for GnuPG build. There is no change for > GnuPG requirements. > > From my viewpoint, it will be a kind of side effect when those libraries > can be used by pkg-config. Hi, Thanks for the description. However, this is not answering the concern raised in this thread, nor this pkg-config support will be usable for the main consumer of gnupg packages which are the gnupg project. The build is broken as I described, exposing pkg-config to other packages will not solve the issues of the gnupg component. Please go over the last message and see that even if you want to use config scripts these scripts should be prefixed by $HOST, should be installed outside of $SYSROOT and be found by AC_TOOL_*. I would like to understand why a simple mode of gnupg packages to leverage the work you are doing of exposing pkg-config file should not be applied. The patch I've created is a trivial patch that is enabled only when user explicitly enable pkg-config support, this will solve all the current issues of the build system without having to fix them for these users who care (downstream maintainers, cross-compile developers). This thread is about modifying the build so that it will have an option to use pkg-config instead of config scripts if downstream maintainer chooses to. Regards, Alon From gniibe at fsij.org Wed Oct 31 07:14:16 2018 From: gniibe at fsij.org (NIIBE Yutaka) Date: Wed, 31 Oct 2018 15:14:16 +0900 Subject: [PATCH Libgpg-error] gpg-error.m4: support pkg-config In-Reply-To: References: <20181011182720.11991-1-alon.barlev@gmail.com> <871s8vptg1.fsf@wheatstone.g10code.de> <87efcokjn6.fsf@wheatstone.g10code.de> <8736snt3kn.fsf@fsij.org> Message-ID: <87d0rqod6v.fsf@fsij.org> Alon Bar-Lev wrote: > This thread is about modifying the build so that it will have an > option to use pkg-config instead of config scripts if downstream > maintainer chooses to. I know. I don't argue this point. I pointed out that the important technical things (no use of --with-*-prefix/SYSROOT with *-config scripts, to build GnuPG and its related libraries) can be already achieved with no new dependency to pkg-config. (I mean, gnupg in master. It's not yet applied to stable branch.) My point is, people don't need to use libraries' *-config scripts any more, but only a single gpgrt-config script now. In the past, there are *-config script from libraries, and for some build environment, there are even prefixed versions for supported hosts, like i686-linux-gnu-gpg-error-config and sh4-linux-gnu-libassuan-config. Now, we only have a gpgrt-config. Still *-config script in each library is not removed, so that it can still support existing software, but it's no need any more when we build gnupg master. --