Proposal: New keydb format

[Daniel Kahn Gillmor]

Hi Neal--

On Thu 2015-10-29 06:52:46 -0400, "Neal H. Walfield" <neal at walfield.org> wrote:
> Some people have complained that GnuPG is slow when accessing the key
> db (e.g., [1]).  To mitigate this, GnuPG has some ugly caches (commit
> 492792), which are not completely transparent and introduce subtle
> bugs that are very hard to find (see, e.g., commits 60bc518 and
> ee7ec12, which demonstrate a cache inconsistency).
>
> Werner was the opinion that the performance problem is due to
> signature verification and not due to DB accesses.  To test this
> theory, I did some measurements yesterday and I implemented a small
> prototype, which uses SQLite as the underlying storage engine.  It can
> be retrieved from git on the neal/next branch.

Wow, thank you for doing this work.  I'm really glad to see it!

>   [1] https://bugs.gnupg.org/gnupg/issue2019
>
> Here are some timing results.  I use the Debian keyring
> (/usr/share/keyrings/debian-keyring.gpg), which contains 751 keys.
>
> Importing the keyring: 2 minutes (keybox) vs. 8 seconds (new):
>
>   $ rm pubring.kdx; time gpg2 --no-default-keyring --primary-keyring
>   gnupg-kbx:pubring.kdx --import debian-keyring.gpg >/dev/null
>   gpg: Total number processed: 751
>   gpg:               imported: 751
>   gpg: public key of ultimately trusted key 2183839A not found
>   gpg: public key of ultimately trusted key BC15C85A not found
>   gpg: public key of ultimately trusted key EE37CF96 not found

Out of curiosity, did you try these with --no-auto-check-trustdb?  the
three lines above suggest that you have some sort of ownertrust in use,
so some of the time spent might be spent doing an automatic trustdb
rebuild, which would confuse the measurements a bit.


>   real	1m52.560s
>   user	0m6.268s
>   sys	0m31.604s

One interesting thing to note here is that the time spent by the CPU in
userspace and the kernel is 38s -- the real ("wall") time is over three
times that.  This suggests that something is going on with your I/O
subsystem (or that gpg itself is thrashing the kernel's disk I/O
subsystem).

still, 38s compared to the 6s from the kdb run is a signficant
difference if we can rule out the trustdb check:

>   real	0m7.729s
>   user	0m5.404s
>   sys	0m0.332s


Have you tried more delicate actions like concurrent updates or keyring
merges?

        --dkg