Proposal: New keydb format
Neal H. Walfield
neal at walfield.org
Fri Oct 30 11:11:27 CET 2015
At Fri, 30 Oct 2015 05:25:35 +0900,
Daniel Kahn Gillmor wrote:
> On Thu 2015-10-29 06:52:46 -0400, "Neal H. Walfield" <neal at walfield.org> wrote:
> > Some people have complained that GnuPG is slow when accessing the key
> > db (e.g., ). To mitigate this, GnuPG has some ugly caches (commit
> > 492792), which are not completely transparent and introduce subtle
> > bugs that are very hard to find (see, e.g., commits 60bc518 and
> > ee7ec12, which demonstrate a cache inconsistency).
> > Werner was the opinion that the performance problem is due to
> > signature verification and not due to DB accesses. To test this
> > theory, I did some measurements yesterday and I implemented a small
> > prototype, which uses SQLite as the underlying storage engine. It can
> > be retrieved from git on the neal/next branch.
> Wow, thank you for doing this work. I'm really glad to see it!
Out of curiosity, are you suggesting that the performance of kbx is a
real problem for you in practice? If so, I'd like to understand your
setup a bit better. (Feel free to reply off-list.)
> >  https://bugs.gnupg.org/gnupg/issue2019
> > Here are some timing results. I use the Debian keyring
> > (/usr/share/keyrings/debian-keyring.gpg), which contains 751 keys.
> > Importing the keyring: 2 minutes (keybox) vs. 8 seconds (new):
> > $ rm pubring.kdx; time gpg2 --no-default-keyring --primary-keyring
> > gnupg-kbx:pubring.kdx --import debian-keyring.gpg >/dev/null
> > gpg: Total number processed: 751
> > gpg: imported: 751
> > gpg: public key of ultimately trusted key 2183839A not found
> > gpg: public key of ultimately trusted key BC15C85A not found
> > gpg: public key of ultimately trusted key EE37CF96 not found
> Out of curiosity, did you try these with --no-auto-check-trustdb? the
> three lines above suggest that you have some sort of ownertrust in use,
> so some of the time spent might be spent doing an automatic trustdb
> rebuild, which would confuse the measurements a bit.
That's a good point. I'll try to keep that in mind the next time I
run the benchmark.
> > real 1m52.560s
> > user 0m6.268s
> > sys 0m31.604s
> One interesting thing to note here is that the time spent by the CPU in
> userspace and the kernel is 38s -- the real ("wall") time is over three
> times that. This suggests that something is going on with your I/O
> subsystem (or that gpg itself is thrashing the kernel's disk I/O
I observed this as well, but I didn't have an explanation other than
latency, which is quite high given that I have an ssd and nothing else
> still, 38s compared to the 6s from the kdb run is a signficant
> difference if we can rule out the trustdb check:
FWIW, the trustdb check would have been done in both cases.
> Have you tried more delicate actions like concurrent updates or keyring
Not yet. This is just a small proof of concept. It's unclear whether
this issue is important enough to dedicate resources to it. Feel free
to make a case.
More information about the Gnupg-devel