Proposal: New keydb format

[Neal H. Walfield]

Hi Daniel,

At Fri, 30 Oct 2015 05:25:35 +0900,
Daniel Kahn Gillmor wrote:
> On Thu 2015-10-29 06:52:46 -0400, "Neal H. Walfield" <neal at walfield.org> wrote:
> > Some people have complained that GnuPG is slow when accessing the key
> > db (e.g., [1]).  To mitigate this, GnuPG has some ugly caches (commit
> > 492792), which are not completely transparent and introduce subtle
> > bugs that are very hard to find (see, e.g., commits 60bc518 and
> > ee7ec12, which demonstrate a cache inconsistency).
> >
> > Werner was the opinion that the performance problem is due to
> > signature verification and not due to DB accesses.  To test this
> > theory, I did some measurements yesterday and I implemented a small
> > prototype, which uses SQLite as the underlying storage engine.  It can
> > be retrieved from git on the neal/next branch.
> 
> Wow, thank you for doing this work.  I'm really glad to see it!

:)

Out of curiosity, are you suggesting that the performance of kbx is a
real problem for you in practice?  If so, I'd like to understand your
setup a bit better.  (Feel free to reply off-list.)

> >   [1] https://bugs.gnupg.org/gnupg/issue2019
> >
> > Here are some timing results.  I use the Debian keyring
> > (/usr/share/keyrings/debian-keyring.gpg), which contains 751 keys.
> >
> > Importing the keyring: 2 minutes (keybox) vs. 8 seconds (new):
> >
> >   $ rm pubring.kdx; time gpg2 --no-default-keyring --primary-keyring
> >   gnupg-kbx:pubring.kdx --import debian-keyring.gpg >/dev/null
> >   gpg: Total number processed: 751
> >   gpg:               imported: 751
> >   gpg: public key of ultimately trusted key 2183839A not found
> >   gpg: public key of ultimately trusted key BC15C85A not found
> >   gpg: public key of ultimately trusted key EE37CF96 not found
> 
> Out of curiosity, did you try these with --no-auto-check-trustdb?  the
> three lines above suggest that you have some sort of ownertrust in use,
> so some of the time spent might be spent doing an automatic trustdb
> rebuild, which would confuse the measurements a bit.

That's a good point.  I'll try to keep that in mind the next time I
run the benchmark.

> 
> 
> >   real	1m52.560s
> >   user	0m6.268s
> >   sys	0m31.604s
> 
> One interesting thing to note here is that the time spent by the CPU in
> userspace and the kernel is 38s -- the real ("wall") time is over three
> times that.  This suggests that something is going on with your I/O
> subsystem (or that gpg itself is thrashing the kernel's disk I/O
> subsystem).

I observed this as well, but I didn't have an explanation other than
latency, which is quite high given that I have an ssd and nothing else
was running.

> still, 38s compared to the 6s from the kdb run is a signficant
> difference if we can rule out the trustdb check:

FWIW, the trustdb check would have been done in both cases.

> Have you tried more delicate actions like concurrent updates or keyring
> merges?

Not yet.  This is just a small proof of concept.  It's unclear whether
this issue is important enough to dedicate resources to it.  Feel free
to make a case.

Thanks!

:) Neal