TOFU performance / DB format
aheinecke at intevation.de
Wed Oct 21 15:32:05 CEST 2015
On Wednesday 21 October 2015 14:36:07 Neal H. Walfield wrote:
> At Tue, 20 Oct 2015 19:46:23 +0200,
> Andre Heinecke wrote:
> > - The performance of the initial tofu db build is quite bad.
> > One of your changes today roughly doubled the time I currently need to
> > build the inititial tofu db. After removing the tofu.d the first
> > keylisting now takes about 45 seconds.
> > I think this needs some profiling and optimization as that's with a
> > homedir on a SSD with an i7 2600k CPU and a keyring with "just" 615 keys.
> I wonder if the slow down is due to 26d457c2, which causes the code to
> retry a few times instead of immediately abort if the database is
> locked. (Was Kleopatra running in the background performing key
Kleopatra was not running.
> As we also spoke about on jabber, there are two database formats: the
> split format and the flat format.
> The split format, which is currently the default, uses one DB per
> email and key. So in your case, you're looking at potentially a few
> thousand small files and the cost of sqlite setting up the initial
> data structures in each one and opening and closing the DBs.
I am a bit unwilling to believe that this is really that expensive. I mean a
clean build of gnupg on this system takes only 20 seconds. And that also
involves a lot of calculations and thousands of files ;-)
> The flat format uses a single file and is, as you observed on jabber,
> about 20 times faster.
That was a misunderstanding then. The 20 times speedup was between the first
and the second run of gpg2 -k
flat / split both were at around 20sec.
Here are some more timinings (tofu.d and tofu.db deleted between runs):
With rev: 9afeb4cc
gpg2 --trust-model=tofu+pgp --tofu-db-format flat -k 3.52s user 0.91s system
21% cpu 20.695 total
gpg2 --trust-model=tofu+pgp --tofu-db-format split -k 3.10s user 2.43s system
12% cpu 44.597 tota
With rev: bc9ff6c85
gpg2 --trust-model=tofu+pgp --tofu-db-format flat -k 1.34s user 0.89s system
13% cpu 16.068 total
gpg2 --trust-model=tofu+pgp --tofu-db-format split -k 3.01s user 2.57s system
12% cpu 45.751 total
With rev: eb8a0b0
gpg2 --trust-model=tofu+pgp --tofu-db-format flat -k 1.14s user 0.87s system
13% cpu 14.532 total
gpg2 --trust-model=tofu+pgp --tofu-db-format split -k 3.18s user 2.49s system
12% cpu 45.165 total
I'm a bit confused with the last result as yesterday on that revision it only
took 22sec. Maybe a problem with my test setup. ATM I don't have time to look
into it more.
Andre Heinecke | ++49-541-335083-262 | http://www.intevation.de/
Intevation GmbH, Neuer Graben 17, 49074 Osnabrück | AG Osnabrück, HR B 18998
Geschäftsführer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner
More information about the Gnupg-devel