TOFU performance / DB format

[Andre Heinecke]

Hi Neal,

On Wednesday 21 October 2015 14:36:07 Neal H. Walfield wrote:
> At Tue, 20 Oct 2015 19:46:23 +0200,
> 
> Andre Heinecke wrote:
> > - The performance of the initial tofu db build is quite bad.
> > One of your changes today roughly doubled the time I currently need to
> > build the inititial tofu db. After removing the tofu.d the first
> > keylisting now takes about 45 seconds.
> > I think this needs some profiling and optimization as that's with a
> > homedir on a SSD with an i7 2600k CPU and a keyring with "just" 615 keys.
> 
> I wonder if the slow down is due to 26d457c2, which causes the code to
> retry a few times instead of immediately abort if the database is
> locked.  (Was Kleopatra running in the background performing key
> listings?)

Kleopatra was not running.

> As we also spoke about on jabber, there are two database formats: the
> split format and the flat format.
> 
> The split format, which is currently the default, uses one DB per
> email and key.  So in your case, you're looking at potentially a few
> thousand small files and the cost of sqlite setting up the initial
> data structures in each one and opening and closing the DBs.

I am a bit unwilling to believe that this is really that expensive. I mean a 
clean build of gnupg on this system takes only 20 seconds. And that also 
involves a lot of calculations and thousands of files ;-)

> The flat format uses a single file and is, as you observed on jabber,
> about 20 times faster.

That was a misunderstanding then. The 20 times speedup was between the first 
and the second run of gpg2 -k

flat / split both were at around 20sec.

Here are some more timinings (tofu.d and tofu.db deleted between runs):

With rev: 9afeb4cc

gpg2 --trust-model=tofu+pgp --tofu-db-format flat -k  3.52s user 0.91s system 
21% cpu 20.695 total

gpg2 --trust-model=tofu+pgp --tofu-db-format split -k  3.10s user 2.43s system 
12% cpu 44.597 tota

With rev: bc9ff6c85

gpg2 --trust-model=tofu+pgp --tofu-db-format flat -k  1.34s user 0.89s system 
13% cpu 16.068 total

gpg2 --trust-model=tofu+pgp --tofu-db-format split -k  3.01s user 2.57s system 
12% cpu 45.751 total

With rev: eb8a0b0

gpg2 --trust-model=tofu+pgp --tofu-db-format flat -k  1.14s user 0.87s system 
13% cpu 14.532 total

gpg2 --trust-model=tofu+pgp --tofu-db-format split -k  3.18s user 2.49s system 
12% cpu 45.165 total

I'm a bit confused with the last result as yesterday on that revision it only 
took 22sec. Maybe a problem with my test setup. ATM I don't have time to look 
into it more. 


Regards,
Andre

-- 
Andre Heinecke |  ++49-541-335083-262  | http://www.intevation.de/
Intevation GmbH, Neuer Graben 17, 49074 Osnabrück | AG Osnabrück, HR B 18998
Geschäftsführer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner