TOFU performance / DB format

[Neal H. Walfield]

At Tue, 20 Oct 2015 19:46:23 +0200,
Andre Heinecke wrote:
> - The performance of the initial tofu db build is quite bad.
> One of your changes today roughly doubled the time I currently need to build 
> the inititial tofu db. After removing the tofu.d the first keylisting now takes 
> about 45 seconds.
> I think this needs some profiling and optimization as that's with a homedir on 
> a SSD with an i7 2600k CPU and a keyring with "just" 615 keys.

I wonder if the slow down is due to 26d457c2, which causes the code to
retry a few times instead of immediately abort if the database is
locked.  (Was Kleopatra running in the background performing key
listings?)

As we also spoke about on jabber, there are two database formats: the
split format and the flat format.

The split format, which is currently the default, uses one DB per
email and key.  So in your case, you're looking at potentially a few
thousand small files and the cost of sqlite setting up the initial
data structures in each one and opening and closing the DBs.

The flat format uses a single file and is, as you observed on jabber,
about 20 times faster.

The split and flat formats offer a trade-off.  The flat format is
faster.  The split format is more amenable to two-way synchronization
(e.g., Union).  If you don't use GnuPG on multiple computers or don't
care about keeping your TOFU DBs synchronized, then using the flat
format is the way to go.  Currently, there is no tool to convert
between the two, but it should be a problem to write such a tool.  In
fact, it might make sense to write a tool that merges multiple DBs.

Do you have any thoughts about this?

Thanks!

:) Neal