TOFU performance / DB format

[Andre Heinecke]

Hi,

On Friday 23 October 2015 19:09:24 Neal H. Walfield wrote:
> Unfortunately, this change only helps with the flat database format.
> For the split forat, we're still looking at over 90 seconds for the
> initial listing and 3 seconds for subsequent listings.

I can confirm that thanks to your performance improvements initializing the flat 
db is now really as fast as I would expect. << 1second. :-)

Great, Thanks!

> I'd be interested to hear people's opinions about whether the split
> format (with its ability to be more easily synced) is still
> interesting despite the slow performance.

I'm not sure that keeping two database formats and making them configurable is 
a good idea.

- It makes things harder to maintain. Obviously you have two codepaths with 
different issues. e.g. The performance for the flat format is now fine while the 
performance for the split format is still bad. You will receive bug reports 
and you will always have to ask "are you using flat / split"

The majority will use whatever is default but you will still have to maintain 
the code for the minority that uses another option.

- It makes things harder to document. "If you are using tofu-db-split format 
you will have to sync that folder. If you are using tofo-db-flat format you 
will have to sync that db." etc.

- I don't see how the split format solves two way sync:

  -- Two way sync is not really supported in GnuPG. We have the monolithic
   keyring and the trust.db and It's just not supported afaik two merge
   two gnupg home dirs.
  -- How should such a two-way sync work? If I would only use the "last
  updated" db from two systems how would I merge them in case of conflicts?

With regards to easier sync. It can be easier Just to copy bulk data if it is 
small then lots and lots of different files. E.g with rsync to a system 
connected over the internet i see the following values:

(Ok that is cheating a bit as the tofu.db does not really contain any data but 
the initialization)

First run:
rsync -rvzP tofu.d remote:/tmp/tofu.d  0.54s user 0.10s system 3% cpu 20.231 
total
rsync -rvzP tofu.db remote:/tmp/tofu.db  0.06s user 0.00s system 1% cpu 4.323 
total

Second run:
rsync -rvzP tofu.d remote:/tmp/tofu.d  0.39s user 0.08s system 9% cpu 4.739 
total
rsync -rvzP tofu.db remote:/tmp/tofu.db  0.02s user 0.00s system 1% cpu 1.498 
total

Regards,
Andre


-- 
Andre Heinecke |  ++49-541-335083-262  | http://www.intevation.de/
Intevation GmbH, Neuer Graben 17, 49074 Osnabrück | AG Osnabrück, HR B 18998
Geschäftsführer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: This is a digitally signed message part.
URL: </pipermail/attachments/20151027/59628ca7/attachment.sig>