TOFU performance / DB format

[Andre Heinecke]

Hi Neal,

On Wednesday 28 October 2015 11:12:32 Neal H. Walfield wrote:
> I think there might be a bit of confusion.  rsync doesn't do two way
> synchronization.  It can be used create a backup and update that
> backup.  But, it doesn't do a two-way sync.

I've tried to make two arguments. First that I don't see how the split DB 
solves two way sync. Secondly that I don't think that is a great improvement 
for backup / one way sync (Using the rsync example).

> Unison works by considering the files the basic unit of
> synchronization and memorizes the state of each file (by saving a
> check sum and the last modification time).  When it synchronizes L and
> D, it sees that D has the original copy of 'l' and sends L's updated
> version of 'l' to D.  Likewise it sees that L has the original version
> of 'd' and sends D's updated version of 'd' to L.  Only if a file is
> updated on both L and D does the user need to manually intervene and
> either choose a copy or manually merge the changes.
> 
> This approach isn't perfect, but in practice it works great: I rarely
> have to manually fix something.  (I've been using this setup for about
> a decade, I think.)
> 
> The difficulty with the TOFU data is that we update the DB whenever we
> see a new signed message.  Thus, for active users of GnuPG, we expect
> there to be a fair amount of churn.  Further, it is not possible to
> merge two DBs by hand.  (We could and probably will write a tool to do
> this.  Nevertheless, we'd then have to teach users about it, which is
> a pain.)  By splitting the DB isn't many small, atomic chunks, the
> chance of a conflict decreases dramatically.  But, equally important,
> if two chunks do diverge, choosing one copy randomly still results in
> a usable DB with little data loss.

My problem with two way sync was exactly the "Merge" case. When I work on my 
Laptop and Desktop usually I'm in communication with similar partners. So I 
would expect a tofu db conflict would be the "usual" case. Similarly my pubring 
/ trustdb files would diverge anyway, breaking a two way merge. For me it is 
easier to do a one way sync from time to time from / to my laptop if I know 
I've made changes I want to have on the other system.

You say that the split DB makes it easier to do a two way merge, I can see the 
point of that. But as it does not really "solve" two way merges imo (as there 
still will be individual conflicts and other data with similar problems) I 
personally don't see that it has enough value to justify having redundant code 
and yet another config option to individualize a setup.

> Personally, I find this improved ability to do two-synchronization is
> a big win. 

Ok. If you think the advantages outweigh the problems I'm totally fine with it.

I just wanted to raise my concerns from a support / maintenance perspective.  
Even the slightest behavioral differences tend to lead to individual Bugs.
Even if the difference in the code is small now I think that the behavior of 
that code might be quite different between the split / flat format.

And this is GnuPG once something is released it is usually maintained for a 
veeery long time ;-) So we probably have to decide now If we want to maintain 
two tofu database formats / layouts "forever"

> But, given the huge performance difference, I think it is
> reasonable to make the flat format the default DB format.

Yes please :-) I have not tested it yet but I expect that the performance 
difference will be even more pronounced on Windows.


Regards,
Andre

-- 
Andre Heinecke |  ++49-541-335083-262  | http://www.intevation.de/
Intevation GmbH, Neuer Graben 17, 49074 Osnabrück | AG Osnabrück, HR B 18998
Geschäftsführer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: This is a digitally signed message part.
URL: </pipermail/attachments/20151028/5718499f/attachment.sig>