Proposal: New keydb format

[Neal H. Walfield]

Hi Guilhem,

At Mon, 23 Nov 2015 20:55:36 +0100,
Guilhem Moulin wrote:
> On Mon, 23 Nov 2015 at 15:15:19 +0100, Neal H. Walfield wrote:
> > I rebased against master and fixed a few bugs including the large
> > memory use that you reported.  Please get the latest code from the
> > neal/kdb branch.
> 
> Awesome, thanks a lot for looking into this!

:)

> Yeah, the max res set is
> kept small now.  I ran the whole benchmark from
> <20151101184906.GA28098 at localhost.localdomain>, but only write down the
> updates worth mentioning here
> 
> >> * Listing keys *
> >>[…]
> >>  ~$ time ./g10/gpg2 --homedir /run/shm/gnupg-kdb --list-keys >/dev/null
> >>  0:00.73 (0.66 user, 0.07 sys)  108716k maxres
> 0:00.74 (0.68 user, 0.05 sys)  35192k maxres
> 
> >> * Listing sigs without nested lookups (--fast-list-mode) *
> >>[…]
> >>  ~$ time ./g10/gpg2 --homedir /run/shm/gnupg-kdb --fast-list-mode --list-sigs >/dev/null
> >>  0:00.96 (0.88 user, 0.08 sys)  108692k maxres
> 0:00.97 (0.89 user, 0.07 sys)  35116k maxres
> 
> >> * Listing sigs with nested lookups *
> >>[…]
> >>  ~$ time ./g10/gpg2 --homedir /run/shm/gnupg-kdb --list-sigs >/dev/null
> >>  0:05.93 (5.75 user, 0.18 sys)  107972k maxres
> 0:06.01 (5.84 user, 0.16 sys)  41888k maxres
> 
> 
> So all in all, while the the running time is unchanged, the memory usage
> is reduced by a factor of 2 or 3.  Sweet!

Thanks for rerunning these.  This is good news.  FWIW, the memory
usage is now O(1) instead of O(N).  And, if necessary, it sounds like
we could reduce the amount of read ahead cache to save even more space
(but I'm not sure this is necessary in practice).

> There is however a typo which fails any attempt to delete a key.  The
> patch is trivial:

Whoops!  Thanks, I'll fix this soon.

Are there are specific situations that the new format doesn't handle
well for you?

Thanks,

:) Neal