performance of gpg --list-secret-keys with large keyrings

[Werner Koch]

On Fri, 10 Oct 2014 09:01, dkg at fifthhorseman.net said:

> keys in the keyring.  The public keys are in pubring.gpg and not
> pubring.kbx (no conversion has happened yet).  Here's the timings:

Yeah, you should do that conversion first ;-)

> I think the move from 0.014s to >7s for --list-secret-keys is because
> gpg 2.1 implements --list-secret-keys by asking the agent about every
> known public key to see if it has the secret material for it.

I have not had this experience but it is quite plausible.  Wi would like
to delay a fix for that to 2.1.1 - if we keep on fixing everything for
2.1.0 we will never see a release.

> Surely it would be more efficient if the agent could just list the keys
> directly.  But maybe there are security reasons that we don't want to

It can't becuase the agent does not now about the OpenPGP protocol and
thus does not know which of its keys belong to an OpenPGP key.  What we
do instead is to compute the "keygrip" from each key and ask the agent
whether it knows the matching private key:

  gpg:
  For each key
     Compute keygrip for each subkey
     Send a HAVEKEY command with all these keygrips to gpg-agent
     gpg-agent:
     For each keygrip
        Build filename
        if access(filename) possible
           return okay to gpg
     gpg:   
     if okay
        list as secret key.

Thus for the command case of a primary and a subkey we have two
directory lookups which should be fast.  Two keygrip computations
(parsing and SHA-1) and the IPC overhead.

There are several places were we could improve things:

- gpg-agent could cache the keygrips to avoid a file system operation.
  But on Linux this should not really matter.

- If the keygrip will be listed caching it with the key like we do for
  the fingerprint would be useful.  Can you run 2.1 using --with-keygrip
  to see whether the numbers are noticeable different than without that
  option?

- We could send a HAVEKEY command for each single keygrip, so that we do
  not need to compute the keygrip for the subkey in advance assuming
  that the agent has the primary key.  IIRC, I did this initially
  but the current solutions was faster.

- We could ask the agent for a list of all private keys it knows
  (KEYINFO --list) and compare against this list.  That makes sense if we
  have to check a lot of public keys but not for just a few ones.  There
  is also the problem what to do if the agent holds a huge numbers of
  keys and does not need to do a full secret key listing (I know of an
  authentication system which does this).

My guess is that the last option would be the best one for the general
case.


Salam-Shalom,

   Werner


-- 
Die Gedanken sind frei.  Ausnahmen regelt ein Bundesgesetz.