gpg --refresh with large keyrings and hkps in 2.1.1

[Ben McGinnes]

On 16/04/2015 7:17 am, Daniel Kahn Gillmor wrote:
> On Wed 2015-04-15 14:44:55 -0400, Ben McGinnes wrote:
>> I have a work-around for something else relating to dirmngr which may
>> also serve as work-around for this.  Now since I've got around 9,400
>> keys in my public keyring, if I can do a full refresh (regardless of
>> keyserver or server connection protocol), then that ought to do the
>> job.  Now granted, I'm not trying to force TLS connections to the
>> keyservers, but the reason for that is that my workaround is to pipe
>> all keyserver traffic through proxychains and Tor.
>>
>> I figure that since the keys are what they are, they can protect
>> themselves.  So the real benefit is in foiling traffic analysis and
>> since all Tor connections use TLS within the Tor network anyway, no
>> one's going to really be able to see which keys I'm requesting or
>> updating.
> 
> Does this protect the privacy of your keyring effectively against people
> snooping between tor and the keyserver, though?

In a sense, there's no clear link to me even if it is open to view.

> Your keyring is likely to be unique to you.

To an extent yeah, but the majority of those keys are the Bitcoin-OTC
user list, the entirety of ncsc.mil and a bunch of really strange
types posting to lists like gnupg-users, gnupg-devel and
enigmail-users.  They really are the strange ones.  ;)

> Tor circuits to a particular endpoint are likely to be stable over the
> period of time it would take to fetch the whole keyring.

In a country with a decent Internet connection, sure.  Over here in
Australia, however, you can be pretty sure that you'll hit the ten
minute window more than once.

> As a result, someone monitoring traffic as it flows between the tor exit
> nodes and the keyservers can see your keyring today, and can probably
> notice when you add new keys to your keyring.

Which is one more reason to add pretty much every key I encounter and
never delete them.  Working out which ones are seriously used is quite
another thing.

Anyway, the next step to mitigate all of that is a rather more obvious
course: install my own SKS keyserver or two (one here on the LAN and
one out in some nebulous hosting space).

>> Now here's the thing, via the proxychains tunnel there is no
>> trouble with a TLS connection (proxychains feeds into a default
>> privoxy instance and then into the Tor SOCKS5 service, all of which
>> is strung together with a handful of bash scripts to load the
>> relevant homedir and optionally restart dimngr through
>> proxychains).  It happily munches its way through approx. 9,300 of
>> those keys.  The remaining hundred or so have hard-coded keyserver
>> preferences pointing to https, hkps or ldap services and those keel
>> over with the same errors that have been reported already or a
>> report that the keyserver cannot be reached.
>
> You can specify keyserver-options no-honor-keyserver-url if you want
> to avoid this last bit.

Ah, I'd forgotten that one.  I'll run it with that shortly and also
put a timer on it (see if I'm right on the sad, sad state of ADSL
here).

> Are you saying that this doesn't work with dirmngr for you?

It will make direct connections, but it won't recognise the
--keyserver-options http-proxy settings.  Nor will it honour shell or
curl http_proxy values (my testing of that monitored the Tor proxy
interfaces with tcpdump).  Which led to digging proxychains out of the
archives.

>> Note also that using proxychains is only necessary because while
>> dirmngr recognises a specified keyserver, it doesn't recognise
>> proxy settings.
>
> This is distressing.  the dirmngr

Yeah, but there are a bunch of not yet implemented dirmngr functions
(like dirmngr --shutdown).

> I see that https://bugs.g10code.com/gnupg/issue1786
> has been opened about this, i've just commented there.
> Let's take this part of the discussion there.

Lay on MacDuff ... I guess creating that login for wiki access will be
used properly after all.

> Now, without taking
>> each packet apart, there is one major difference between the Tor
>> TLS connections and GPG's TLS connections.  That, of course, is
>> that most notorious of SSL/TLS libraries: GnuTLS.  My suspicion is
>> that this is the source of the failure to reach hkps and https
>> keyservers, regardless of variations in the error messages GPG
>> responds with.  >

> Hm, there was no "failure to reach hkps and
> https keyservers" -- just that the refresh process terminates
>> after some number of keys are > fetched, instead of continuing
>> through the entire list.

Ah, different errors then.  Mine attempts to work it's way through the
entire list but complains about not finding (not reaching) the small
percentage of keys that set keyserver settings.

It does, however, also produce these errors when when explicitly
pursuing https or hkps servers and no proxy set:

gpg: error searching keyserver: General error
gpg: keyserver search failed: General error

With the proxy reactivated, the gpg.conf setting pointing to the
ipv4.pool (in order to bypass the ipv6 conflict previously reported)
overrides command line options to access the hkps.pool.  Changing that
to https or hkps produces the general errors above.

> I don't think > the rest of your proposal
>> addresses the concerns i've raised in this > thread -- if the proxy
>> situation is working, that doesn't remove the need for hkps.

No arguments there, though I still am inclined to lean towards the
fault being with the means by which GPG attempts to communicate across
TLS, especially when it clearly has no trouble when going through the
proxychains/Tor/openssl tunnel, nor is there any trouble with the
1.4.19 code (I made a copy of my homedir, so one is live with 1.4 and
the other with 2.1), which relies on libcurl instead.

Anyway, it should be easy enough to prove, just break down every
single step that dirmngr performs and run it manually.  Either that or
try replacing GnuTLS with OpenSSL (obviously releasing that is out of
the question) and seeing if it works.  Obviously if it does, GnuTLS is
back in the naughty corner.  Anyway, that's why I'm more in favour of
abstracting that transport component out of GPG so a the best
transport method for that network can be specified by a user (well, an
advanced user or sysadmin).  Since HKP/S is just a cut down HTTP/S
then as long as dirmngr can speak HTTP 1.1 it should be able to cope
just fine.

>> PS I think calling GnuTLS "most notorious" is bizarre.

Really?  OpenSSL is by no stretch of the imagination perfect either,
but at least it's not trying to pass itself off as part of the GNU
Project when it's developer turned his back on said project several
years ago.  It's only the default lib for GNU & FSF because said
developer didn't drop the GPL.  I'll skip the itemised vulnerabilities
its presented because I don't want to have to encrypt this in order to
compress it down to a size SMTP will deal with ... and that's only a
slight exaggeration.

Yeah, OpenSSL went for a bigger impact with Heartbleed and what not,
but that's because of the market share it has and why?  Because the
only "alternative" was GnuTLS ... which has been on a semi-permanent
blacklist as a result of the stupid vulnerabilities.  Round and round
we go ...  ;)

That said, I'd love to be proven wrong and to see that GnuTLS is a
genuinely viable alternative to OpenSSL.  I just doubt it's reached
that point yet.  To put it another way, I certainly wouldn't bet any
of my own money on the prospect of GnuTLS not being the culprit or
adversely affecting the results here.


Regards,
Ben

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 630 bytes
Desc: OpenPGP digital signature
URL: </pipermail/attachments/20150416/38e4fb41/attachment.sig>