Unattended use of gpg across a wide range of gpg versions, Ubuntu edition. --debug-quick-random taking evasive action.

[Dan Kegel]

tl;dr: anyone know what's up with --debug-quick-random?  Also, handy
script for unattended key generation across many versions of gpg.

Hi all.  This topic has been beaten to death on many forums and in many
bug reports, but here's a user report from the field that sums up what
works.  It's mostly just stitching together known workarounds, plus
one little mystery
with --debug-quick-random in gpg 2.1.15 (the one on Ubuntu 17.04).
I'll list the problems, then at the bottom show the full solution I'm using.

I'm writing a test script that uses gpg, so I reviewed
https://www.gnupg.org/documentation/manuals/gnupg/Unattended-Usage-of-GPG.html
but it doesn't quite handle all the situations I ran into.
This kind of test script has to satisfy requirements like:
- work on current OS as well as last few LTS releases
- use the OS's default gpg
- work in both interactive and headless situations
- leave the user's normal environment unchanged
- work even in deeply nested directories
That means I can't follow some of the advice in the manual (e.g. "use
GPGME" or "use --quick-addkey").

For the purposes of testing, let's say I want to generate a key with the command
   gpg --gen-key
for use with apt on an Ubuntu 17.04 desktop, as well as in freshly
installed headless older systems.
(For instance, containers created with the commands
  lxc-create -n ubu1204 -t download -- --dist ubuntu --release precise
--arch amd64
  lxc-create -n ubu1404 -t download -- --dist ubuntu --release trusty
--arch amd64
  lxc-create -n ubu1604 -t download -- --dist ubuntu --release xenial
--arch amd64
  lxc-create -n ubu1704 -t download -- --dist ubuntu --release zesty
--arch amd64 )
Easy, right?

Challenges and solutions I ran into, rearranged in a less embarassing
order than I ran into them:

0. Googling for solutions to problems finds stale or incomplete info
from random people
Solution: RTFM.  Really.  Go find *the manual* for gpg and read it.

1. Running a test script that creates keys affects user's keyring
Solution: follow
https://www.gnupg.org/documentation/manuals/gnupg/Ephemeral-home-directories.html
i.e. create a directory for the test, and set GNUPGHOME to the
absolute path to that dir
Works on all systems

2. 'gpg --gen-key' prompts user for key parameters, and aborts if
/dev/tty can't be opened (e.g. with noninteractive ssh )
Solution: follow
https://www.gnupg.org/documentation/manuals/gnupg/Unattended-GPG-key-generation.html
i.e. create a file foo.dat containing the responses, e.g.
Key-Type: 1
Key-Length: 2048
Subkey-Type: 1
Subkey-Length: 2048
Name-Real: My Real Name
Name-Email: foo at example.com
Expire-Date: 30
and change the command to 'gpg --batch --gen-key foo.dat'
Works on ubuntu 16.04 and below

3. On ubuntu 16.04, which straddles gpg and gpg2, the command
 'gpg --export | gpg2 --import -'
appears to be required to get apt to notice a key you've generated
with gpg, but 'gpg2 --import' aborts with
gpg: can't connect to the agent: Invalid value passed to IPC
gpg: error getting the KEK: No agent running

Solution: 'sudo apt-get install gnupg-agent', then
use "gpg-agent --daemon -- gpgcommand..." to create a transient
gpg-agent just for the duration of the gpg command.
This works on Ubuntu 12.04 through 16.04.

4. also on ubuntu 17.04, the previous fix isn't quite enough.
gpg-agent fails with
gpg-agent[1631]: command 'GENKEY' failed: Inappropriate ioctl for
device <Pinentry>
gpg: agent_genkey failed: Inappropriate ioctl for device
which sounds like https://dev.gnupg.org/T2680
Evidently it wants a tty, which isn't going to be possible.
Solution:
echo allow-loopback-pinentry > $GNUPGHOME/gpg-agent.conf
and add --pinentry-mode loopback to the gpg command.
This requires ubuntu 17.04 and up; you can't use it with ubuntu 12.04
through 16.04.

5. gpg hangs with message
Not enough random bytes available.  Please do some other work...
Solutions:
a) stuff the system rng somewhat securely; e.g. on Ubuntu, 'sudo
apt-get install haveged'
b) tell gpg to use an insecure RNG, e.g.
if gpg --quick-random --version >/dev/null 2>&1 ; then
    echo quick-random >> "$GNUPGHOME"/gpg.conf
elif gpg --debug-quick-random --version >/dev/null 2>&1 ; then
    echo debug-quick-random >> "$GNUPGHOME"/gpg.conf
fi
Either works on all tested ubuntu versions up to ubuntu 16.04.

6. On Ubuntu 17.04, gpg (2.1.15) takes several minutes to run, complaining
gpg-agent[6385]: can't connect my own socket: IPC connect call failed
gpg-agent[6385]: this process is useless - shutting down
even with --debug-quick-random in gpg.conf (or gpg-agent.conf).
Oddly, the same two workarounds fix this, more or less:
a) stuff the system rng somewhat securely; e.g. on Ubuntu, 'sudo
apt-get install haveged'
b) tell gpg-agent to use an insecure RNG; only way is to pass
--debug-quick-random option on gpg-agent's commandline!
Neither conf file will do anymore.  That socket error is very odd, and
so is the fact
that tweaking the rng in these two ways makes it go away.  Bug?  Feature?

7. When running tests in directories with long names, gpg aborts with
socket name '/some/long/path is too long
Solution: GNUPGHOME must be shorter than UNIX_PATH_MAX, which can be
as short as 94 bytes on some systems.
(Fixed in latest gpg, but you have to work around it for older ones.)

8. gpg might write to the tty at some point
Solution: as documented, add --no-tty options on all gpg calls or in conf file

Here's a consensus script for unattended key generation demonstrating
most of the workarounds.
--- snip ---
#!/bin/sh
set -x
set -e

# Check to see if gpg requires agent & supports loopback prompt
if gpg --version | head -n 1 | grep ' 2\.'
then
   gpg_use_loopback=true
else
   gpg_use_loopback=false
fi

# Check to see if ubuntu 16.04-specific workaround needed for apt
if test -x /usr/bin/gpg2
then
   gpg_copy_to_gpg2_needed=true
else
   gpg_copy_to_gpg2_needed=false
fi

# Avoid 'socket name too long' error with older gpg
GNUPGHOME=/tmp/gpg-isolation-demo-unique-and-obscure-path
export GNUPGHOME
rm -rf $GNUPGHOME
mkdir -m700 $GNUPGHOME

# 1st half of workaround to allow bypassing pinentry prompts; insecure-ish
if $gpg_use_loopback
then
    echo allow-loopback-pinentry > $GNUPGHOME/gpg-agent.conf
fi

# Never send output to tty
echo no-tty >> $GNUPGHOME/gpg.conf
# Never ask questions
echo batch >> $GNUPGHOME/gpg.conf

# Work around lack of entropy; we don't need it for this test.  Insecure.
QUICK=""
if gpg --quick-random --version >/dev/null 2>&1 ; then
    echo quick-random >> "$GNUPGHOME"/gpg.conf
elif gpg --debug-quick-random --version >/dev/null 2>&1 ; then
    echo debug-quick-random >> "$GNUPGHOME"/gpg.conf
    QUICK=--debug-quick-random
fi

keyemail=foo at example.com
cat > gpg.in.tmp <<_EOF_
Key-Type: 1
Key-Length: 2048
Subkey-Type: 1
Subkey-Length: 2048
Name-Real: My Real Name
Name-Email: $keyemail
Expire-Date: 30
_EOF_

if $gpg_use_loopback
then
    time gpg-agent $QUICK --daemon -- \
    gpg --pinentry-mode loopback --passphrase '' --gen-key gpg.in.tmp
else
    gpg --passphrase '' --gen-key gpg.in.tmp < /dev/null

    # Extra step only needed for ubuntu 16.04's apt?
    if $gpg_copy_to_gpg2_needed
    then
        gpg --passphrase '' --armor --export-secret-keys $keyemail \
        | gpg-agent --daemon gpg2 --passphrase "" --import -
    fi
fi
--- snip ---