Corruption of archive files with afio and gpg

[Marcus Williams]

On 14/01/2004, Marcus Williams wrote:
> I saw an earlier message on this list about getting corrupted files
> within an afio archive when using gpg to "compress" the afio files. I
> have the same intermittent problem. I'm using a slightly different cmd
> line:

> find *|afio -b 10240 -c 1000 -o -Z -U -P gpg -Q --symmetric \
>             -Q --passphrase-fd=3 -Q --no-verbose \
>             -Q --batch -Q --no-options -3 3 \
>             /dev/nst0 \
>             3<my_passphrasefile

> I get errors about various files failing to compress like "afio:
> /path/to/some/file: Error zipping file, written damaged copy to
> archive". Again, without gpg there are no errors. In an archive of
> about 700000 files I'm seeing probably 50-100 files corrupt.

[I'm not subscribed to the list but I've tried my best to keep this
threaded properly...]

Turns out this is a feature of afio (or possibly a bug depending on
how you look at it). The problem is that if you switch compression on
with gpg, you cant guarantee the encrypted file will have the same
filesize on different runs (presumably different ciphertext is
generated on each run and so the internal compression will be
better/worse each time).

When afio is run it tries to keep the "compressed" file in memory (the
max size of which can be adjusted using -M), but if the compressed
file if bigger than this max memory size it ends up zipping it twice.
Once to get the length, the second time to get the actual file.

Later on in the afio source the size of the original run is compared
to the number of bytes read in from the second run and if they
mismatch the error I was getting occurs.

I'm not sure if the size differences end up causing corruption of the
file later on in the afio process (I've contacted the author about
this).

It looks like the solution is to use a simple script that gzips the
source plaintext file before piping to gpg (with compression off). You
then use this script in place of gpg in the afio command line. For
unencrypting it you pipe the output of gpg through gzip in a different
script and use that script for restoring the backup set.

Hope that helps someone else,

Marcus

-- 
Marcus Williams -- http://www.quintic.co.uk
Quintic Ltd, 39 Newnham Road, Cambridge, UK
  This message is private [ ] public [*]