A new test framework: request for comments and testing

[Justus Winter]

Hello,

I have been working on a new test framework at the beginning of the
year.  We perceived two main problems with the current Bourne
shell-based test framework:

* we cannot run it on Windows
* writing shell scripts that are portable even across POSIXish systems
  is a challenge
(* the language is not very nice, leading to brittle code with a lot
   of copying)

So we sat down at the Chaos Communication Congress and came up with
the idea of using TinySCHEME, a very small and self-contained
interpreter.  My task was to evaluate the feasibility of that
approach.  One of the concerns was that choosing Scheme over Bourne
shell will be an higher barrier to writing tests.

So I took TinySCHEME, created bindings for GnuPGs platform
abstractions, and ported the OpenPGP test suite (tests/openpgp) to my
new interpreter to make sure that I implemented enough features, like
process management.

So a test that would look like this with the old framework

~~~ snip ~~~
. $srcdir/defs.inc || exit 3

info Checking decryption of supplied files
for i in $plain_files ; do
    progress $i
    $GPG -o y --yes $srcdir/$i.asc
    cmp $i y || error "$i: mismatch"
done

progress_end
~~~ snap ~~~

can now be expressed like this:

~~~ snip ~~~
(load (with-path "defs.scm"))

(for-each-p
 "Checking decryption of supplied files"
 (lambda (name)
   (tr:do
    (tr:open (in-srcdir (string-append name ".asc")))
    (tr:gpg "" '(--yes))
    (tr:assert-identity name)))
 plain-files)
~~~ snap ~~~

And while it might be a little more verbose I believe the advantages
outweigh any inconveniences:

* it is completely self-contained, doesn't rely on e.g. cmp
* it works on Windows
* one has a powerful little language at hand facilitating code reuse
* robust error handling

I have tested the code on Hurd, Linux, and Windows.  Please test this
patch series on any weird and wonderful platform that you have access
to.  Using it is easy, just do make check.

I pushed my work periodically to the GnuPG repository at
git.gnupg.org.  The latest iteration is in the branch 'justus/scm-9'.

Please accept my apologies for not communicating earlier what I was
doing and where we want to go.  We plan to merge this after the 2.1.12
release.

Please let me know what you think and how it breaks on your favorite
platform.


Cheers,
Justus