New regression test driver

[Daniel Kahn Gillmor]

On Wed 2016-01-27 11:10:18 -0500, Werner Koch wrote:

> Justus started to work on a new regression test system for GnuPG.
> Although the old system works nicely it has one one major problem: We
> can't run the regression tests on the target platform if we are
> cross-compiling.  In particular for Windows this is a major drawback
> because we can only do manual tests.
>
> The old test system is a pure Unix one and requires a POSIX shell.  Such
> a shell is not easily available on Windows and the rest of the test
> framework is also not easy to port to Windows (or other target
> platforms).  So instead of writing a Bourne shell for Windows we came up
> with the idea to use something simpler: A Scheme interpreter is much
> easy to do than a shell and chaning the existing tests to Scheme is
> doable [1].

I'm really glad to see this work, and happy that the project is thinking
about cross-platform testing as well.  hopefully a more comprehensive
test suite will help avoid regressions in the future, and will encourage
more bold refactorings/enhancements.

For debian, i'm inclined to run such a test suite via autopkgtest
because it will be useful for cross-compiled archives.

> We do not want a full fledged Scheme system but the simplest thing we
> can use - speed is not an issue here.  Thus gpgscm is based in
> TinySCHEME (~7000 lines of C code).  The GnuPG branch justus/scm-3 has
> the current code with all tests converted to Scheme.  This basically
> works except that the output needs some cleanup.  Some of the tests may
> now also be run on Windows and we are working to get the other tests
> running there.  The final goal is that a user should be able to run the
> regression tests on the target platform without installing extra tools.
>
> This test driver is now limited to GnuPG but our other libraries and
> tools may also benefit from a portable way of running the tests.  The
> libraries mostly use test programs written in C and are run using the
> standard automake generated driver.  Those test programs themself are
> runable on the target platform but they miss the framework to run the
> entire suite on the target platform.
>
> The question is whether to move the new test framework (gpgscm and the
> supporting Scheme code) from GnuPG to Libgpg-error.  Libgpg-error is
> anyway a build dependency for all our software and thus it would the
> logical place for shared code.

I'm fine with having it in libgpg-error, but i'm not sure i'd build it
for the debian packages.  In particular, tinyscheme is already packaged
for debian, and i'd just as soon rely on it during build and autopkgtest
rather than trying to maintain a fork explicitly.  do you expect gpgscm
to diverge at all from features available in tinyscheme?

   --dkg