A new test framework: request for comments and testing

[Bernhard Reiter]

Hello Justus, Hello GnuPG-people,

Am Donnerstag, 21. April 2016 17:13:34 schrieb Justus Winter:
> I have been working on a new test framework at the beginning of the
> year.  We perceived two main problems with the current Bourne
> shell-based test framework:
>
> * we cannot run it on Windows
> * writing shell scripts that are portable even across POSIXish systems
>   is a challenge
> (* the language is not very nice, leading to brittle code with a lot
>    of copying)

I agree with the problems, especially the portability question is important.

> So we sat down at the Chaos Communication Congress and came up with
> the idea of using TinySCHEME, a very small and self-contained
> interpreter.  My task was to evaluate the feasibility of that
> approach.  One of the concerns was that choosing Scheme over Bourne
> shell will be an higher barrier to writing tests.

I've looked up 
https://en.wikipedia.org/wiki/TinyScheme
http://tinyscheme.sourceforge.net/home.html

and I share that concern of a higher barrier.
 If we want to have more people improving and
writing automatic tests, they should be especially easy to unterstand and 
write. The problem with an exotic language dialect is: 
* No good documentation, no tutorial.
* Small community, often not well maintained.
* Lacking functions and libraries.

Checking this for TinyScheme: 
* It does not appear somewhere near the top in popular rankings
   https://en.wikipedia.org/wiki/Measuring_programming_language_popularity
   (even with all the problems considered it can be called an exootic dialect
    scheme itself of course appears in the top 50 in a number of rankings.)
* The manual in the tinyscheme-1.41.tar.gz only seems to be directred
   at people already knowing some other scheme things. No tutorial.
* Last release from  April 2013, 3 years ago. Three commits in 2016,
   five commits in 2015 to head branch.
* There is a regular expression library (useful for testing frameworks),
   but not that many others.

So why not using a small-footprint interpreter for a more popular language?
Lua may be a candidate.
(There are small print C interpreters https://github.com/zsaleeba/picoc,
but they are probably even more exotic.)
Personally I would not mind using a larger popular language for testing,
like Python. To me clarity and approachability of the language would be more 
important than the small footfront and direct integration.

> I have tested the code on Hurd, Linux, and Windows.  Please test this
> patch series on any weird and wonderful platform that you have access
> to.  Using it is easy, just do make check.
>
> I pushed my work periodically to the GnuPG repository at
> git.gnupg.org.  The latest iteration is in the branch 'justus/scm-9'.

It would be cool to have a wiki.gnupg.org page with instructions how
to test this.

Thanks for working on improving the test framework of GnuPG!

Best Regards,
Bernhard

-- 
www.intevation.de/~bernhard   +49 541 33 508 3-3
Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998
Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: This is a digitally signed message part.
URL: </pipermail/attachments/20160504/ef0dda32/attachment.sig>