running all the tests

[Justus Winter]

Werner Koch <wk at gnupg.org> writes:

> On Tue,  8 Aug 2017 09:58, wk at gnupg.org said:
>
>> Given that --enable-maintainer-mode has its own problems and may fail
>> when run by a user, I propose to add a separate configure option to run
>> the extensive test suite.  For the very long running tests in Libgcrypt
>
> To move on with this I just pushed a change with adds the option
> --enable-all-tests to gnupg's configure.  Only if this option is used
> the full range of tests is done.  Thus --enable-maintainer-mode is not
> used to decide on this.

I'm still bewildered by this discussion and the resulting changes.  For
the record, running all GPGME tests with C++ and 2x Python takes about
1min25s:

    .../gpgme/obj $ time make check
    [...]
    make check  4.68s user 1.06s system 6% cpu 1:25.24 total

Whereas running all GnuPG tests with --enable-all-tests and (!!!)
including all the GPGME tests takes just 8 (eight) seconds longer on my
very humble machine:

    .../gnupg/obj $ time make check-all
    [...]
    ===================
    266 tests run, 266 succeeded, 0 failed, 0 failed expectedly, 0 succeeded unexpectedly, 0 skipped.
    ===================
    make check-all  164.97s user 5.12s system 182% cpu 1:33.35 total

One interpretation is that the GnuPG test framework is so efficient at
running the GPGME tests, that you get all the other 206 tests
essentially for free (or for 8 seconds).

I don't know where to go from here.  Either 90 seconds is an acceptable
run time for the test suite, or it is not.  In the former case, all the
recent test suite changes aimed at reducing the runtime seem moot.  In
the latter, we need to disable some GPGME tests.

Justus
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: </pipermail/attachments/20170821/c5b5938b/attachment.sig>