running all the tests

[Werner Koch]

On Mon,  7 Aug 2017 23:12, dkg at fifthhorseman.net said:

> did you run "make check" before you ran "time make -j3 check" ?  if not,

I did a "make clean" between all tests, so that the "make check" had to
build the software.  I also use --enable-gpgtar --enable-g13
--enable-wks-tools with configure, but that does not make a substantial
difference.  gcc 6.4.   Pure build times are (I did new test runs):

  $ make -j3
  real    0m59.515s
  user    2m33.732s
  sys     0m6.852s

> the build daemons have seen so much worse than you don't even want to
> know about it. Try looking at the thunderbird build, or at libreoffice
> if you're into this sort of thing ;)

I recall discussion about problems on Amiga and other low-end platforms
;-).  As you know, I really take care about portability.  Running the
new long make check on my Soekris box is not going to work.  Also, my CA
laptop is a T770 which already takes ages to run even the old "make
check".  I recall complaints from SuSE about too long running regression
checks as well as from customers on non-Linux Unices.

GNU software is known to have a uniform build system so that there is no
need for an admin to read long and complicated build instructions.  The
usual triple jump plus a final make install is very well known and also
documented in many internal company documents.  Thus this should be the
default.

> This is the work that build daemons are made to do.  If it means better
> coverage for an important suite like GnuPG, i don't think anyone would
> begrudge their use.

The thing is that we run a Jenkins with a lot of configurations to run
all tests on a lot of platforms - that catches the majority of standard
configurations.  Those which we do not catch are either hard to test
(e.g. network), in non-standard environments, with interesting
combinations of private and public keys, or which exhibit themselves
only by manual tests and inspections.

> I'm pretty curious about the differences in our timings, though, and i'd
> like to understand why they're different.

They are not that different after subtracting the build time.

>     make
>     make check
>     time make check TESTFLAGS=--parallel
>     time make check

I did not do that first make check to condition the box, and I rand the
non-parallel check first.

> parallel:
> real	1m11.491s
> user	2m6.226s
> sys	0m8.157s

  $ make -j3 check TESTFLAGS=--parallel
  real    1m16.963s
  user    1m58.316s
  sys     0m8.816s

So that is basically the same.

> non-parallel:
> real	7m10.321s
> user	2m0.721s
> sys	0m16.063s

  $ make -j3 check
  real    7m47.714s
  user    1m55.772s
  sys     0m10.828s

Ditto.  FWIW, I also run the same thing without maintainer mode (also
after having build the whole thing):

  $ make -j3 check TESTFLAGS=--parallel
  real    0m35.281s
  user    0m39.604s
  sys     0m3.024s

  $ make -j3 check
  real    2m46.663s
  user    0m40.568s
  sys     0m3.868s

> But the wall clock numbers don't lie, nor does the CPU idle statistics.
> If the concern is that the tests are too slow, it sounds like we should
> just encourage the use of TESTFLAGS=parallel, rather than disabling the
> tests.  Alternately, we could figure out why the non-parallel test

Still, I doubt that users will catch and report make check failures from
testing a bunch of combinations of non-standard configurations.  It is
unlikely that the extra tests run at the user site we will catch bugs
due to certain combinations later used at the user site.  There will be
bugs, and we help the users to find the.  The obvious method then will
be to run the build again with extensive test enabled.

Given that --enable-maintainer-mode has its own problems and may fail
when run by a user, I propose to add a separate configure option to run
the extensive test suite.  For the very long running tests in Libgcrypt
we anyway use separate option --enable-large-data-tests.  Using a make
target instead of an configure option does not seem to be a good idea
because there are already several check targets (e.g. check and
installcheck).  Such an option should then also default to parallel test
builds.

I would prefer if build systems, which are based on some kind of
configure script, could agree on a name for enabling such tests.
--enable-extensive-tests or --enable-all-tests might be good candidate
names.  Thus it is up the the user to decide whether to run those tests.


Shalom-Salam,

   Werner


-- 
Die Gedanken sind frei.  Ausnahmen regelt ein Bundesgesetz.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: </pipermail/attachments/20170808/c263f49d/attachment-0001.sig>