Autofonce, GNU Autotests Revisited
Since 2022, OCamlPro has been contributing to GnuCOBOL, the only fully
open-source compiler for the COBOL language. To speed-up our
contributions to the compiler, we developed a new tool,
to be able to easily run and modify the testsuite of the compiler,
originally written as a GNU Autoconf testsuite. This article describes
this tool, that could be useful for other project testsuites.
Since 2022, OCamlPro has been involved in a big modernization project for the French state: the goal is to move a large COBOL application, running on a former Bull mainframe (GCOS) to a cluster of Linux computers. The choice was made to use the most open-source compiler, GnuCOBOL, that had already been used in such projects.
One of the main problems in such migration projects is that most COBOL proprietary compilers provide extensions to the COBOL language standard, that are not supported by other compilers. Fortunately, GnuCOBOL has good support for several mainstream COBOL dialects, such as IBM or Micro-Focus ones. Unfortunately, GnuCOBOL had no support at the time for the GCOS COBOL dialect developed by Bull for its mainframes.
As a consequence, OCamlPro got involved in the project to extend GnuCOBOL with the support for the GCOS dialect needed for the application. This work implied a lot of (sometimes very deep) modifications of the compiler and its runtime library, both of them written in the C language. And of course, our modifications had first to pass the large existing testsuite of COBOL examples, and then extend it with new tests, so that the new dialect would continue to work in the future.
This work lead us to develop
autofonce, a modern open-source
runner for GNU Autoconf
Testsuites, the framework used in GnuCOBOL to manage its
testsuite. Our tool is available on Github, with Linux and Windows
binaries on the release page.
The GNU Autoconf Testsuite of GnuCOBOL⚓
GNU Autoconf is a set of powerful tools, developed to help developers of open-source projects to manage their projects, from configuration steps to testing and installation. As a very old technology, GNU Autoconf relies heavily on M4 macros both as its own development language, and as its extension language, typically for tests.
As an example, a typical test from syn_copy.at looks like:
AT_SETUP([INITIALIZE constant]) AT_KEYWORDS([misc]) AT_DATA([prog.cob], [ IDENTIFICATION DIVISION. PROGRAM-ID. prog. DATA DIVISION. WORKING-STORAGE SECTION. 01 CON CONSTANT 10. 01 V PIC 9. 78 C78 VALUE 'A'. PROCEDURE DIVISION. INITIALIZE CON. INITIALIZE V. INITIALIZE V, 9. INITIALIZE C78, V. ]) AT_CHECK([$COMPILE_ONLY prog.cob], , , [prog.cob:10: error: invalid INITIALIZE statement prog.cob:12: error: invalid INITIALIZE statement prog.cob:13: error: invalid INITIALIZE statement ]) AT_CLEANUP
Actually, we were quite pleased by the syntax of tests, it is easy to
generate test files (using
AT_DATA macro) and to test the execution
of commands (using
AT_CHECK macro), checking its exit code, its
standard output and error output separately. It is even possible to
combine checks to run additional checks in case of error or
success. In general, the testsuite is easy to read and complete.
However, there were still some issues:
At every update of the code or the tests, the testsuite runner has to be recompiled;
Running the testsuite requires to be in the correct sub-directory, typically within the
By default, tests are ran sequentially, even when many cores are available.
The output is pretty verbose, showing all tests that have been executed. Failed tests are often lost in the middle of other successful tests, and you have to wait for the end of the run to start investigating them;
## -------------------------------------------- ## ## GnuCOBOL 3.2-dev test suite: GnuCOBOL Tests. ## ## -------------------------------------------- ## General tests of used binaries 1: compiler help and information ok 2: compiler warnings ok 3: compiler outputs (general) ok 4: compiler outputs (file specified) ok 5: compiler outputs (path specified) ok 6: compiler outputs (assembler) ok 7: source file not found ok 8: temporary path invalid ok 9: use of full path for cobc ok 10: C Compiler optimizations ok 11: invalid cobc option ok 12: cobcrun help and information ok 13: cobcrun validation ok 14: cobcrun -M DSO entry argument ok 15: cobcrun -M directory/ default ok [...]
There is no automatic way to update tests, when their output has changed. Every test has to be updated manually.
In case of error, it is not always easy to rerun a specific test within its directory.
autofonce, we tried to solve all of these issues...
Main Features of Autofonce⚓
autofonce is written in a modern language, OCaml, so that it can
handle a large testsuite much faster than GNU Autoconf. Since we do
not expect users to have an OCaml environment set up, we provide
binary versions of
autofonce for both Linux (static executable) and
Windows (cross-compiled executable) on Github.
autofonce does not use
m4, instead, it has a limited support for a
small set of predefined m4 macros, typically supporting m4 escape
sequences (quadrigraphs), but not the addition of new m4 macros, and
neither the execution of shell commands outside of these macros (yes,
testsuites in GNU Autoconf are actually
sh shell scripts with m4
macros...). In the case of GnuCOBOL, we were lucky enough that the
testsuite was well written and avoided such problems (we had to fix
only a few of them, such as including shell commands into
macros). The syntax of tests is documented here.
Some interesting features of
autofonce are :
autofonceexecutes the tests in parallel by default, using as many cores as available. Only failed tests are printed, so that the developer can immediately start investigating them;
autofoncecan be run from any directory in the project. A
.autofoncefile has to be present at the root of the project, to describe where the tests are located and in which environment they should be executed;
autofoncemakes it easy to re-execute a specific test that failed, by generating, within the test sub-directory, a script for every step of the test;
autofonceprovides many options to filter which tests should be executed. Tests can be specified by number, range of numbers, keywords, or negative keywords. The complete list of options is easily printable using
autofonce run --helpfor example;
autofonce implements a powerful promotion mechanism
to update tests, with the
example, if you update a warning message in the compiler, you would
like all tests where this message appears to be modified. With
autofonce, it is as easy as:
# Run all tests at least once autofonce run # Print the patch that would be applied in case of promotion autofonce promote # Apply the patch above autofonce promote --apply # Combine running and promotion 10 times: autofonce run --auto-promote 10
The last command iterates promotion 10 times: indeed, since a test may have multiple checks, and only the first failed check of the test will be updated during one iteration (because the test aborts at the first failed check), as many iterations as the maximal number of failed checks within a test may be needed.
Also, as for GNU Autoconf,
autofonce generates a final log file containing the results with a full log of errors and files needed to reproduce the error. This file can be uploaded into the artefacts of a CI system to easily debug errors after a CI failure.
During our work on GnuCOBOL,
autofonce improved a lot our user
experience of running the testsuite, especially using the
auto-promotion feature to update tests after modifications.
autofonce could be used for other open-source projects that
already use the GNU Autoconf testsuite. Of course, it requires that
the testsuite does not make heavy use of shell features and mainly
relies on standard m4 macros.
We found that the format of GNU Autoconf tests to be quite powerful to
easily check exit codes, standard outputs and error outputs of shell
autofonce could be used to help using this format in
projects, that do not want to rely on an old tool like GNU Autoconf,
and are looking for a much more modern test framework.
Au sujet d'OCamlPro :
OCamlPro développe des applications à haute valeur ajoutée depuis plus de 10 ans, en utilisant les langages les plus avancés, tels que OCaml et Rust, visant aussi bien rapidité de développement que robustesse, et en ciblant les domaines les plus exigeants (méthodes formelles, cybersécurité, systèmes distribués/blockchain, conception de DSLs). Fort de plus de 20 ingénieurs R&D, avec une expertise unique sur les langages de programmation, aussi bien théorique (plus de 80% de nos ingénieurs ont une thèse en informatique) que pratique (participation active au développement de plusieurs compilateurs open-source, prototypage de la blockchain Tezos, etc.), diversifiée (OCaml, Rust, Cobol, Python, Scilab, C/C++, etc.) et appliquée à de multiples domaines. Nous dispensons également des [formations sur mesure certifiées Qualiopi sur OCaml, Rust, et les méthodes formelles] (https://training.ocamlpro.com/) Pour nous contacter : email@example.com.