Autofonce, GNU Autotests Revisited

Authors: Fabrice Le Fessant

Date: 2023-06-27

Category: OCaml

Tags: ocaml, tests, gnucobol, autoconf, autofonce, cobol

Since 2022, OCamlPro has been contributing to GnuCOBOL, the only fully open-source compiler for the COBOL language. To speed-up our contributions to the compiler, we developed a new tool, autofonce, to be able to easily run and modify the testsuite of the compiler, originally written as a GNU Autoconf testsuite. This article describes this tool, that could be useful for other project testsuites.

Introduction

Since 2022, OCamlPro has been involved in a big modernization project for the French state: the goal is to move a large COBOL application, running on a former Bull mainframe (GCOS) to a cluster of Linux computers. The choice was made to use the most open-source compiler, GnuCOBOL, that had already been used in such projects.

One of the main problems in such migration projects is that most COBOL proprietary compilers provide extensions to the COBOL language standard, that are not supported by other compilers. Fortunately, GnuCOBOL has good support for several mainstream COBOL dialects, such as IBM or Micro-Focus ones. Unfortunately, GnuCOBOL had no support at the time for the GCOS COBOL dialect developed by Bull for its mainframes.

As a consequence, OCamlPro got involved in the project to extend GnuCOBOL with the support for the GCOS dialect needed for the application. This work implied a lot of (sometimes very deep) modifications of the compiler and its runtime library, both of them written in the C language. And of course, our modifications had first to pass the large existing testsuite of COBOL examples, and then extend it with new tests, so that the new dialect would continue to work in the future.

This work lead us to develop autofonce, a modern open-source runner for GNU Autoconf Testsuites, the framework used in GnuCOBOL to manage its testsuite. Our tool is available on Github, with Linux and Windows binaries on the release page.

The GNU Autoconf Testsuite of GnuCOBOL

GNU Autoconf is a set of powerful tools, developed to help developers of open-source projects to manage their projects, from configuration steps to testing and installation. As a very old technology, GNU Autoconf relies heavily on M4 macros both as its own development language, and as its extension language, typically for tests.

In GnuCOBOL, the testsuite is in a sub-directory tests/, containing a file testsuite.at, itself including other files from a sub-directory testsuite.src/.

As an example, a typical test from syn_copy.at looks like:

AT_SETUP([INITIALIZE constant])
AT_KEYWORDS([misc])
AT_DATA([prog.cob], [
       IDENTIFICATION   DIVISION.
       PROGRAM-ID.      prog.
       DATA             DIVISION.
       WORKING-STORAGE  SECTION.
       01  CON          CONSTANT 10.
       01  V            PIC 9.
       78  C78          VALUE 'A'.
       PROCEDURE DIVISION.
           INITIALIZE CON.
           INITIALIZE V.
           INITIALIZE V, 9.
           INITIALIZE C78, V.
])
AT_CHECK([$COMPILE_ONLY prog.cob], [1], [],
[prog.cob:10: error: invalid INITIALIZE statement
prog.cob:12: error: invalid INITIALIZE statement
prog.cob:13: error: invalid INITIALIZE statement
])
AT_CLEANUP

Actually, we were quite pleased by the syntax of tests, it is easy to generate test files (using AT_DATA macro) and to test the execution of commands (using AT_CHECK macro), checking its exit code, its standard output and error output separately. It is even possible to combine checks to run additional checks in case of error or success. In general, the testsuite is easy to read and complete.

However, there were still some issues:

• At every update of the code or the tests, the testsuite runner has to be recompiled;

• Running the testsuite requires to be in the correct sub-directory, typically within the _build/ sub-directory;

• By default, tests are ran sequentially, even when many cores are available.

• The output is pretty verbose, showing all tests that have been executed. Failed tests are often lost in the middle of other successful tests, and you have to wait for the end of the run to start investigating them;

  ## -------------------------------------------- ##
  ## GnuCOBOL 3.2-dev test suite: GnuCOBOL Tests. ##
  ## -------------------------------------------- ##
    
  General tests of used binaries
    
    1: compiler help and information                   ok
    2: compiler warnings                               ok
    3: compiler outputs (general)                      ok
    4: compiler outputs (file specified)               ok
    5: compiler outputs (path specified)               ok
    6: compiler outputs (assembler)                    ok
    7: source file not found                           ok
    8: temporary path invalid                          ok
    9: use of full path for cobc                       ok
   10: C Compiler optimizations                        ok
   11: invalid cobc option                             ok
   12: cobcrun help and information                    ok
   13: cobcrun validation                              ok
   14: cobcrun -M DSO entry argument                   ok
   15: cobcrun -M directory/ default                   ok
   [...]
  

• There is no automatic way to update tests, when their output has changed. Every test has to be updated manually.

• In case of error, it is not always easy to rerun a specific test within its directory.

With autofonce, we tried to solve all of these issues...

Main Features of Autofonce

autofonce is written in a modern language, OCaml, so that it can handle a large testsuite much faster than GNU Autoconf. Since we do not expect users to have an OCaml environment set up, we provide binary versions of autofonce for both Linux (static executable) and Windows (cross-compiled executable) on Github.

autofonce does not use m4, instead, it has a limited support for a small set of predefined m4 macros, typically supporting m4 escape sequences (quadrigraphs), but not the addition of new m4 macros, and neither the execution of shell commands outside of these macros (yes, testsuites in GNU Autoconf are actually sh shell scripts with m4 macros...). In the case of GnuCOBOL, we were lucky enough that the testsuite was well written and avoided such problems (we had to fix only a few of them, such as including shell commands into AT_CHECK macros). The syntax of tests is documented here.

Some interesting features of autofonce are :

• autofonce executes the tests in parallel by default, using as many cores as available. Only failed tests are printed, so that the developer can immediately start investigating them;

• autofonce can be run from any directory in the project. A .autofonce file has to be present at the root of the project, to describe where the tests are located and in which environment they should be executed;

• autofonce makes it easy to re-execute a specific test that failed, by generating, within the test sub-directory, a script for every step of the test;

• autofonce provides many options to filter which tests should be executed. Tests can be specified by number, range of numbers, keywords, or negative keywords. The complete list of options is easily printable using autofonce run --help for example;

Additionnally, autofonce implements a powerful promotion mechanism to update tests, with the autofonce promote sub-command. For example, if you update a warning message in the compiler, you would like all tests where this message appears to be modified. With autofonce, it is as easy as:

# Run all tests at least once
autofonce run
# Print the patch that would be applied in case of promotion
autofonce promote
# Apply the patch above
autofonce promote --apply
# Combine running and promotion 10 times:
autofonce run --auto-promote 10

The last command iterates promotion 10 times: indeed, since a test may have multiple checks, and only the first failed check of the test will be updated during one iteration (because the test aborts at the first failed check), as many iterations as the maximal number of failed checks within a test may be needed.

Also, as for GNU Autoconf, autofonce generates a final log file containing the results with a full log of errors and files needed to reproduce the error. This file can be uploaded into the artefacts of a CI system to easily debug errors after a CI failure.

Conclusion

During our work on GnuCOBOL, autofonce improved a lot our user experience of running the testsuite, especially using the auto-promotion feature to update tests after modifications.

We hope autofonce could be used for other open-source projects that already use the GNU Autoconf testsuite. Of course, it requires that the testsuite does not make heavy use of shell features and mainly relies on standard m4 macros.

We found that the format of GNU Autoconf tests to be quite powerful to easily check exit codes, standard outputs and error outputs of shell commands. autofonce could be used to help using this format in projects, that do not want to rely on an old tool like GNU Autoconf, and are looking for a much more modern test framework.

About OCamlPro:

OCamlPro is a R&D lab founded in 2011, with the mission to help industrial users benefit from experts with a state-of-the-art knowledge of programming languages theory and practice.

• We provide audit, support, custom developer tools and training for both the most modern languages, such as Rust, Wasm and OCaml, and for legacy languages, such as COBOL or even home-made domain-specific languages;
• We design, create and implement software with great added-value for our clients. High complexity is not a problem for our PhD-level experts. For example, we helped the French Income Tax Administration re-adapt and improve their internally kept M language, we designed a DSL to model and express revenue streams in the Cinema Industry, codename Niagara, and we also developed the prototype of the Tezos proof-of-stake blockchain from 2014 to 2018.
• We have a long history of creating open-source projects, such as the Opam package manager, the LearnOCaml web platform, and contributing to other ones, such as the Flambda optimizing compiler, or the GnuCOBOL compiler.
• We are also experts of Formal Methods, developing tools such as our SMT Solver Alt-Ergo (check our Alt-Ergo Users' Club) and using them to prove safety or security properties of programs.

Please reach out, we'll be delighted to discuss your challenges: contact@ocamlpro.com or book a quick discussion.