HACKING.adoc

Running the testsuite

Useful Makefile targets

make parallel

Runs the tests in parallel using the GNU parallel tool: tests run twice as fast with no difference in output order.

make all-foo, make parallel-foo

Runs only the tests in the directories whose name starts with foo: parallel-typing, all-lib, etc.

make one DIR=tests/foo

Runs only the tests in the directory tests/foo. This is often equivalent to cd tests/foo && make, but sometimes the latter breaks the test makefile if it contains fragile relative filesystem paths. Such errors should be fixed if you find them, but make one DIR=…​ is the more reliable option as it runs exactly as make all which is heavily tested.

make one TEST=tests/foo/bar.ml

Runs only the specific test tests/foo/bar.ml.

make one LIST=tests.txt

Runs only the tests in the directories listed in the file tests.txt. The file should contain one directory per line; for instance, if the contents of tests.txt are:

tests/foo
tests/bar
tests/baz

then this will run all the tests in those three directories.

make promote DIR=tests/foo, make promote TEST=tests/foo/bar.ml, make promote LIST=file.txt

Most tests run a program and compare the result of the program, stored in a file foo.result, with a reference output, stored in foo.reference; the test fails if the two outputs differ. Similarly, many other tests are expect tests, with the expected output following the code inline in the test file. In both cases, sometimes a change in the result is innocuous, as it comes from an intended change in output instead of a regression. make promote is like make one, but for each failing test, it copies the new results into the reference files (or into the expect test expected output), making the failing test pass again. Whenever you use this rule please check carefully, using git diff, that the changes really correspond to an intended output difference, and not to a regression. You then need to commit the changes to the reference files (or expect test output), and your commit message should explain why the output changed. make promote takes the same variables as make one to determine which tests to run (there is no analog to make all).

Useful environment variables

KEEP_TEST_DIR_ON_SUCCESS=1

Keeps temporary output files from a test run. This is handy to validate the content of temporary output files, run a produced executable by hand, etc.

OCAMLTESTDIR=/tmp/foo

Changes the output directory to the specified one. This should be combined with KEEP_TEST_DIR_ON_SUCCESS=1 to inspect the test output. By default OCAMLTESTDIR is _ocamltest.

Creating a new test

Dimensioning the tests

By default, tests should run well on small virtual machines (2 cores, 2 Gb RAM, 64 or 32 bits), taking at most one minute, and preferably less than 10 seconds, to run on such a machine.

Some machines used for continuous integration are more capable than that. They use the OCAML_TEST_SIZE environment variable to report the available resources:

OCAML_TEST_SIZE	Resources	Word size
1 or unset	2 cores, 2 Gb RAM	32 or 64 bits
2	4 cores, 4 Gb RAM	64 bits
3	8 cores, 8 Gb RAM	64 bits