Testing and Regression Tests

During programming you will end up testing your program on small datasets to ensure it works as expected. You should aim to develop your program so that you can easily test it, and keep copies of the tests available for future use.

You should create a tests directory within your project and keep all the tests you have used within it. Include a makefile with a top-level target make check that can check if your code is working as expected. This is useful firstly as you develop, but also for porters and other developers to check that they haven't broken anything. It also gives practical examples of how the program was supposed to work!

A good practice in this regard is to make it possible to run a code in batch mode, taking input from a file and outputting to a file. The first time round you can visually inspect the output to ensure it is correct. Next, keep a copy of the correct output, and you can check that the code has not regressed later by re-running the test and ensuring the output remains the same.

If you keep the output in a simple style, without putting datestamps, etc. in it, you can then compare two sets of output using the Unix diff command.

Here is an example of comparing outputs with a standard file and incorporating it into a makefile. To execute the tests type make all_tests

all_tests: test1 test2 test2
test1: test1.orig test1.out
@(diff test1.orig test1.out > /dev/null) || echo "Test1 Failed"

For more complex numerical output, we may need to check an array is within a small delta rather than exactly identical to an original output. Again keep the output format simple, but parse the output in python script, etc. and let it return a TRUE/FALSE if the test script, and signal an error using the exit code with sys.exit(1).

For more information about how to construct a GNU makefile, go here.

Useful Make Idioms