Sunday, January 1, 2017

on tests

I prefer my tests simple and flexible, designed around the application rather than imposed on it. These days, I usually start with basic functions and assertions; and add what I need when I need it. The holy grail for me would be a reusable library that allows the same progressive approach without getting in the way.

The flexibility I'm looking for is the freedom to structure my tests any way I feel like, to change my mind as many times as I have to without rewriting everything; and the capability to include/exclude tests in a run without touching the code.

On the subject of simplicity, besides being a requirement for flexibility; benchmarking is worth mentioning. I find that most test frameworks either completely skip the subject, or make to big a deal out of it. I want the option to warm up caches, repeat tests and print a table of run times without bending over backwards.

I've found that implementing tests as regular functions helps fulfill above stated goals. And I've grown fond of the idea of reusing regular assertions for signaling errors. In the end, all assertions are tests; which means that you want to deal with them the same way in any given scenario, embedded or external makes no difference. Common sense recognizes the fact that organizing something as volatile as tests into rigid hierarchies is futile; tagging tests with keywords allows a more dynamic, gradual approach.

Each test in libc4life is implemented as a regular function that contains a set of tags in it's definition. Tests can be grouped into suites, further modularized using sub tests to specify additional tags, and called directly from within other tests. When running a suite of tests, two sets of tags may be specified; one set of tests to include, and one containing tests to skip. Tests matching tags in the first set are included, unless they match tags in the second set.

Each test run includes a specified number of warmups and repetitions, and prints a formatted table of test names and run times to stdout. Assertion errors are first class and can be dealt with like any other error, and defining NDEBUG allows test code to control if failures should terminate or be reported at the end of the run.

peace, out