The Specs Benchmark benchmarks the Specs behaviour‐driven‐design framework by running a selection of Spec’s own test suite. Each workload of the Specs Benchmark runs several unit tests or specifications.
The unit tests and specifications have been selected according to the following criteria:
Note that a few of the selected unit tests and specifications exhibit test failures; this is due to the use of a benchmark harness resulting in subtle, but unavoidable differences in setup.
Although both the testing framework and its test suite consist of executable code, the benchmark harness treats the latter special: Its constituent test classes are loaded with a temporary class loader and discarded after each iteration of the benchmark. The tests are effectively one‐off code that is loaded anew for each iteration; only the framework itself is loaded just once. This particular benchmarking methodology, which emphasises class loading and one‐off execution, is deemed appropriate for testing frameworks like Specs, whose primary input is executable code.