Introduction

The Specs Benchmark benchmarks the Specs behaviour‐driven‐design framework by running a selection of Spec’s own test suite. Each workload of the Specs Benchmark runs several unit tests or specifications.

The unit tests and specifications have been selected according to the following criteria:

No class loader leakage
Several unit tests and specifications leak class loaders when run for several iterations.
No internal use of the Scala compiler
org.specs.literate.literateSpecifications relies on the Scala compiler at runtime, which is already exercised by a different benchmark.
No disproportionate amounts of file system activity
org.specs.io.ioSpecifications primarily exercises the file system rather than the VM.
Compatibility with a separate test class loader
Several unit tests and specifications make overly strong assumptions about the class loader hierarchy, which prevents the use of a separate test class loader (see below).

Note that a few of the selected unit tests and specifications exhibit test failures; this is due to the use of a benchmark harness resulting in subtle, but unavoidable differences in setup.

Although both the testing framework and its test suite consist of executable code, the benchmark harness treats the latter special: Its constituent test classes are loaded with a temporary class loader and discarded after each iteration of the benchmark. The tests are effectively one‐off code that is loaded anew for each iteration; only the framework itself is loaded just once. This particular benchmarking methodology, which emphasises class loading and one‐off execution, is deemed appropriate for testing frameworks like Specs, whose primary input is executable code.

Threading

The Specs Benchmark is externally single‐threaded and internally single‐threaded.

Acknowledgments

We are grateful to Eric Torreborre, the author of Specs, for giving us a sneak preview of Specs2 for evaluation.