Project Documentation
Core Projects
The Scala Benchmark Suite is based on the latest release (9.12, nicknamed “Bach”) of the DaCapo benchmark suite, a suite already popular among JVM researchers which specifically strives for “ease of use.” The Scala Benchmark Suite adds 12 Scala benchmarks, summarized in the table below, to the 14 Java benchmarks of the DaCapo benchmark suite.
| Benchmark | Description | Inputs (#) |
|---|---|---|
| actors | Trading sample with Scala and Akka actors | tiny–gargantuan (6) |
| apparat | Framework to optimize ABC, SWC, and SWF files | tiny–gargantuan (6) |
| factorie | Toolkit for deployable probabilistic modeling | tiny–gargantuan (6) |
| kiama | Library for language processing | small–default (2) |
| scalac | Compiler for the Scala 2 language | small–large (3) |
| scaladoc | Scala documentation tool | small–large (3) |
| scalap | Scala classfile decoder | small–large (3) |
| scalariform | Code formatter for Scala | tiny–huge (5) |
| scalatest | Testing toolkit for Scala and Java programmers | small–huge (4) |
| scalaxb | XML data‐binding tool | tiny–huge (5) |
| specs | Behaviour‐driven design framework | small–large (3) |
| tmt | Stanford Topic Modeling Toolbox | tiny–huge (5) |
To allow for easy experimentation with different inputs, several benchmarks come with more than the two to four input sizes (small, default, large, and huge) supported by the DaCapo benchmarks. For the Scala Benchmark Suite, this gives rise to 51 unique workloads, i.e., benchmark‐input combinations. The DaCapo benchmark suite offers 44 such workloads.
To ensure its representativeness, the Scala benchmark suite is based on a large set of applications from a range of different domains. In fact, only two categories of application are completely absent from the Scala benchmark suite but present in the latest DaCapo benchmark suite: client/server applications (tomcat, tradebeans, and tradesoap) and in‐memory databases (h2), the former of which is also absent in the DaCapo suite’s earlier release (2006-10). The absence of client/server applications is explained by the fact that all such DaCapo benchmarks rely on either a Servlet container or an application server, a dependency which a corresponding Scala benchmark will have to share. The absence of in‐memory databases is explained by the fact that, to the best of our knowledge, no such Scala application yet exists that is more than a thin wrapper around Java code.
While the range of domains covered is broad, several benchmarks nevertheless occupy the same niche. This was a deliberate choice made to avoid bias from preferring one application over another in a domain where Scala is frequently used: automated testing (scalatest, specs), source‐code processing (scaladoc, scalariform), or machine‐learning (factorie, tmt). It has been shown that the inclusion of several applications from the same domain is indeed justified; in particular, the respective benchmarks all exhibit a distinct instruction mix.