JMH

Your JMH benchmarks can generate a number of JSON files storing benchmark results, but it is difficult to produce a good-looking graph from them. Carte-JMH was written to produce graphs and tables that can be displayed on a website (see the CSS4J Benchmark results).

Running the JMH report application

To run the benchmark report app on JMH-produced JSON files, you have to prepare a configuration file and execute:

1. ./gradlew build uberjar
2. java -jar carte-jmh/build/libs/carte-jmh-<version>-all.jar --config=<path-to-config-file> *.json

Rather than describing all the elements in the configuration file, it is probably easier to use a real example.

Learning by example

https://github.com/css4j/carte/blob/master/carte-jmh/examples/sac-benchmark.xml

As described in the Wiki Home, it has a <reportdef> element with a <name> child.

Then, look at the following lines that declare a benchmark based on SACBenchmark.java:

    <benchmark id="sac-226kb">
        <class>io.sf.carte.mark.css.SACBenchmark</class>
        <caption>SAC benchmark</caption>
        <subcaption>Medium-large file (226 kB)</subcaption>
    </benchmark>

You can add as many <benchmark> elements as you like.

One should provide a description for the X axis:

    <x-axis-title>SAC Implementation</x-axis-title>

And you probably want to assign more readable descriptions (together with specific colours) to the methods in the benchmarks. For which you will use regular expressions in <method> elements:

    <methods>
        <method regex=".+SACParseStyleSheet$" color="#c04d4d">Css4j</method>
        <method regex=".+Batik$" color="#61c061">Batik CSS</method>
        <method regex=".+SSParser$" color="#4d4dc0">SS CSSParser</method>
    </methods>

Now, give a description to the units:

    <units>
        <unit-title unit-id="ops/s">Throughput (ops/s)</unit-title>
        <unit-title unit-id="ops/ms">Throughput (ops/ms)</unit-title>
        <unit-title unit-id="ops/us">Throughput (ops/μs)</unit-title>
        <unit-title unit-id="ops/ns">Throughput (ops/ns)</unit-title>
        <unit-title unit-id="s/op">Average time per operation (s)</unit-title>
        <unit-title unit-id="ms/op">Average time per operation (ms)</unit-title>
        <unit-title unit-id="us/op">Average time per operation (μs)</unit-title>
        <unit-title unit-id="ns/op">Average time per operation (ns)</unit-title>
    </units>

The document stores

• The element with a fileStore id tells that the directory ~/www/css4j.github.io/benchmark would be a FileStore containing the fallback images.

• In the element with documentStore id of that file, it is configured that the file ~/www/css4j.github.io/sac-mark.html is a DocumentStore where the SVG graphs will be put.

    <storage>
        <store id="fileStore" classname="io.sf.carte.report.FileStore">
            <directory>${java.io.tmpdir}/benchmark</directory>
        </store>
        <store id="documentStore" classname="io.sf.carte.report.DocumentStore">
            <pathname>${user.home}/www/css4j.github.io/sac-mark.html</pathname>
            <fallback baseuri="benchmark" format="png">
                ${user.home}/www/css4j.github.io/benchmark
            </fallback>
        </store>
    </storage>

All that you have to do is to download your copy of the sac-mark.html file and modify the paths in sac-benchmark.xml as necessary.

Then you could prepare a small script called sac-benchmark-charts.sh and put it in carte-jmh/examples:

#!/usr/bin/bash
CARTE_VERSION=0.1.0-SNAPSHOT
java -jar ../build/libs/carte-jmh-${CARTE_VERSION}-all.jar --config=sac-benchmark.xml "$@"

Now open a shell and execute it (again, in carte-jmh/examples which contains all the JSON files):

cd carte-jmh/examples
./sac-benchmark-charts.sh sac*.json

And you will reproduce the JMH charts.