README.perf.md

Legion Performance Test Infrastructure

This document describes Legion's performance regression test infrastructure. Before reading this document, we recommend you read the documentation on the test infrastructure for correctness, because the performance infrastructure makes significant use of the same tools.

As with correctness, the performance test infrastructure has a number of components. The flow is roughly:

• The test.py script again encapsulates all test logic.
• Another script perf.py runs individual tests and uploads the resulting measurements.
• Raw measurements are stored in the perf-data repository.
• The script tools/perf_chart.py parses the raw measurements to produce a table aggregating the results, which it uploads to the same repository.
• An HTML-based front-end renders the aggregated results.

This may seem complicated, but critically all of the moving parts are configured explicitly, and versioned and stored in Git repositories (including the data). There is no local configuration that is required and no local state that can be corrupted or lost.

This is in opposition to, primarily, a system you might install locally: something with a database, a web server, and a job scheduler. The advantage of such an approach is that it is completely self-hosted and minimizes reliance on external services. However, this approach also suffers from high operational complexity. Without special care, local installations of tools can come to depend on the implicit configuration of the machine, making those installations brittle. We suffered a hard drive crash on the machine serving our previous system, lost both the database and locally modified configuration of the tool, and found that our documentation on the installation procedure was insufficient to get the system running again. We built our current solution to ensure this would not happen again.

Running Performance Tests

All tests run through the test.py script:

PERF_ACCESS_TOKEN=... DEBUG=0 ./test.py --test=perf

A couple of notes:

• PERF_ACCESS_TOKEN is a Github personal access token. The token must be for a user account with write permission to the perf-data repository. We have a special Github account which we use for this purpose.
• DEBUG=0 is required to ensure performance.
• --test=perf requests that the performance tests be run.

These performance tests are enabled in Gitlab, so they run automatically on every commit. In addition to this, there is a cronjob that runs nightly on Sapling to measure performance on a machine with beefier hardware. Ideally these would also run via Gitlab, but proper cluster management would be required and is beyond the scope of this document.

Nightly Performance Tests

We conduct nightly performance tests on a variety of machines that are not practical to hook up to Gitlab. The configurations for these tests (along with installation instructions) are kept in the tools/nightly_perf/MACHINE directory for each machine.

How to Add New Tests

Tests are in test.py under the perf test stage. The specific way in which you add a new test depends on the kind of test you're adding:

• C++ test in the main Legion repository: Add the test to legion_cxx_perf_tests at the top of the file.
• Regent test in the main Legion repository: Add the test to regent_perf_tests at the top of the file.
• External C++ or Regent test: Add code to the end of run_test_perf to clone, build, and run the repository. Remember to run the command through perf.py as described below.

Capturing Measurements

Measurement Storage and Processing

Visualization Front-end