Benchmarks

This page collects data on benchmark measurements of Lingua Franca programs. The hardware on which these tests are run is detailed below.

Internal Benchmarks

These are programs that are not part of any standard benchmark suite, but which we use to measure the impact of design changes.

• test/C/TimeLimit.lf: This program sends a counting sequence of length 10,000,002 from a source to a destination. Is is meant to measure the basic overhead of reaction invocation.
  Mac1: 600 msec (Oct. 14, 2020)
  AGX: 1222 msec (AVG), 1213 msec (MIN), 1281 msec (MAX), 1223 msec (MED) (100 runs - Oct. 14, 2020)
• test/C/TimeLimitThreaded.lf: This version of the previous test uses the threaded runtime. There is no exploitable parallelism, so no speedup is expected.
  Mac1 (with threads = 8): 1.1 sec (Oct. 13, 2020)
  AGX: 1717 msec (AVG), 1712 msec (MIN), 1810 msec (MAX), 1718 msec (MED) (100 runs - Oct. 14 2020)

Savina Benchmarks

The Savina benchmark suite for actor languages and frameworks provides a number of useful patterns for measuring performance of Lingua Franca programs. See the paper on Savina. None of these benchmarks has any notion of time, and not all comparisons with the actor implementations will be fair.

• benchmark/C/Savina/PingPong.lf: This benchmark tests a feedback interaction where Ping sends a message to Pong, which responds, triggering Ping to send another message. This gets repeated 1,000,000 times. This version of the benchmark runs in the unthreaded runtime. Based on [https://www.scala-lang.org/old/node/54].
  Mac1: 105 msec (Oct. 13, 2020)
• benchmark/C/Savina/PingPongThreaded.lf: Version of the previous benchmark runs in the threaded runtime with 8 worker threads. There is no parallelism, so no speedup is expected.
  Mac1: 165 msec (Oct. 13, 2020)
• benchmark/C/Savina/PingPongMultiThreaded.lf: Version of the previous benchmark runs in the threaded runtime with 8 worker threads and has two independent Ping-Pong benchmarks running together. There is parallelism, so speedup is expected, but instead, we get dramatic slowdown!
  Mac1: 32.6 sec (Oct. 13, 2020)

Hardware

Mac1: Model Name: MacBook Pro
Model Identifier: MacBookPro15,1
Processor Name: Intel Core i9
Processor Speed: 2.3 GHz
Number of Processors: 1
Total Number of Cores: 8
L2 Cache (per Core): 256 KB
L3 Cache: 16 MB
Hyper-Threading Technology: Enabled
Memory: 32 GB
Operating System: macOS 10.14.6 (Mojave)

AGX: Model Name: NVIDIA Jetson AGX Xavier
Model Identifier: Jetson-AGX
Processor Name: ARMv8 Processor rev 0 (v8l)
Max Processor Speed: 2265.6001 MHz (fixed for this benchmark)
Number of Processors: 1
Total Number of Cores: 8
L2 Cache (per Core): 2048 KB
L3 Cache: 4096 KB
Hyper-Threading Technology: N/A
Memory: 16 GB
Operating System: Ubuntu 18.04 (Bionic) - JetPack 4.3