Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve the benchmark runner script #24

Open
cmnrd opened this issue Nov 22, 2021 · 0 comments
Open

Improve the benchmark runner script #24

cmnrd opened this issue Nov 22, 2021 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@cmnrd
Copy link
Contributor

cmnrd commented Nov 22, 2021

The benchmark runner script is already quite powerful, but there are some features lacking and optimizations that should be made:

  1. Currently, the runner uses an environment variable to point to the LF repository, and it assumes a fixed version. We should make this fully configurable, allowing to compare two different versions of lfc. This is particularly important to assist future development and assess the impact of optimizations or big changes in the code base.
  2. The script builds each benchmark before running it, even if the same benchmark configuration has been build before. This not only consumes time, but the build artifacts also accumulate and take a significant amount of hard disk space. We should try to optimize this and have some kind of build cache. Doing this, however, is not trivial as hydra expects all runs to be fully independent. We would need to break out of this, but luckily Python should be flexible enough. Possibly, a custom Sweeper could help here. Ideally, this cache would also be persistent across invocations of the benchmark runner. However, then we need to detect somehow when benchmarks need to be rebuilt.
  3. For some experiments, it could be beneficial to run benchmarks in a random order (see @petervdonovan's comment here). This should be possible by implementing a custom Sweeper that orders jobs randomly. By setting iterations=1 and iterating over another variable (for instance +it=range(0,30)) we could run benchmarks N times in a random order.
@cmnrd cmnrd added the enhancement New feature or request label Nov 22, 2021
@cmnrd cmnrd self-assigned this Nov 22, 2021
@cmnrd cmnrd transferred this issue from lf-lang/lingua-franca Jul 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant