You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The benchmark runner script is already quite powerful, but there are some features lacking and optimizations that should be made:
Currently, the runner uses an environment variable to point to the LF repository, and it assumes a fixed version. We should make this fully configurable, allowing to compare two different versions of lfc. This is particularly important to assist future development and assess the impact of optimizations or big changes in the code base.
The script builds each benchmark before running it, even if the same benchmark configuration has been build before. This not only consumes time, but the build artifacts also accumulate and take a significant amount of hard disk space. We should try to optimize this and have some kind of build cache. Doing this, however, is not trivial as hydra expects all runs to be fully independent. We would need to break out of this, but luckily Python should be flexible enough. Possibly, a custom Sweeper could help here. Ideally, this cache would also be persistent across invocations of the benchmark runner. However, then we need to detect somehow when benchmarks need to be rebuilt.
For some experiments, it could be beneficial to run benchmarks in a random order (see @petervdonovan's comment here). This should be possible by implementing a custom Sweeper that orders jobs randomly. By setting iterations=1 and iterating over another variable (for instance +it=range(0,30)) we could run benchmarks N times in a random order.
The text was updated successfully, but these errors were encountered:
The benchmark runner script is already quite powerful, but there are some features lacking and optimizations that should be made:
lfc
. This is particularly important to assist future development and assess the impact of optimizations or big changes in the code base.iterations=1
and iterating over another variable (for instance+it=range(0,30)
) we could run benchmarksN
times in a random order.The text was updated successfully, but these errors were encountered: