Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmark biaised due to no fence around input #493

Open
ogxd opened this issue Dec 22, 2023 · 0 comments
Open

Benchmark biaised due to no fence around input #493

ogxd opened this issue Dec 22, 2023 · 0 comments

Comments

@ogxd
Copy link

ogxd commented Dec 22, 2023

Context

The benchmarks uses of black_box as a "fence" around outputs to prevent compiler optimization such as optimizing them out (because the compiler would otherwise consider them not used). This is what criterion.rs applies by default.

For instance in this lookup benchmark:

black_box(m.get(&i));

However, the same must be done for inputs, or they might get optimized along with the code of the benched function. This can make a significant difference in some scenarios. This article explains it well:
https://gendignoux.com/blog/2022/01/31/rust-benchmarks.html#effect-of-stdhintblack_box-on-a-benchmarking-loop

a crucial point is to put a black_box both on the inputs and on the outputs

I have tested it on my side, and I do see significant differences (tested on a MacBook pro M1).

Todo

Change:

for i in $keydist.take(SIZE) {
     black_box(m.get(&i));
}

To:

for i in $keydist.take(SIZE) {
     black_box(m.get(black_box(&i)));
}

Same probably applies for other parts of this benchmark.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant