Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTML report shows wrong data with multiple param values #108

Open
jtippet opened this issue Feb 25, 2020 · 0 comments · May be fixed by #109
Open

HTML report shows wrong data with multiple param values #108

jtippet opened this issue Feb 25, 2020 · 0 comments · May be fixed by #109

Comments

@jtippet
Copy link

jtippet commented Feb 25, 2020

Create a param named X and run a test with multiple param values, like benchmark --reporter=html --param=X:+:0:1:10. The HTML report will show a nice summary page that shows a line plot of how the benchmarks fare on each param value.

Unfortunately, the data is just wrong. The data is correct only for the first (leftmost) column; all other columns have the data mixed up between the various series.

Here's an example:
image

Note how the orange and blue series seem to trade off on looking bad. That's not backed up by the raw data -- in the raw data, the orange series should always be bad, and the blue should always be okay.

A corrected graph would look like this:
image

@jtippet jtippet linked a pull request Feb 25, 2020 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant