Bump benchee from 1.0.1 to 1.3.0

[dependabot]

Bumps benchee from 1.0.1 to 1.3.0.

Release notes

Sourced from benchee's releases.

1.3.0

A big swath of bug fixes and improvements. The highlights certainly are fixes and conveniences around saved benchmarks and loading them again via the new Benchee.report/1. The other big one is saving a lot of memory (and time!) when processing big inputs. Sadly the latter comes with some breaking changes for plugins, but they are well justified and shouldn't actually affect any plugin in practice.

There's also a known issue for elixir 1.14.0 to 1.16.0-rc.0 - read up about it.

Features (User Facing)

• System information now includes whether or not the JIT is enabled (erlang docs).
• Benchee now let's you know when it's calculating statistics or running the formatters. Helps when you wonder what's taking so long or blows up memory.
• Benchee.report/1 introduced if you just want to load saved benchmarks and report on them (---> run formatters).
• Configuration times will now be displayed in a more human friendly format (1 min 12 s vs. 1.2 min). Thanks to @​drobnyd.

Bugfixes (User Facing)

• Memory usage should be massively reduced when dealing with larger sets of data in inputs or benchmarking functions. They were needlessly sent to processes calculating statistics or formatters which could lead to memory blowing up.
• Similarly, inputs and benchmarking functions will no longer be saved when using the :save option, this makes it immensely faster and depending on the size of the data a lot slower (I have an example with a factor 200x for the size). The side effect of this is that you also can't use :load and run the benchmarks saved again from just the file, this was never an intended use case though (as loading happens after benchmarking by default). You also still should have the benchmarking script so it's also not needed.
• Fix a bug where relative statistics would always rely on the inputs provided in the config, which can break when you load saved benchmarks but don't specify the inputs again.

Breaking Changes (Plugins)

Woopsie, didn't wanna do any of these in 1.x, sorry but there's good reason :( However, counting them as bugs.

• Formatters have lost access to benchmarking functions and the inputs, this is to enable huge memory and run time savings when using a lot of data. I also believe they should not be needed for formatters, please get in touch if this is a problem so we can work it out. In detail this means:
  • Each Benchee.Scenario struct in a formatter will have :function and :input set to nil
  • The inputs list in Configuration retains the input names, but the values will be set to :scrubbed_see_1_3_0_changelog. It may be completely scrubbed in the future, use the newly introduced input_names instead if you need easy access to all the input names at once.
  • Technically speaking formatters haven't generally lost access, only if they are processed in parallel - so not if it's the only formatter or if it's used via a function (formatters: [fn suite -> MyFormatter.output(suite) end]. Still, should not be used or relied upon.

Features (Plugins)

• jit_enabled? is exposed as part of the suite.system struct
• Yes, Benchee.System is now a struct so feel easier about relying on the fields
• Configuration now has an input_names key that holds the name of all inputs, for the reasoning, see above.
• The more human friendly formats are accessible via Format.format_human/2 or Duration.format_human/1 (& friends)

1.2.0

Long time, huh? 😉 I'm not gonna repeat all that, but I'm happy benchee is in a place where it just works and doesn't need too much support. Biggest feature here is the implementation of the Table.Reader protocol for a better Livebook experience.

Features (User Facing)

• Trying to benchmark evaluated functions will now result in a warning. Thanks @​BrooklinJazz, @​czrpb, @​aar2dee2,@​ReecesPeanutButterCodes.
• Add support for the Table.Reader protocol so that benchee works out of the box in Livebook. See #369, big shoutout to @​akoutmos and sorry for keeping it unreleased for so long.

Bugfixes (User Facing)

• Removed Elixir 1.16 compiler warnings around +0.0 and -0.0. Thanks @​tomciop.
• Building an escript and running it is fixed. See #384 and thanks @​Munksgaard!

1.1.0

Long time, huh? I'm sorry, combination of priorities, difficult to fix bugs, stress and arm problems kept me away too long. This release brings major features long developed and now finally released (reduction measurements + profiler after run), along with a critical bugfix around measurement accuracy for very fast functions (nanoseconds).

Features (User Facing)

• Reduction counting/measurements was implemented. Basically, it's a rather stable unit of execution implemented by the BEAM that measures in more abstract manner how much work was done. It's helpful, as it shouldn't be affected by load on the system. Check out the docs.
• You can now dive straight into profiling from your benchmarks by using the profiling feature. See the docs - thanks @​pablocostass

Default Behavior Changes (User Facing)

... (truncated)

Changelog

Sourced from benchee's changelog.

1.3.0 (2023-12-22)

A big swath of bug fixes and improvements. The highlights certainly are fixes and conveniences around saved benchmarks and loading them again via the new Benchee.report/1. The other big one is saving a lot of memory (and time!) when processing big inputs. Sadly the latter comes with some breaking changes for plugins, but they are well justified and shouldn't actually affect any plugin in practice.

There's also a known issue for elixir 1.14.0 to 1.16.0-rc.0 - read up about it.

Features (User Facing)

• System information now includes whether or not the JIT is enabled (erlang docs).
• Benchee now let's you know when it's calculating statistics or running the formatters. Helps when you wonder what's taking so long or blows up memory.
• Benchee.report/1 introduced if you just want to load saved benchmarks and report on them (---> run formatters).
• Configuration times will now be displayed in a more human friendly format (1 min 12 s vs. 1.2 min). Thanks to @​drobnyd.

Bugfixes (User Facing)

• Memory usage should be massively reduced when dealing with larger sets of data in inputs or benchmarking functions. They were needlessly sent to processes calculating statistics or formatters which could lead to memory blowing up.
• Similarly, inputs and benchmarking functions will no longer be saved when using the :save option, this makes it immensely faster and depending on the size of the data a lot slower (I have an example with a factor 200x for the size). The side effect of this is that you also can't use :load and run the benchmarks saved again from just the file, this was never an intended use case though (as loading happens after benchmarking by default). You also still should have the benchmarking script so it's also not needed.
• Fix a bug where relative statistics would always rely on the inputs provided in the config, which can break when you load saved benchmarks but don't specify the inputs again.

Breaking Changes (Plugins)

Woopsie, didn't wanna do any of these in 1.x, sorry but there's good reason :( However, counting them as bugs.

• Formatters have lost access to benchmarking functions and the inputs, this is to enable huge memory and run time savings when using a lot of data. I also believe they should not be needed for formatters, please get in touch if this is a problem so we can work it out. In detail this means:
  • Each Benchee.Scenario struct in a formatter will have :function and :input set to nil
  • The inputs list in Configuration retains the input names, but the values will be set to :scrubbed_see_1_3_0_changelog. It may be completely scrubbed in the future, use the newly introduced input_names instead if you need easy access to all the input names at once.
  • Technically speaking formatters haven't generally lost access, only if they are processed in parallel - so not if it's the only formatter or if it's used via a function (formatters: [fn suite -> MyFormatter.output(suite) end]. Still, should not be used or relied upon.

Features (Plugins)

• jit_enabled? is exposed as part of the suite.system struct
• Yes, Benchee.System is now a struct so feel easier about relying on the fields
• Configuration now has an input_names key that holds the name of all inputs, for the reasoning, see above.
• The more human friendly formats are accessible via Format.format_human/2 or Duration.format_human/1 (& friends)

1.2.0 (2023-11-09)

Long time, huh? 😉 I'm not gonna repeat all that, but I'm happy benchee is in a place where it just works and doesn't need too much support. Biggest feature here is the implementation of the Table.Reader protocol for a better Livebook experience.

Features (User Facing)

• Trying to benchmark evaluated functions will now result in a warning. Thanks @​BrooklinJazz, @​czrpb, @​aar2dee2,@​ReecesPeanutButterCodes.
• Add support for the Table.Reader protocol so that benchee works out of the box in Livebook. See #369, big shoutout to @​akoutmos and sorry for keeping it unreleased for so long.

Bugfixes (User Facing)

• Removed Elixir 1.16 compiler warnings around +0.0 and -0.0. Thanks @​tomciop.
• Building an escript and running it is fixed. See #384 and thanks @​Munksgaard!

1.1.0 (2022-03-08)

Long time, huh? I'm sorry, combination of priorities, difficult to fix bugs, stress and arm problems kept me away too long. This release brings major features long developed and now finally released (reduction measurements + profiler after run), along with a critical bugfix around measurement accuracy for very fast functions (nanoseconds).

Features (User Facing)

• Reduction counting/measurements was implemented. Basically, it's a rather stable unit of execution implemented by the BEAM that measures in more abstract manner how much work was done. It's helpful, as it shouldn't be affected by load on the system. Check out the docs.
• You can now dive straight into profiling from your benchmarks by using the profiling feature. See the docs - thanks @​pablocostass

... (truncated)

Commits

• 5da43c6 add link/hint to the known issues section to changelog
• fcca8c8 Extend configuration docs about the progress prints
• 11cae4d Update output example in README
• 54b7c9b report adjustments
• 21845bf Set version number to 1.3
• ef85961 prep changelog
• 2e588d9 run newest credo
• 79dd7e3 find and fix particularly nasty bug that lead to real bad output always appen...
• 3af6a7b changelog for unit changes
• 0f4c2fd Implement format_human and required callbacks for all units
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)