[move] Benchmarking historical transactions #15329
Open
+1,175
−145
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
⏱️ 2h 53m total CI duration on this PR
|
9 tasks
This stack of pull requests is managed by Graphite. Learn more about stacking. |
9 tasks
georgemitenkov
changed the title
georgemitenkov
force-pushed
the
george/loader-v2-benchmark
branch
from
5f0697c
to
9d3bc98
Compare
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
georgemitenkov
force-pushed
the
george/loader-v2-todos-script-location
branch
from
3b035a7
to
429acfd
Compare
georgemitenkov
force-pushed
the
george/loader-v2-benchmark
branch
2 times, most recently
from
4beb0d6
to
67ec63d
Compare
gelash
reviewed
Nov 20, 2024
georgemitenkov
force-pushed
the
george/loader-v2-todos-script-location
branch
from
429acfd
to
a351639
Compare
georgemitenkov
force-pushed
the
george/loader-v2-benchmark
branch
from
67ec63d
to
2d8c42b
Compare
Base automatically changed from
george/loader-v2-todos-script-location
to
main
November 20, 2024 21:51
georgemitenkov
force-pushed
the
george/loader-v2-benchmark
branch
2 times, most recently
from
9ff5305
to
8d71d53
Compare
georgemitenkov
force-pushed
the
george/loader-v2-benchmark
branch
3 times, most recently
from
bc4d63b
to
e5d9058
Compare
georgemitenkov
requested review from
gregnazario,
JoshLind,
davidiw,
wrwg,
zekun000 and
vgao1996
as code owners
georgemitenkov
changed the title
georgemitenkov
changed the title
georgemitenkov
requested review from
msmouse,
runtian-zhou,
igor-aptos,
gelash and
ziaptos
georgemitenkov
force-pushed
the
george/loader-v2-benchmark
branch
2 times, most recently
from
570e144
to
fc76656
Compare
georgemitenkov
force-pushed
the
george/loader-v2-benchmark
branch
from
26a4e0a
to
85f4f11
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR introduces a tool to benchmark past transactions (and correctly, unlike existing
aptos-debugger move execute-past-transactions ...).Summary
Aptos debugger tool uses an RPC under the hood when running transactions. This means state view access latencies can be huge. Also, only executes transactions in blocks, without shared caches across blocks like the real executor does.
This PR introduces a better tool to benchmark execution of past transactions. The user has to provide first and last versions of the interval that need to be benchmarked. The tool partitions transactions in the specified closed interval into blocks, and runs all blocks end-to-end, measuring the overall time. During this run, executor is shared (and so are environment and module caches).
There is no commit here, only execution time. For each block, we maintain the state (read-set estimated from 1 run before the benchmarks) on top of which it should run. And so, the benchmark just runs in sequence blocks on top of their initial states (outputs are only used for comparison against the on-chain data).
The tool allows one to override configs to experiment with new features, or with how the execution would look like without some features. For now, we support:
In the future, we can add more overrides: gas schedule, modules, etc.
It also computes the diffs between expected and new overridden outputs, e.g.:
Example
For example, say we have a new feature, e.g.,
ENABLE_LOADER_V2. We can benchmark how historical transactions perform with this flag on/off.The flag is off by default:
With the flag on:
Great - we now can quantify the effect of the feature on runtime.
Other related changes
Levelto be able to use it from CLI. The behaviour should be the same.BlockAptosVM::execute_blockwithAptosVMBlockExecutor::new().execute_blockwhere possible (benchmark, debugger) so that we use the high-level wrapper, and not the inner type.How Has This Been Tested?
Manually running benchmarks.
Key Areas to Review
N/A, probably checking logger's
Levelis still correct.Type of Change
Which Components or Systems Does This Change Impact?
Checklist