[cp][aptos-release-v1.24] [loader-v2] Fixing global cache reads & read-before-write on publish

[github-actions]

Backport

This will backport the following commits from main to aptos-release-v1.24:

• [loader-v2] Fixing global cache reads & read-before-write on publish (#15285)

Questions ?

Please refer to the Backport tool documentation

[trunk-io]

⏱️ 1h 52m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
rust-images / rust-all	32m	🟥 🟥 🟥 🟩
execution-performance / single-node-performance	25m	🟩
rust-unit-tests	22m	🟩
rust-move-tests	9m	🟩
rust-doc-tests	5m	🟩
execution-performance / test-target-determinator	5m	🟩
test-target-determinator	5m	🟩
check	4m	🟩
rust-cargo-deny	2m	🟩
check-dynamic-deps	2m	🟩
fetch-last-released-docker-image-tag	2m	🟩
general-lints	27s	🟩
semgrep/ci	24s	🟩
file_change_determinator	21s	🟩
file_change_determinator	12s	🟩

🚨 1 job on the last run was significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
execution-performance / single-node-performance	25m	17m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[github-actions]

✅ Forge suite realistic_env_max_load success on d3181eb56cd53399f2488eb0378fb728bb455b27

two traffics test: inner traffic : committed: 14453.93 txn/s, latency: 2749.23 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 3300 ms), latency samples: 5495700
two traffics test : committed: 100.04 txn/s, latency: 1497.74 ms, (p50: 1400 ms, p70: 1400, p90: 1600 ms, p99: 9100 ms), latency samples: 1740
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.017, avg: 1.550", "ConsensusProposalToOrdered: max: 0.315, avg: 0.291", "ConsensusOrderedToCommit: max: 0.377, avg: 0.363", "ConsensusProposalToCommit: max: 0.668, avg: 0.655"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.88s no progress at version 2859754 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.76s no progress at version 2859752 (avg 8.76s) [limit 15].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite framework_upgrade success on a0ec6ba11bfe4cfc5b586edc9e227aba4909e8fe ==> d3181eb56cd53399f2488eb0378fb728bb455b27

Compatibility test results for a0ec6ba11bfe4cfc5b586edc9e227aba4909e8fe ==> d3181eb56cd53399f2488eb0378fb728bb455b27 (PR)
Upgrade the nodes to version: d3181eb56cd53399f2488eb0378fb728bb455b27
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1473.43 txn/s, submitted: 1476.74 txn/s, failed submission: 3.32 txn/s, expired: 3.32 txn/s, latency: 2086.20 ms, (p50: 2100 ms, p70: 2400, p90: 2700 ms, p99: 4200 ms), latency samples: 133320
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1175.42 txn/s, submitted: 1177.21 txn/s, failed submission: 1.78 txn/s, expired: 1.78 txn/s, latency: 3287.74 ms, (p50: 2100 ms, p70: 2400, p90: 13200 ms, p99: 15400 ms), latency samples: 92260
5. check swarm health
Compatibility test for a0ec6ba11bfe4cfc5b586edc9e227aba4909e8fe ==> d3181eb56cd53399f2488eb0378fb728bb455b27 passed
Upgrade the remaining nodes to version: d3181eb56cd53399f2488eb0378fb728bb455b27
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1423.10 txn/s, submitted: 1425.34 txn/s, failed submission: 2.24 txn/s, expired: 2.24 txn/s, latency: 2145.84 ms, (p50: 2100 ms, p70: 2400, p90: 3300 ms, p99: 5200 ms), latency samples: 126980
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite compat success on a0ec6ba11bfe4cfc5b586edc9e227aba4909e8fe ==> d3181eb56cd53399f2488eb0378fb728bb455b27

Compatibility test results for a0ec6ba11bfe4cfc5b586edc9e227aba4909e8fe ==> d3181eb56cd53399f2488eb0378fb728bb455b27 (PR)
1. Check liveness of validators at old version: a0ec6ba11bfe4cfc5b586edc9e227aba4909e8fe
compatibility::simple-validator-upgrade::liveness-check : committed: 15788.51 txn/s, latency: 1861.19 ms, (p50: 1800 ms, p70: 2000, p90: 2100 ms, p99: 2400 ms), latency samples: 588400
2. Upgrading first Validator to new version: d3181eb56cd53399f2488eb0378fb728bb455b27
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7417.24 txn/s, latency: 3704.87 ms, (p50: 4000 ms, p70: 4200, p90: 4800 ms, p99: 5100 ms), latency samples: 139440
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7899.11 txn/s, latency: 4042.39 ms, (p50: 4100 ms, p70: 4200, p90: 6400 ms, p99: 6600 ms), latency samples: 260540
3. Upgrading rest of first batch to new version: d3181eb56cd53399f2488eb0378fb728bb455b27
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 5985.79 txn/s, latency: 4593.15 ms, (p50: 4600 ms, p70: 5300, p90: 5600 ms, p99: 6200 ms), latency samples: 109660
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7666.02 txn/s, latency: 4177.99 ms, (p50: 4300 ms, p70: 4500, p90: 6300 ms, p99: 6500 ms), latency samples: 254000
4. upgrading second batch to new version: d3181eb56cd53399f2488eb0378fb728bb455b27
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 12681.96 txn/s, latency: 2208.79 ms, (p50: 2400 ms, p70: 2500, p90: 2600 ms, p99: 2700 ms), latency samples: 219720
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10935.46 txn/s, latency: 2884.74 ms, (p50: 2600 ms, p70: 2600, p90: 5700 ms, p99: 7600 ms), latency samples: 375820
5. check swarm health
Compatibility test for a0ec6ba11bfe4cfc5b586edc9e227aba4909e8fe ==> d3181eb56cd53399f2488eb0378fb728bb455b27 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking