Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docker] use cloudfront debian mirrors for bullseye #15303

Merged
merged 5 commits into from
Nov 18, 2024

Conversation

rustielin
Copy link
Contributor

@rustielin rustielin commented Nov 18, 2024

Description

commandeer #15230 for debian bullseye

  • Use the mirror by editing /etc/apt/sources.list directly via sed during the debian-base and rust-base images build
  • Move shared utils into the debian-base image to prevent re-install in each leaf build
  • alpha-sort the installed packages

How Has This Been Tested?

Key Areas to Review

Type of Change

  • New feature
  • Bug fix
  • Breaking change
  • Performance improvement
  • Refactoring
  • Dependency update
  • Documentation update
  • Tests

Which Components or Systems Does This Change Impact?

  • Validator Node
  • Full Node (API, Indexer, etc.)
  • Move/Aptos Virtual Machine
  • Aptos Framework
  • Aptos CLI/SDK
  • Developer Infrastructure
  • Move Compiler
  • Other (specify)

Checklist

  • I have read and followed the CONTRIBUTING doc
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I identified and added all stakeholders and component owners affected by this change as reviewers
  • I tested both happy and unhappy path of the functionality
  • I have made corresponding changes to the documentation

This should improve build time and reliability

Test Plan: build images
@rustielin rustielin added the CICD:run-e2e-tests when this label is present github actions will run all land-blocking e2e tests from the PR label Nov 18, 2024
@rustielin rustielin requested a review from a team as a code owner November 18, 2024 19:44
Copy link

trunk-io bot commented Nov 18, 2024

⏱️ 5h 40m total CI duration on this PR
Slowest 15 Jobs Cumulative Duration Recent Runs
execution-performance / test-target-determinator 1h 6m 🟩🟩🟩🟩🟩 (+9 more)
test-target-determinator 39m 🟩🟩🟩🟩🟩 (+6 more)
check 33m 🟩🟩🟩🟩 (+8 more)
check-dynamic-deps 21m 🟩🟩🟩🟩🟩 (+9 more)
fetch-last-released-docker-image-tag 19m 🟩🟩🟩🟩🟩 (+6 more)
rust-cargo-deny 17m 🟩🟩🟩🟩🟩 (+6 more)
forge-compat-test / forge 14m 🟩
rust-move-tests 12m 🟩
rust-move-tests 12m 🟩
rust-move-tests 7m 🟥
semgrep/ci 7m 🟩🟩🟩🟩🟩 (+9 more)
rust-move-tests 7m 🟥
rust-move-tests 7m 🟥
rust-move-tests 7m 🟥
rust-doc-tests 5m 🟩

🚨 1 job on the last run was significantly faster/slower than expected

Job Duration vs 7d avg Delta
execution-performance / single-node-performance 10s 18m -99%

settingsfeedbackdocs ⋅ learn more about trunk.io

@rustielin rustielin changed the title Rustielin/cloudfront debian mirror [docker] use cloudfront debian mirrors for bookworm Nov 18, 2024
Comment on lines 4 to 6
# TODO upgrade to bookworm
RUN echo "deb https://cloudfront.debian.net/debian/ bullseye main" > /etc/apt/sources.list.d/bullseye.list && \
echo "Package: *\nPin: release n=bullseye\nPin-Priority: 50" > /etc/apt/preferences.d/bullseye
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR title and TODO suggest upgrading to bookworm, but the code still configures bullseye as the debian repository and pin. To complete the upgrade, replace both instances of bullseye with bookworm in the repository URL and pin configuration.

Spotted by Graphite Reviewer

Is this helpful? React 👍 or 👎 to let us know.

@rustielin rustielin force-pushed the rustielin/cloudfront-debian-mirror branch from 290121d to 21c4558 Compare November 18, 2024 19:45
@rustielin rustielin changed the title [docker] use cloudfront debian mirrors for bookworm [docker] use cloudfront debian mirrors for bullseye Nov 18, 2024
@rustielin rustielin force-pushed the rustielin/cloudfront-debian-mirror branch 2 times, most recently from 78ae46a to dcfbd5a Compare November 18, 2024 20:13
@rustielin rustielin force-pushed the rustielin/cloudfront-debian-mirror branch 2 times, most recently from 7bb19c4 to a48f83f Compare November 18, 2024 20:16
@rustielin rustielin force-pushed the rustielin/cloudfront-debian-mirror branch from a48f83f to 1b640ae Compare November 18, 2024 20:21
@rustielin rustielin requested a review from a team November 18, 2024 20:43
Copy link
Contributor

@ibalajiarun ibalajiarun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's land it to unblock. I will work on it later to pull a base image instead of building it during cache invalidations.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

@rustielin rustielin enabled auto-merge (squash) November 18, 2024 21:20

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

@rustielin rustielin merged commit 7cf7637 into main Nov 18, 2024
61 of 93 checks passed
@rustielin rustielin deleted the rustielin/cloudfront-debian-mirror branch November 18, 2024 22:20
github-actions bot pushed a commit that referenced this pull request Nov 20, 2024
* [builder] Point docker builds to aws hosted cloudfront mirror

This should improve build time and reliability

Test Plan: build images

* [docker] use cloudfront debian mirrors for bullseye

* [docker] debian-base install

* [docker] cleanup stuff into debian-base

---------

Co-authored-by: Perry Randall <[email protected]>
(cherry picked from commit 7cf7637)
github-actions bot pushed a commit that referenced this pull request Nov 20, 2024
* [builder] Point docker builds to aws hosted cloudfront mirror

This should improve build time and reliability

Test Plan: build images

* [docker] use cloudfront debian mirrors for bullseye

* [docker] debian-base install

* [docker] cleanup stuff into debian-base

---------

Co-authored-by: Perry Randall <[email protected]>
(cherry picked from commit 7cf7637)
github-actions bot pushed a commit that referenced this pull request Nov 20, 2024
* [builder] Point docker builds to aws hosted cloudfront mirror

This should improve build time and reliability

Test Plan: build images

* [docker] use cloudfront debian mirrors for bullseye

* [docker] debian-base install

* [docker] cleanup stuff into debian-base

---------

Co-authored-by: Perry Randall <[email protected]>
(cherry picked from commit 7cf7637)
Copy link
Contributor

💔 Some backports could not be created

Status Branch Result
aptos-release-v1.23
aptos-release-v1.24 An unhandled error occurred. Please see the logs for details

Manual backport

To create the backport manually run:

backport --pr 15303

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details

Copy link
Contributor

💚 All backports created successfully

Status Branch Result
aptos-release-v1.23
aptos-release-v1.24

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details

This comment has been minimized.

This comment has been minimized.

Copy link
Contributor

✅ Forge suite realistic_env_max_load success on 990c804b1fdf78d92e156900f3a7d9f427f04a4c

two traffics test: inner traffic : committed: 14316.00 txn/s, latency: 2779.37 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 3300 ms), latency samples: 5443340
two traffics test : committed: 99.94 txn/s, latency: 1752.62 ms, (p50: 1400 ms, p70: 1400, p90: 1600 ms, p99: 10700 ms), latency samples: 1800
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.046, avg: 1.591", "ConsensusProposalToOrdered: max: 0.319, avg: 0.295", "ConsensusOrderedToCommit: max: 0.368, avg: 0.359", "ConsensusProposalToCommit: max: 0.663, avg: 0.654"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.00s no progress at version 1606936 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.68s no progress at version 5923083 (avg 8.52s) [limit 15].
Test Ok

Copy link
Contributor

✅ Forge suite compat success on be3ccc58ef518259d63fab7f3e3613f2a6214268 ==> 990c804b1fdf78d92e156900f3a7d9f427f04a4c

Compatibility test results for be3ccc58ef518259d63fab7f3e3613f2a6214268 ==> 990c804b1fdf78d92e156900f3a7d9f427f04a4c (PR)
1. Check liveness of validators at old version: be3ccc58ef518259d63fab7f3e3613f2a6214268
compatibility::simple-validator-upgrade::liveness-check : committed: 16511.21 txn/s, latency: 2048.67 ms, (p50: 2000 ms, p70: 2100, p90: 2500 ms, p99: 3300 ms), latency samples: 538140
2. Upgrading first Validator to new version: 990c804b1fdf78d92e156900f3a7d9f427f04a4c
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7376.20 txn/s, latency: 3764.11 ms, (p50: 4000 ms, p70: 4500, p90: 4800 ms, p99: 5200 ms), latency samples: 136260
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7281.89 txn/s, latency: 4390.98 ms, (p50: 4500 ms, p70: 4600, p90: 6300 ms, p99: 6600 ms), latency samples: 246500
3. Upgrading rest of first batch to new version: 990c804b1fdf78d92e156900f3a7d9f427f04a4c
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7469.89 txn/s, latency: 3848.61 ms, (p50: 4300 ms, p70: 4500, p90: 4600 ms, p99: 4700 ms), latency samples: 140380
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 6634.19 txn/s, latency: 4571.98 ms, (p50: 4700 ms, p70: 4800, p90: 4900 ms, p99: 6700 ms), latency samples: 253200
4. upgrading second batch to new version: 990c804b1fdf78d92e156900f3a7d9f427f04a4c
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 12602.59 txn/s, latency: 2136.92 ms, (p50: 2100 ms, p70: 2400, p90: 2800 ms, p99: 2900 ms), latency samples: 220480
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10608.82 txn/s, latency: 2524.59 ms, (p50: 2200 ms, p70: 2800, p90: 2900 ms, p99: 5800 ms), latency samples: 413820
5. check swarm health
Compatibility test for be3ccc58ef518259d63fab7f3e3613f2a6214268 ==> 990c804b1fdf78d92e156900f3a7d9f427f04a4c passed
Test Ok

msmouse pushed a commit that referenced this pull request Nov 20, 2024
* [builder] Point docker builds to aws hosted cloudfront mirror

This should improve build time and reliability

Test Plan: build images

* [docker] use cloudfront debian mirrors for bullseye

* [docker] debian-base install

* [docker] cleanup stuff into debian-base

---------

Co-authored-by: Perry Randall <[email protected]>
(cherry picked from commit 7cf7637)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CICD:run-e2e-tests when this label is present github actions will run all land-blocking e2e tests from the PR v1.23 v1.24
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants