Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: lower time for fuzz/race tests #15141

Merged
merged 13 commits into from
Nov 8, 2024
Merged

Conversation

erikburt
Copy link
Collaborator

@erikburt erikburt commented Nov 6, 2024

Lowers the runtime of CI Core fuzz/race tests during executions triggered by PRs.

Changes

Race Tests:

  • Lower timeouts to 15s, and count to 5 (on PRs only, still at 10m/50 on scheduled runs)
  • Add build cache

Fuzz Tests:

  • Lower fuzz timeout to 4 minutes on PRs, 10m on scheduled runs
  • Calculating --seconds parameter passed to python fuzz script, based on FUZZ_TIMEOUT_MINUTES
  • Better calculation of time elapsed, and individual fuzzer runtimes in python script

Testing

https://github.com/smartcontractkit/chainlink/actions/runs/11734083428?pr=15141

  • Core Integ Tests: 2m25s
  • Deployment Tests: 2m32s
  • Race Tests: 4m39s
  • Core Tests: 5m34s
  • Fuzz Tests: 5m55s

In best case scenario, long pole is now "Clean Go Tidy & Generate"


RE-3205

@erikburt erikburt self-assigned this Nov 6, 2024
@erikburt erikburt requested review from a team as code owners November 6, 2024 21:05
@erikburt erikburt requested a review from george-dorin November 6, 2024 21:05
chainchad
chainchad previously approved these changes Nov 6, 2024
momentmaker
momentmaker previously approved these changes Nov 6, 2024
@momentmaker momentmaker enabled auto-merge November 6, 2024 21:12
@erikburt erikburt disabled auto-merge November 6, 2024 21:18
momentmaker
momentmaker previously approved these changes Nov 6, 2024
jmank88
jmank88 previously approved these changes Nov 6, 2024
@erikburt erikburt dismissed stale reviews from jmank88 and momentmaker via c040274 November 6, 2024 23:40
@smartcontractkit smartcontractkit deleted a comment from github-actions bot Nov 6, 2024
@smartcontractkit smartcontractkit deleted a comment from github-actions bot Nov 6, 2024
momentmaker
momentmaker previously approved these changes Nov 7, 2024
@erikburt erikburt force-pushed the chore/lower-race-fuzz-tests branch from 9934f37 to 9c9711c Compare November 7, 2024 18:59
@erikburt erikburt force-pushed the chore/lower-race-fuzz-tests branch from 9c9711c to aea4fa7 Compare November 7, 2024 19:21
@erikburt erikburt force-pushed the chore/lower-race-fuzz-tests branch from 451794e to e4c63f6 Compare November 7, 2024 20:53
@erikburt erikburt marked this pull request as ready for review November 7, 2024 21:07
jmank88
jmank88 previously approved these changes Nov 7, 2024
@jmank88
Copy link
Contributor

jmank88 commented Nov 7, 2024

Looks good, but could be a little faster? 🤷
image

@erikburt
Copy link
Collaborator Author

erikburt commented Nov 7, 2024

Looks good, but could be a little faster? 🤷

Ok I've lowered them even further. I was having trouble lowering race tests at all, seems no matter what it sits around 8 minutes runtime. I'm thinking it might be build time that is the majority of the runtime.

@jmank88
Copy link
Contributor

jmank88 commented Nov 7, 2024

Looks good, but could be a little faster? 🤷

Ok I've lowered them even further. I was having trouble lowering race tests at all, seems no matter what it sits around 8 minutes runtime. I'm thinking it might be build time that is the majority of the runtime.

You could try dropping the -json flag from the race tests. I don't know why that is enabled, but it creates extra noise by enabling -v. Might not matter though 🤷

@erikburt
Copy link
Collaborator Author

erikburt commented Nov 7, 2024

You could try dropping the -json flag from the race tests. I don't know why that is enabled, but it creates extra noise by enabling -v. Might not matter though 🤷

That seem to have helped by about 30 seconds. Down to 7.5min execution on the race test.

@erikburt
Copy link
Collaborator Author

erikburt commented Nov 8, 2024

Ok the build cache does help - down to 5m10s...

https://github.com/smartcontractkit/chainlink/actions/runs/11732567357?pr=15141

@erikburt erikburt requested a review from jmank88 November 8, 2024 00:59
@erikburt erikburt force-pushed the chore/lower-race-fuzz-tests branch from e33a7bb to cd071fb Compare November 8, 2024 01:01
@erikburt erikburt enabled auto-merge November 8, 2024 17:56
@erikburt erikburt added this pull request to the merge queue Nov 8, 2024
Merged via the queue into develop with commit 4bec5f4 Nov 8, 2024
90 checks passed
@erikburt erikburt deleted the chore/lower-race-fuzz-tests branch November 8, 2024 19:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants