Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve Flakeguard: Increased Test Runs, Improved Summaries, Fixes for Notifications, Parsing, and Logs #15541

Merged
merged 41 commits into from
Dec 10, 2024
Merged

Improve Flakeguard: Increased Test Runs, Improved Summaries, Fixes for Notifications, Parsing, and Logs #15541

merged 41 commits into from
Dec 10, 2024
+122 −125

Conversation

lukaszcl
Copy link
Collaborator

@lukaszcl lukaszcl commented Dec 6, 2024 •
edited
Loading

This PR improves Flakeguard by addressing multiple enhancements and fixes:

  1. Increase Test Run Count

  2. Enhanced GitHub Summary and PR Comments

    • Improved Flakeguard's GitHub summary and PR comments for better clarity and usability.
    • JIRA Ticket: TT-1879

  3. Fixed Slack Notifications

    • Resolved issues causing incorrect or missing Slack notifications for Flakeguard.
    • JIRA Ticket: TT-1886

  4. Clickable Links to JSON Logs

    • Added functionality to allow clicking on failed tests in PR or nightly runs to directly access the corresponding JSON logs.
    • JIRA Ticket: TT-1885

  5. Error Parsing Test Results

Workflow Run: GitHub Actions Run

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 6, 2024 •
edited
Loading

AER Report: CI Core ran successfully ✅

aer_workflow , commit

AER Report: Operator UI CI ran successfully ✅

aer_workflow , commit

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 6, 2024

Flakeguard Summary

Setting Value
Project github.com/smartcontractkit/chainlink/v2
Max Pass Ratio 100.00%
Test Run Count 3
Race Detection false
Excluded Tests TestChainComponents

Found Flaky Tests ❌

Category Total
Tests 329
Panicked Tests 0
Raced Tests 0
Flaky Tests 2
Flaky Test Ratio 0.61%
Runs 987
Passes 981
Failures 6
Skips 0
Pass Ratio 99.39%
Name Pass Ratio Panicked? Timed Out? Race? Runs Successes Failures Skips Package Package Panicked? Avg Duration Code Owners
TestResolver_CronSpec 0.00% false false false 3 0 3 0 github.com/smartcontractkit/chainlink/v2/core/web/resolver false 0s @smartcontractkit/deployment-automation, @smartcontractkit/foundations, @smartcontractkit/core
TestResolver_CronSpec/cron_spec_success 0.00% false false false 3 0 3 0 github.com/smartcontractkit/chainlink/v2/core/web/resolver false 40ms @smartcontractkit/deployment-automation, @smartcontractkit/foundations, @smartcontractkit/core

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 6, 2024

Flakeguard Summary

Ran new or updated tests between develop and ae51d8c (refactor-flakeguard-report).

View Flaky Detector Details | Compare Changes

Found Flaky Tests ❌

Name Pass Ratio Panicked? Timed Out? Race? Runs Successes Failures Skips Package Package Panicked? Avg Duration Code Owners
TestImmediateFail 0.00% false false false 3 0 3 0 github.com/smartcontractkit/chainlink/v2/core/web/resolver/failtest false 0s @smartcontractkit/deployment-automation, @smartcontractkit/foundations, @smartcontractkit/core

@lukaszcl lukaszcl force-pushed the refactor-flakeguard-report branch from 54cb29a to 4619445 Compare December 6, 2024 13:15
@lukaszcl lukaszcl mentioned this pull request Dec 6, 2024
Merged
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 6, 2024

Flakeguard Summary

Ran new or updated tests between develop and 4619445 (refactor-flakeguard-report).

View Flaky Detector Details | Compare Changes

Found Flaky Tests ❌

Name Pass Ratio Panicked? Timed Out? Race? Runs Successes Failures Skips Package Package Panicked? Avg Duration Code Owners
TestImmediateFail 0.00% false false false 3 0 3 0 github.com/smartcontractkit/chainlink/v2/core/web/resolver/failtest false 0s @smartcontractkit/deployment-automation, @smartcontractkit/foundations, @smartcontractkit/core

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 9, 2024

Flakeguard Summary

Ran new or updated tests between develop and 7213d1a (refactor-flakeguard-report).

View Flaky Detector Details | Compare Changes

Found Flaky Tests ❌

Name Pass Ratio Panicked? Timed Out? Race? Runs Successes Failures Skips Package Package Panicked? Avg Duration Code Owners
TestImmediateFail 0.00% false false false 3 0 3 0 github.com/smartcontractkit/chainlink/v2/core/web/resolver/failtest false 0s @smartcontractkit/deployment-automation, @smartcontractkit/foundations, @smartcontractkit/core

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 9, 2024

Flakeguard Summary

Ran new or updated tests between develop and e387d3d (refactor-flakeguard-report).

View Flaky Detector Details | Compare Changes

Found Flaky Tests ❌

Name Pass Ratio Panicked? Timed Out? Race? Runs Successes Failures Skips Package Package Panicked? Avg Duration Code Owners
TestImmediateFail 0.00% false false false 3 0 3 0 github.com/smartcontractkit/chainlink/v2/core/web/resolver/failtest false 0s @smartcontractkit/deployment-automation, @smartcontractkit/foundations, @smartcontractkit/core

@lukaszcl lukaszcl marked this pull request as ready for review December 9, 2024 14:19
@lukaszcl lukaszcl requested review from a team as code owners December 9, 2024 14:19
@lukaszcl lukaszcl changed the title Refactor flaky test detection reports Refactor flaky test detection (Flakeguard) reports Dec 9, 2024
@lukaszcl lukaszcl changed the title Refactor flaky test detection (Flakeguard) reports Improve Flakeguard: Increased Test Runs, Improved Summaries, Fixes for Notifications, Parsing, and Logs Dec 9, 2024
@lukaszcl lukaszcl requested a review from kalverra December 10, 2024 13:44
kalverra
kalverra previously approved these changes Dec 10, 2024
View reviewed changes
@lukaszcl lukaszcl added this pull request to the merge queue Dec 10, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Dec 10, 2024
AKhozya
AKhozya requested changes Dec 10, 2024
View reviewed changes
.github/workflows/flakeguard.yml Outdated Show resolved Hide resolved
@lukaszcl lukaszcl dismissed stale reviews from kalverra and gheorghestrimtu via 37cb6f3 December 10, 2024 14:52
@lukaszcl lukaszcl added this pull request to the merge queue Dec 10, 2024
Merged via the queue into develop with commit 0e0d19e Dec 10, 2024
103 checks passed
@lukaszcl lukaszcl deleted the refactor-flakeguard-report branch December 10, 2024 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kalverra kalverra kalverra approved these changes

@AKhozya AKhozya AKhozya approved these changes

@gheorghestrimtu gheorghestrimtu gheorghestrimtu left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@lukaszcl @AKhozya @kalverra @gheorghestrimtu