Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Add retry on flaky tests #458

Merged
merged 3 commits into from
Sep 25, 2024

Conversation

ykim-akamai
Copy link
Contributor

@ykim-akamai ykim-akamai commented Sep 18, 2024

📝 Description

Following list of tests failed occasionally in recent builds, refer to TOD for details.

  • test_lke_cluster_labels_and_taints
  • test_get_sshkeys
  • test_ssh_key_create
  • test_image_create
  • test_image_replication
  • test_save_linode_noforce
  • test_save_linode_force
  • test_linode_resize

This PR uses existing pytest library called pytest-rerunfailures to rerun tests with delay. Flaky tests now can simply be marked with

@pytest.mark.flaky(reruns=3, reruns_delay=2)

✔️ How to Test

This retry logic can be tested with simply forcing one of the tests listed above to fail. (E.g. Insert assert False in top section of the test) then run that following test using command like below.

make TEST_CASE="test_lke_cluster_labels_and_taints -s" testint

Each retry should be marked with "R" in terminal

e.g.

python3 -m pytest test/integration/ -k test_lke_cluster_labels_and_taints 

platform darwin -- Python 3.10.14, pytest-8.1.1, pluggy-1.4.0
rootdir: /Users/ykim/linode/ykim/linode_api4-python
configfile: pyproject.toml
plugins: anyio-4.3.0, rerunfailures-14.0
collected 173 items / 172 deselected / 1 selected                                                                                                                                                                                                                                                                                                                                                                                    

test/integration/models/lke/test_lke.py 
R [100%]R [100%]R [100%]F [100%]

📷 Preview

If applicable, include a screenshot or code snippet of this change. Otherwise, please remove this section.

@ykim-akamai ykim-akamai requested a review from a team as a code owner September 18, 2024 23:07
@ykim-akamai ykim-akamai requested review from jriddle-linode and lgarber-akamai and removed request for a team September 18, 2024 23:07
@lgarber-akamai
Copy link
Contributor

@ykim-akamai ykim-akamai changed the title test: Add retries on flaky tests test: Add retry on flaky tests Sep 19, 2024
Copy link
Contributor

@lgarber-akamai lgarber-akamai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test failures in the linked suite run look unrelated and the relevant tests are passing on my end, nice work!

@ykim-akamai ykim-akamai added the testing for updates to the testing suite in the changelog. label Sep 23, 2024
@ykim-akamai ykim-akamai requested review from a team, zliang-akamai and yec-akamai and removed request for a team September 23, 2024 21:23
Copy link
Member

@zliang-akamai zliang-akamai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ykim-akamai ykim-akamai merged commit dddecc8 into linode:dev Sep 25, 2024
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
testing for updates to the testing suite in the changelog.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants