Use test-level tags in CI #850
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Assigns a test level to each integration test and ensures that the integration tests fail if a tag is missing for a test. Also, enables the tags in the CI
eaudetcobello
approved these changes
Nov 28, 2024
HomayoonAlimohammadi
approved these changes
Nov 28, 2024
There was a problem hiding this comment.
Great work, thanks a lot @bschimke95! LGTM. Just minor comments and questions.
I'm not sure I fully understand which tag should be applied to which each test.
AFAIU, we mark the core/critical functionalities with PULL_REQUEST, less critical ones with NIGHTLY and even less critical ones with WEEKLY. Overall I think this is definitely the right direction and we can tune those tags as needed in the future.
louiseschmidtgen
approved these changes
Nov 28, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Adds a test level to each integration test and use them in the CI
Rationale
The integration tests for each PR take way too long and test a lot of things that are very unlikely to break. It is sufficient to run those in the nightly test runs or even weekly (e.g. for conformance tests).
Changes
Note that there is currently no weekly CI but I already added the tag for it. This is work for later.