Skip to content

Commit

Permalink
ci: add ability to run additional test tags against PRs (#5658)
Browse files Browse the repository at this point in the history
### Intent
This PR enhances our test-based workflows by making them more modular,
composable, and dispatchable (is that a word?). In addition, we can now
specify additional tags with an ampersand (`&critical`, etc) anywhere in
PR text to run additional tests against a pull request.

### Changes

Modular Workflows
* Split core tests (e2e-linux, e2e-windows, unit-tests,
integration-tests) into standalone workflows.
   * Parent workflows can now combine these tests as needed.
* Added parameters/inputs to e2e workflows for tagging and filtering e2e
tests.
* Added bash script to handle combining Playwright tags to run a subset
of tests:
   * Enables AND logic for tags where needed.
   * Handles “base tags” like `@win` or `@web`

Dynamic Tag Parsing
* Allows tags (`&critical`, etc) in PR descriptions to be parsed and
included dynamically.
* Ensures at minimum that critical tests always run (even if not tagged)
* Added a bash script to handle parsing PR text to pass along to e2e
test runner

### QA Notes
* Ran the full test suite and confirmed it passed and reports were
available as expected.
* Once merged, I plan to manually verify each of the workflow dispatches
works as expected, but I did try to test this as much as I could
already.

### Screenshots

test-full-suite.yml dispatch
<img width="360" alt="full-suite"
src="https://github.com/user-attachments/assets/7bbc55d2-c0cb-4976-b3f5-afb45bd65c7c">

test-e2e-linux.yml dispatch
<img width="354" alt="linux"
src="https://github.com/user-attachments/assets/dff56e8b-9d9b-4cbb-bdd0-85fc059f7199">

test-e2e-windows.yml dispatch
<img width="351" alt="windows"
src="https://github.com/user-attachments/assets/c760f80b-3e6a-4d86-b3c4-282b400cd5f1">

test-e2e-release.yml dispatch
<img width="359" alt="latest-release"
src="https://github.com/user-attachments/assets/1b2f65e1-5fc6-4e7f-9626-9adf1b998c5e">
  • Loading branch information
midleman authored Dec 11, 2024
1 parent 6c6f6c4 commit c06aa01
Show file tree
Hide file tree
Showing 21 changed files with 710 additions and 660 deletions.
100 changes: 0 additions & 100 deletions .github/actions/e2e-report/action.yml

This file was deleted.

27 changes: 27 additions & 0 deletions .github/actions/gen-report-dir/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
name: "Generate Report Directory & Send to GH Summary"
description: "Generates a unique REPORT_DIR and appends the report URL to the GitHub Step summary"
inputs:
bucket-url:
description: "Base URL of the S3 bucket or CDN for the report"
required: false
default: "https://d38p2avprg8il3.cloudfront.net"
runs:
using: "composite"
steps:
- name: Generate REPORT_DIR
shell: bash
run: |
# Generate a unique REPORT_DIR
RANDOM_SUFFIX=$RANDOM
REPORT_DIR="playwright-report-${{ github.run_id }}-${RANDOM_SUFFIX}"
# Export REPORT_DIR for downstream steps
echo "REPORT_DIR=$REPORT_DIR" >> $GITHUB_ENV
echo "Generated REPORT_DIR: $REPORT_DIR"
# Generate the REPORT_URL
REPORT_URL="${{ inputs.bucket-url }}/$REPORT_DIR/index.html"
echo "Report URL: $REPORT_URL"
# Append REPORT_URL to the GitHub Step summary
echo "📄 [Playwright Report]($REPORT_URL) <br>" >> $GITHUB_STEP_SUMMARY
5 changes: 4 additions & 1 deletion .github/actions/upload-report-to-s3/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@ inputs:
role-to-assume:
description: "The AWS role to assume"
required: true
report-dir:
description: "The path to the directory containing the Playwright report"
required: true

runs:
using: "composite"
Expand All @@ -19,4 +22,4 @@ runs:
if: ${{ !cancelled() }}
shell: bash
run: |
aws s3 cp playwright-report/. s3://positron-test-reports/playwright-report-${{ github.run_id }} --recursive
aws s3 cp playwright-report/. s3://positron-test-reports/${{ inputs.report-dir }} --recursive
10 changes: 7 additions & 3 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,16 @@ main branch by either pulling or rebasing.
<!--
Describe briefly what problem this pull request resolves, or what
new feature it introduces. Include screenshots of any new or altered
UI. Link to any GitHub issues but avoid "magic" keywords that will
automatically close the issue. If there are any details about your
approach that are unintuitive or you want to draw attention to, please
UI. Link to any GitHub issues but avoid "magic" keywords that will
automatically close the issue. If there are any details about your
approach that are unintuitive or you want to draw attention to, please
describe them here.
-->

### Tests to Run
<!-- See `test/e2e/helpers/test-tags.ts` for a complete list of available tags. -->
`@:critical`

### QA Notes

<!--
Expand Down
Loading

0 comments on commit c06aa01

Please sign in to comment.