build: release v6.81.0 #6777

LinHuiqing · 2023-10-05T06:45:51Z

New

feat(virus-scanner): frontend orchestration #6767
feat: add well-known domain #6774
build: merge release v6.80.0 into develop #6775
build: release v6.80.0 #6773

Dependencies

fix(deps): bump zod from 3.22.2 to 3.22.3 in /shared #6768
fix(deps): bump zod from 3.21.4 to 3.22.3 #6769

Dev-Dependencies

Tests

feat(virus-scanner): frontend orchestration `#6767`

@KenLSM

Preperation

Go to https://www.eicar.org/download-anti-malware-testfile/ and download the test malicious file eicar.com.txt.
Find 2 non-malicious files.
In the DB featureflags collection, set the encryption-boundary-shift-virus-scanner flag as { enabled: true }.
In growthbook's feature page for "encryption-boundary-shift-virus-scanner", turn on the feature flag for the respective env and make sure the value returned will be ON.

Test that virus scanner works

Test that BE feature flag works

@tshuli

In the DB featureflags collection, set the encryption-boundary-shift-virus-scanner flag as { enabled: false }.
- This means that the virus scanner should be enabled on the FE but not BE.
Go to the respondent page. Upload any file and submit the form.
- The submission should return a network call with an error saying that the feature is disabled.
- The submission should still succeed upon auto fetch fallback
In the DB featureflags collection, restore the encryption-boundary-shift-virus-scanner flag as { enabled: true }.

Test that the FE feature flag works

@tshuli

In growthbook's feature page for "encryption-boundary-shift-virus-scanner", turn off the feature flag for the respective env.
Go to the respondent page. Open the network panel. Upload the test malicious file and submit the form.
- Based on the network panel, the request sent should have version 2.
- The submission should succeed.
- Check that the submission went through to the admin portal. Download and check that the file name and content for both files is correct.
In growthbook's feature page for "encryption-boundary-shift-virus-scanner", turn on the feature flag for the respective env.

Test that FE falls back to submission without virus scan if virus scanning fails

@tshuli

Change the function name in the env var (VIRUS_SCANNER_LAMBDA_FUNCTION_NAME) to an invalid function name and redeploy the app.
Go to the respondent page. Open the network panel. Upload the test malicious file and submit the form.
- Should receive a 500 network response that virus scan failed
- Should automatically resubmit and retry should succeed, with version 2 in the request
- Check that the submission went through to the admin portal. Download and check that the file name and content for both files is correct.
Restore the function name to the appropriate function name:
- dev: function
- prod: virus-scanner-production-virus-scanner
- staging: virus-scanner-staging-virus-scanner
- uat: virus-scanner-uat-virus-scanner
(optional, if we want to test the virus scanner more) Redeploy the app.

Test that `/submission/storage` will be used if growthbook can't be instantiated

Change the growthbook client key in the env var (GROWTHBOOK_CLIENT_KEY) to an invalid one.
Go to the respondent page of any storage form. Open the network panel.
Requests to retrieve growthbook feature definitions should be observed to be failing. They're the ones with sdk-...
Submit the form. The submission should still go to the /submissions/storage endpoint rather than /submissions/encrypt.
Restore the growthbook client key in the env var (GROWTHBOOK_CLIENT_KEY) to the appropriate one.

Regression tests

Submit a storage mode form with logic.
Submit a email mode form with logic.

Deploy Notes

In the DB featureflags collection, set the encryption-boundary-shift-virus-scanner flag as { enabled: true }.

New GrowthBook flags:

Make sure that the encryption-boundary-shift-virus-scanner flag is disabled for prod before deployment. Rollout to 1% after deployment.
encryption-boundary-shift-virus-scanner: FE flag for virus scanner feature.

New AWS configs:

Make sure that the prod quarantine S3 bucket has the right CORS policy as described below.

Quarantine bucket CORS policy:

prod.virus.scanner.quarantine

[
    {
        "AllowedHeaders": [],
        "AllowedMethods": [
            "POST"
        ],
        "AllowedOrigins": [
            "https://form.gov.sg"
        ],
        "ExposeHeaders": []
    }
]

staging.virus.scanner.quarantine

[
    {
        "AllowedHeaders": [],
        "AllowedMethods": [
            "POST"
        ],
        "AllowedOrigins": [
            "https://staging.form.gov.sg",
            "https://staging-alt.form.gov.sg",
            "https://staging-alt2.form.gov.sg"
        ],
        "ExposeHeaders": []
    }
]

uat.virus.scanner.quarantine

[
    {
        "AllowedHeaders": [],
        "AllowedMethods": [
            "POST"
        ],
        "AllowedOrigins": [
            "https://uat.form.gov.sg"
        ],
        "ExposeHeaders": []
    }
]

feat: add well-known domain `#6774`

@LinHuiqing

Regression

Existing well-known routes are not affected

Ensure that /singpass/.well-known/jwks.json does not return 4xx

Existing static and public routes are not affected

Ensure that form.gov.sg is loaded with images and js running

Deploy Notes

New aasa file in .well-known:

Upload apple-developer-merchantid-domain-association file into s3 static bucket
Make sure that the /.well-known/apple-developer-merchantid-domain-association does not return 404

Bumps [zod](https://github.com/colinhacks/zod) from 3.21.4 to 3.22.3. - [Release notes](https://github.com/colinhacks/zod/releases) - [Changelog](https://github.com/colinhacks/zod/blob/master/CHANGELOG.md) - [Commits](colinhacks/zod@v3.21.4...v3.22.3) --- updated-dependencies: - dependency-name: zod dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [zod](https://github.com/colinhacks/zod) from 3.22.2 to 3.22.3. - [Release notes](https://github.com/colinhacks/zod/releases) - [Changelog](https://github.com/colinhacks/zod/blob/master/CHANGELOG.md) - [Commits](colinhacks/zod@v3.22.2...v3.22.3) --- updated-dependencies: - dependency-name: zod dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

build: merge release v6.80.0 into develop

* feat: add .well-known fetch * chore: detect specific directory * refactor: use combined regex

* feat: move submission versions to shared * refactor: make version for clear mutates optional * feat: FE get presigned post data * feat: share AttachmentPresignedPostDataMapType * feat: fe upload files to quarantine bucket * feat: throw error if attachment upload failed * feat: storage submission with virus scans * fix: virus scan conditional * refactor: check feature value right before using * fix: csp headers for virus scanner * fix: csp headers again (typo) * fix: local virus scans if >1 attachment submitted * style: improve virus scan failed error msg * docs: add comments to PublicFormProvider * fix: rm version in SubmitStorageFormClearArgs * docs: createClearSubmissionWithVirusScanningFormData * docs: submitStorageModeClearFormWithVirusScanning * fix: repeated attachment responses on local dev * fix: use virusScannerQuarantineS3BucketUrl defined - chore: rm unused URL for clean bucket - fix: broken tests due to CSP change * fix: broken tests due to schema without clean url * feat: default true for encryption boundary shift FE flag * chore: rm dangling comment from testing * style: virus scan error msg * fix: crash for submissions without attachments * style: syncVirusScanning -> devModeSyncVirusScanning * perf: on local dev - exit for...of once err * fix: status code for malicious file should be 400 * feat: filename in virus scan failed error * fix: encrypt-submission.service tests * refactor: break in attachment res in for...of * chore: rm TODO * fix: playwright tests blocked by feature flag * fix: get-s3-presigned-post-data crash when [] * fix: use fetch fallback only mutateasync fails * docs: explain what onError for mutateAsync does

tshuli · 2023-10-05T08:31:13Z

overriding BE tests to merge as they passed in the PR before release

LinHuiqing · 2023-10-05T08:37:53Z

fyi @tshuli overwrote the tests w observed behaviour!

dependabot bot and others added 6 commits October 4, 2023 07:46

Merge pull request #6775 from opengovsg/release-al2

a039a6d

build: merge release v6.80.0 into develop

feat: add well-known domain (#6774)

2f8caab

* feat: add .well-known fetch * chore: detect specific directory * refactor: use combined regex

chore: bump version to v6.81.0

f059d44

LinHuiqing temporarily deployed to staging October 5, 2023 06:45 — with GitHub Actions Inactive

LinHuiqing mentioned this pull request Oct 5, 2023

fix: rm fallback from submit v2.1 mutation #6778

Closed

3 tasks

tshuli approved these changes Oct 5, 2023

View reviewed changes

tshuli merged commit 380593c into release-al2 Oct 5, 2023
83 of 86 checks passed

tshuli deleted the release_v6.81.0 branch October 5, 2023 08:31

tshuli mentioned this pull request Oct 16, 2023

build: release v6.82.0 #6800

Merged

48 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: release v6.81.0 #6777

build: release v6.81.0 #6777

LinHuiqing commented Oct 5, 2023 •

edited

Loading

tshuli commented Oct 5, 2023

LinHuiqing commented Oct 5, 2023

build: release v6.81.0 #6777

build: release v6.81.0 #6777

Conversation

LinHuiqing commented Oct 5, 2023 • edited Loading

New

Dependencies

Dev-Dependencies

Tests

feat(virus-scanner): frontend orchestration #6767

Preperation

Test that virus scanner works

Test that BE feature flag works

Test that the FE feature flag works

Test that FE falls back to submission without virus scan if virus scanning fails

Test that /submission/storage will be used if growthbook can't be instantiated

Regression tests

Deploy Notes

feat: add well-known domain #6774

Existing well-known routes are not affected

Existing static and public routes are not affected

Deploy Notes

tshuli commented Oct 5, 2023

LinHuiqing commented Oct 5, 2023

LinHuiqing commented Oct 5, 2023 •

edited

Loading

feat(virus-scanner): frontend orchestration `#6767`

Test that `/submission/storage` will be used if growthbook can't be instantiated

feat: add well-known domain `#6774`