build: release v6.79.0

[KenLSM]

New

• feat: rename NRIC field to NRIC/FIN #6759
• chore: fix snyk vulnerabilities #6728
• chore: update hydrogen alpine #6752
• feat(payment): invoice through s3 #6733
• fix: reinstate MyInfo verified field indicator for MyInfo child DOB #6751
• feat(virus-scanner): download clean file #6747
• fix: rm growthbook legacy streaming prop #6744
• chore: use correct branch for scanner cron job #6746
• revert: "chore: update hydrogen-alpine version (chore: update hydrogen-alpine version #6720)" #6748
• feat(virus-scanner): invoke lambda to scan file #6734
• fix: redirect to frontend app after myinfo log in (for local dev environment) #6743
• chore: update hydrogen-alpine version #6720
• fix: display error message when myinfo child fields are not filled in #6735
• build: merge v6.78.2 back into develop #6742
• fix: hotfix for 500 errors thrown on email bounce notification endpoint #6741
• build: merge v6.78.1 into develop #6738

Dependencies

• fix(deps): bump libphonenumber-js from 1.10.43 to 1.10.45 in /shared #6753

Dev-Dependencies

Tests

feat: rename NRIC field to NRIC/FIN #6759

@KenLSM

• Create a form with NRIC field before this change was pushed. Use the default field question of "NRIC".
• Deploy this change. Form submission successful on the existing NRIC field. In the submission, the field question is still "NRIC" instead of "NRIC/FIN".
• Now create a new NRIC/FIN field. Form submission successful on the new NRIC/FIN field, and in the submission, the field question is "NRIC/FIN"

Before & After Screenshots

BEFORE:

AFTER:

chore: fix snyk vulnerabilities #6728

@KenLSM

• Should be able to see forms and storage mode responses
• Should be able to receive OTP verification in local dev environment
• Should be able to receive OTP verification in staging environment
• Should be able to submit Singpass Myinfo forms correctly

chore: update hydrogen alpine #6752

@KenLSM

• Email and storage mode submissions should work
• Email responses should be receivable
• Storage mode responses should be retrievable
• PDF response for email mode should be generated correctly
• Payment invoice should be generated correctly

feat(payment): invoice through s3 #6733

@LinHuiqing

Regression
Downloading proof of payments should respond with pdf file

• Complete a payment on a payment form
  • Ensure that downloading of invoice will save a PDF file
    • Ensure that Proof of payment should correctly reflect amount, quantity, and value of items purchased
  • Ensure that subsequent download of invoice will still save a PDF file
• Downloading proof of payments should respond within acceptable time limit (<10 seconds)

Disconnected Payment Forms should still allow user to download proof of payment
(record the download link from the test above)

• Disconnect payment in the Payment Settings Page
• Ensure that downloading of invoice will save a PDF file

New Feature

• Ensure that subsequent download of invoice will still save a PDF file
• hasReceiptStoredInS3 should be set in the DB for the particular payment

Deploy Notes

1. Create s3 buckets for payment proof on staging, staging-alt, prod, uat
2. Add config PAYMENT_PROOF_S3_BUCKET to be added on staging, staging-alt, prod, uat
3. Update acl to include GET, PUT access on staging, prod, uat roles (see comment on linear issue)

New environment variables:

• env var : PAYMENT_PROOF_S3_BUCKET to configure the name of payment proof s3 bucket

This PR is risky, as there were some issues with nginx when it was deployed to staging but unreproducible on staging-alt, staging-alt2, and uat. To be safe, we need to deploy during the low traffic period.

To verify that the issue doesn't occur, we will need to trigger the invoice download multiple times, and ensure that we don't see a 5xx from the ec2. Note that our express server isn't the one returning 5xx, but nginx's reverse proxy itself.

fix: reinstate MyInfo verified field indicator for MyInfo child DOB #6751

@tshuli

• Fill in a form that contains the MyInfo child DOB field
• Select a child, the child's DOB should be automatically prefilled
• Submit the form
• Check that in the email response, the DOB field name is prefixed with [MyInfo]

feat(virus-scanner): download clean file #6747

@foochifa

Make sure clean files can be downloaded from clean bucket

Preparations:

Get the storage submission request:

• Ensure that your app:
  1. Is connected to growthbook - ensure that the client key is loaded.
  2. Will direct storage submission requests to /storage.
• Create a storage submission form with 2 attachment fields (at least 1 of them optional) and open it.
• Go to the respondent page and open the network panel.
• Make a submission including an attachment. To make your life easier with the tests below, attach a small file (e.g. text file with a few bytes).
  • Hint: Content-Types:
    • text/plain
    • application/pdf
    • image/png
    • image/gif
• Copy the cURL request A for the submission. It should be to the endpoint that shows up as storage?recaptcha....
• Modify A to change the version to 2.1. Save this as request B.

Put a valid file in the quarantine bucket:

• Create a file with a valid uuid (e.g. 1b90195b-ce8a-4590-810b-04ebaef8e4dd).
• Upload the file to the quarantine bucket.
  • If you are on local:
    1. Go to formsg-backend-1's terminal.
    2. Create a text file with something like echo "testing 123" > 1b90195b-ce8a-4590-810b-04ebaef8e4dd.
    3. Copy the file over to the quarantine bucket with aws --endpoint=http://localhost:4566 s3 cp 1b90195b-ce8a-4590-810b-04ebaef8e4dd s3://local-virus-scanner-quarantine-bucket
    4. Make sure that the file has been successfully copied over with aws --endpoint=http://localhost:4566 s3 ls s3://local-virus-scanner-quarantine-bucket
  • If you are not on local, in S3, look for [staging|prod|uat].virus.scanner.quarantine.

Allow virus scanner to be used on BE:

• In the DB's featureflags collection, add the flag encryption-boundary-shift-virus-scanner and set enabled to true.

Make sure that virus scanner runs and clean file is downloaded when feature flag is on and version number is 2.1+:

• Modify cURL request B by changing the attachment field's answer to the file that you've added to the quarantine bucket. The file you've added should have a key that's a valid uuid. Save this as request C.
• Run request C.
  • The storage submission should succeed.
  • Go to the admin results page of the form. The latest response should have the attachment that you previously uploaded to the quarantine bucket, NOT the file that you used to create the cURL response.
• Rerun request C. This should now fail with a virus scan failure message since the quarantine bucket file has poofed.

Regression

Make sure that encryption-boundary-shift-virus-scanner feature flag works

• Unset the encryption-boundary-shift-virus-scanner feature flag or disable it.
• Restore the file in the quarantine bucket.
• Run cURL request C. The file in the quarantine bucket should remain.

Make sure that versioning works

• Enable the encryption-boundary-shift-virus-scanner feature flag.
• Ensure that the file is in the quarantine bucket.
• Run modified request C by changing the version number to 2. The file in the quarantine bucket should remain.

Make sure that only uuid file keys are accepted

• Run modified request C, where the filename is not a valid UUID. An error should be thrown flagging that the file key is not a valid UUID.

Normal E2E flows

• Submit a email mode form with multiple attachments.
• With encryption-boundary-shift-virus-scanner flag disabled, submit a storage mode form (/storage endpoint) with multiple attachments.
• Try this with some of the attachment fields empty.

Deploy Notes

Modified AWS policies:

• formsg-staging-virus-scanner-s3-lambda policy modified (for formsg-staging-ec2-role-1), in the Statement for Sid "GetFileFromClean", GetObject is replaced by GetObjectVersion. As such, the statement should look like the following:

{
  "Sid": "GetFileFromClean",
  "Effect": "Allow",
  "Action": "s3:GetObjectVersion",
  "Resource": "arn:aws:s3:::staging.virus.scanner.clean/*"
}

• Corresponding policies have been added to uat and prod too.

fix: rm growthbook legacy streaming prop #6744

@wanlingt
Before tests: Make sure that growthbook is connected.

Regression tests:

• Create a storage submission form.
• Set the encryption-boundary-shift flag on growthbook to on.
• Make a submission on the storage mode form. This should be made to the /submissions/storage endpoint.
• Change the encryption-boundary-shift flag on growthbook to off.
• Make a submission on the storage mode form. This should be made to the /submissions/encrypt endpoint.

feat(virus-scanner): invoke lambda to scan file #6734

@justynoh

Sed I can't put the requests for this set of tests on Insomnia :'(

Make sure virus scanning lamda is invoked under the right conditions:

Preparations:

Get the storage submission request:

• Ensure that your app:
  1. Is connected to growthbook - ensure that the client key is loaded.
  2. Will direct storage submission requests to /storage.
• Create a storage submission form with 2 attachment fields and open it.
• Go to the respondent page and open the network panel.
• Make a submission including an attachment. To make your life easier with the tests below, attach a small file (e.g. text file with a few bytes).
• Copy the cURL request A for the submission. It should be to the endpoint that shows up as storage?recaptcha....
• Modify A to change the version to 2.1. Save this as request B.

Put a valid file in the quarantine bucket:

• Create a file with a valid uuid (e.g. 1b90195b-ce8a-4590-810b-04ebaef8e4dd).
• Upload the file to the quarantine bucket.
  • If you are on local:
    1. Go to formsg-backend-1's terminal.
    2. Create a text file with something like echo "testing 123" > 1b90195b-ce8a-4590-810b-04ebaef8e4dd.
    3. Copy the file over to the quarantine bucket with aws --endpoint=http://localhost:4566 s3 cp 1b90195b-ce8a-4590-810b-04ebaef8e4dd s3://local-virus-scanner-quarantine-bucket
    4. Make sure that the file has been successfully copied over with aws --endpoint=http://localhost:4566 s3 ls s3://local-virus-scanner-quarantine-bucket
  • If you are not on local, in S3, look for [staging|prod|uat].virus.scanner.quarantine.

Allow virus scanner to be used on BE:

• In the DB's featureflags collection, add the flag encryption-boundary-shift-virus-scanner and set enabled to true.

Make sure that virus scanner runs when feature flag is on and version number is 2.1+:

• Modify cURL request B by changing the filename to the file that you've added to the quarantine bucket. The file you've added should have a key that's a valid uuid. Save this as request C.
• Run request C. If encryption-boundary-shift-hard-validation is disabled,
  • The storage submission should be submitted successfully. Note that the name of the file is now the uuid you've pasted.
  • In the logs (cloudwatch or local), there should be a warning for invalid file extension thrown by the submission validator.
  • The file you've uploaded to the quarantine bucket should have poofed! A new file will now be in the clean bucket. This file will be exactly the same as the file you've uploaded.
  • Restore the quarantine bucket file if you're still testing for effect of the encryption-boundary-shift-hard-validation flag.
• Run request C. If encryption-boundary-shift-hard-validation is enabled,
  • The storage submission should fail with a validation error for invalid file extension.
  • The file you've uploaded to the quarantine bucket should have poofed! A new file will now be in the clean bucket. This file will be exactly the same as the file you've uploaded.
  • Restore the quarantine bucket file if you're still testing for effect of the encryption-boundary-shift-hard-validation flag.
  • Set encryption-boundary-shift-hard-validation to disabled.
• Rerun request C. This should now fail with a virus scan failure message since the quarantine bucket file has poofed.

Make sure that encryption-boundary-shift-virus-scanner feature flag works

• Unset the encryption-boundary-shift-virus-scanner feature flag or disable it.
• Restore the file in the quarantine bucket.
• Run cURL request C. The file in the quarantine bucket should remain.

Make sure that versioning works

• Enable the encryption-boundary-shift-virus-scanner feature flag.
• Ensure that the file is in the quarantine bucket.
• Run modified request C by changing the version number to 2. The file in the quarantine bucket should remain.

Make sure that only uuid file keys are accepted

• Run modified request C, where the filename is not a valid UUID. An error should be thrown flagging that the file key is not a valid UUID.

Regression tests:

• Submit a email mode form with multiple attachments.
• With encryption-boundary-shift-virus-scanner flag disabled, submit a storage mode form (/storage endpoint) with multiple attachments.
• Try this with some of the attachment fields empty.

Deploy Notes

Note: In production, ensure that in the DB's featureflags collection, the flag encryption-boundary-shift-virus-scanner is either unset or set to false.

New environment variables:

• VIRUS_SCANNER_LAMBDA_FUNCTION_NAME : name of virus scanner lambda
  • dev: function
  • prod: virus-scanner-production-virus-scanner
  • staging: virus-scanner-staging-virus-scanner
  • uat: virus-scanner-uat-virus-scanner

New dependencies:

• @aws-sdk/client-lambda: lambda package from aws-sdk v3, as part of FRM-993.

New AWS policies:

• formsg-staging-virus-scanner-s3-lambda policy added (for formsg-staging-ec2-role-1):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "TriggerVirusScanningLambda",
      "Effect": "Allow",
      "Action": "lambda:InvokeFunction",
      "Resource": "arn:aws:lambda:ap-southeast-1:652261905145:function:virus-scanner-staging-virus-scanner"
    },
    {
      "Sid": "PutFileInQuarantine",
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::staging.virus.scanner.quarantine/*"
    },
    {
      "Sid": "GetFileFromClean",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::staging.virus.scanner.clean/*"
    }
  ]
}

• Corresponding policies have been added to uat and prod too.

fix: redirect to frontend app after myinfo log in (for local dev environment) #6743

@justynoh
Staging/Prod

• Log into a MyInfo form
• Upon logging in successfully, you should be redirected back to FormSG

Local dev

• Log into a MyInfo form
• Upon logging in successfully, you should be redirected back to FormSG on localhost:3000

Deploy Notes

New environment variables:

• FE_APP_URL : Frontend application URL (for local development use only)

chore: update hydrogen-alpine version #6720

@LinHuiqing

• Email and storage mode submissions should work
• Email responses should be receivable
• Storage mode responses should be retrievable

fix: display error message when myinfo child fields are not filled in #6735

@LinHuiqing

• On a MyInfo form, add a child field. Choose to collect all types of child data
• Open up the form
• Sign in as a user that has a child (Staging NRICs here)
• Submit the form without populating any child fields (do not select a child)
• An error message should appear for each MyInfo child field
• Select a child and populate any myinfo child fields are still blank
• The error messages should disappear once the fields are populated
• Submit the form
• The form response should reflect the child's information

### fix: hotfix for 500 errors thrown on email bounce notification endpoint #6741
@tshuli

Prior to release, this should be tested on staging by triggering a bounce notification to staging prior to and after deployment. If a 500 error was thrown on staging prior to deployment, we should expect it not to happen after deployment.

Update: Not able to replicate 500s, but we have tested multiple times on email submissions and the bounce notifications were successfully parsed and returned 200.

[linear]

FRM-993 Upgrade to AWS SDK v3

Description

We see a lot of warnings in logs and tests because AWS will be retiring the rAWS SDK V2 in 2023, and we are urged to upgrade to AWS SDK v3

Please migrate your code to use AWS SDK for JavaScript (v3).
For more information, check the migration guide at https://a.co/7PzMCcy
(node:97498) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.

[LinHuiqing]

tested and thanks everyonee