Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Migrate grants_interested data to grant_followers and update PUT handler #3593

Merged

Conversation

sushilrajeeva
Copy link
Contributor

@sushilrajeeva sushilrajeeva commented Oct 8, 2024

…T

Ticket #2961

Description

This PR addresses the requirements outlined in the issue #2961 as follows:

  1. Knex Migration: Migrates existing data from the grants_interested table to the grant_followers table where the interested_code references records with status_code = 'Interested'.
  2. PUT Handler Update: Modified the PUT /:grantId/interested/:agencyId route to keep the grant_followers table in sync when users update their interest status. After marking a grant as "Interested" via the markGrantAsInterested() function, the handler now follows/unfollows grants based on the interestedCode provided.

Screenshots / Demo Video

No UI changes. Backend logic updates and database migrations.

Testing

Automated and Unit Tests

  • Added Unit tests

Manual tests for Reviewer

  • Added steps to test feature/functionality manually

Checklist

  • Provided ticket and description
  • Provided screenshots/demo
  • Provided testing information
  • Provided adequate test coverage for all new code
  • Added PR reviewers

…T handler

- Created Knex migration to migrate existing data from grants_interested to grant_followers for records with 'Interested' status.
- Updated PUT /:grantId/interested/:agencyId route to ensure grant_followers table is kept up-to-date when users mark interest in grants.
- Added logic to follow/unfollow grants based on interestedCode status using followGrant and unfollowGrant functions.
- Ensured backward compatibility by handling follower logic after calling markGrantAsInterested.
@github-actions github-actions bot added database-changes Includes schema migrations or other critical changes enhancement New feature or request javascript Pull requests that update Javascript code labels Oct 8, 2024
Copy link

github-actions bot commented Oct 8, 2024

QA Summary

QA Check Result
🌐 Client Tests
🔗 Server Tests
🤝 E2E Tests
📏 ESLint
🧹 TFLint

Test Coverage

Coverage report for `packages/client`
St File % Stmts % Branch % Funcs % Lines Uncovered Line #s
🔴 All files 35.54 34.41 32.34 37.12
🔴  src 0 100 100 0
🔴   App.vue 0 100 100 0 2-9
🔴  src/arpa_reporter 0 100 100 0
🔴   App.vue 0 100 100 0 2-13
🟡  ...ter/components 58.58 48 46.8 59.78
🟡   AlertBox.vue 80 75 50 80 13
🔴   ...oadButton.vue 20 0 0 20 2-7,38-67
🟢   ...ileButton.vue 100 100 100 100
🟢   ...ttonSmall.vue 100 100 100 100
🟢   ...mplateBtn.vue 100 100 100 100
🟡   ...avigation.vue 67.74 63.63 52.63 67.74 ...13-219,228-235
🔴   StandardForm.vue 45 50 41.66 45.45 ...24-128,135-157
🟢  ...porter/helpers 84.61 79.48 87.5 84.61
🟢   form-helpers.js 84.21 79.48 85.71 84.21 7,16,25,81-83
🟢   short-uuid.js 100 100 100 100
🔴  ...eporter/router 0 0 0 0
🔴   index.js 0 0 0 0 20-131
🔴  ...reporter/store 3.92 0 2.17 4.12
🔴   index.js 3.92 0 2.17 4.12 11-14,32-261
🔴  ...reporter/views 40.98 25.13 41.37 42.91
🟢   AgenciesView.vue 100 0 100 100 16
🔴   AgencyView.vue 40.74 36.36 50 45.83 53-62,70-96
🔴   HomeView.vue 26.66 34.14 36.36 26.66 36-66,113,137-207
🔴   LoginView.vue 29.62 36.36 20 30.76 1,4,19-34,72-100
🔴   ...plateView.vue 27.02 35.71 45.45 25 ...2,30-37,69-113
🔴   ...ploadView.vue 24.24 16.66 33.33 26.66 1,30-31,116-144
🔴   ...eriodView.vue 48 18.18 75 52.17 64-90
🟡   ...riodsView.vue 57.57 28.57 60 59.37 101,132,149-171
🔴   ...pientView.vue 40 10.52 30.76 41.02 73-93,110-152
🔴   ...ientsView.vue 48.48 8.33 46.66 53.33 ...46,165-190,203
🟡   UploadView.vue 61.53 43.75 62.5 61.36 ...41-442,448-449
🔴   UploadsView.vue 16.66 0 0 18.18 59,110-287
🔴   UserView.vue 46.34 33.33 68.75 48.64 84,97-137
🟡   UsersView.vue 60 7.69 62.5 66.66 91-92,105-107,123
🔴   ...ationView.vue 37.03 18.18 26.66 41.66 ...19,238,246-270
🟡  src/components 53.6 44.2 55.31 54.31
🔴   ...vityTable.vue 18.75 0 28.57 19.35 115-185
🟡   BaseLayout.vue 69.56 53.84 60 69.56 172,219-231
🔴   CopyButton.vue 30 33.33 40 30 5,53-62
🟢   ...tActivity.vue 80.76 62.85 83.33 80.76 ...31-136,179,201
🟡   GrantNotes.vue 78.57 76.19 77.77 78.04 163-165,184-195
🔴   GrantsTable.vue 45.03 38.77 45 46.85 ...51-455,463-543
🔴   ...dUploader.vue 24.24 33.33 37.5 24.24 16,25,74-111
🔴   SearchFilter.vue 40.74 28 37.5 40 ...52,64,67,72-82
🔴   ShareGrant.vue 38.7 64.28 23.07 40 ...08-112,131-152
🟢   ...ivityItem.vue 96.15 90.9 100 96.15 64
🟡   UserAvatar.vue 70 75 100 62.5 35-37
🟢   ...eaderText.vue 100 100 100 100
🔴  ...ponents/Modals 29.21 44.34 33.1 32.94
🔴   ...anization.vue 17.39 65.62 30 22.22 1-14,149-178
🔴   AddTeam.vue 45.45 55.55 61.53 57.69 ...04,210,222-245
🔴   AddUser.vue 36.66 66.66 60 37.5 ...40,145,148-176
🔴   ...anization.vue 15.38 64.28 14.28 16.66 1-15,58-78
🔴   EditTeam.vue 18.18 26.43 33.33 20.83 ...29,208,216-301
🔴   EditUser.vue 21.05 66.66 25 22.22 1,101-128
🔴   ...ilsLegacy.vue 22.03 0 0 24.07 131,177,205-369
🟢   ...Followers.vue 83.33 88.88 57.14 83.33 10,114-115
🟡   ImportTeams.vue 50 41.17 50 53.33 28,64-69,81-82
🔴   ImportUsers.vue 42.85 60 40 46.15 29,65-80
🔴   ...archPanel.vue 27.9 15.78 23.52 28.57 ...68-178,211-255
🔴   SearchPanel.vue 21.62 63.26 32 27.58 ...77-380,386-458
🔴  src/helpers 18.12 17.92 18.6 18.88
🟢   constants.js 100 100 100 100
🟢   currency.js 100 100 100 100
🟡   dates.js 66.66 100 33.33 100
🔴   fetchApi.js 6 13.79 5.26 6.12 10-12,20-132
🔴   filters.js 4 0 0 4.54 19-51
🔴   form-helpers.js 0 0 0 0 5-82
🟡   gtag.js 77.77 90 75 77.77 12,51
🟢   testHelpers.js 100 100 100 100
🔴   ...patWarning.js 0 0 0 0 39-61
🟢  ...s/featureFlags 84.61 100 71.42 84.61
🟡   index.js 60 100 60 60 8,16
🟢   utils.js 100 100 100 100
🔴  src/mixin 20 0 28.57 20
🔴   ...zableTable.js 20 0 28.57 20 16-31,36-37,42
🔴  src/router 18.91 14.28 11.11 18.91
🔴   index.js 18.91 14.28 11.11 18.91 ...76-177,181-200
🟢  src/store 100 100 100 100
🟢   index.js 100 100 100 100
🔴  src/store/modules 2.89 0 4.51 3.01
🔴   agencies.js 5.26 100 8.33 5.55 13-70
🔴   alerts.js 20 100 20 20 10-24
🔴   grants.js 1.03 0 0.98 1.06 61-445
🔴   organization.js 33.33 100 33.33 33.33 21-25
🔴   roles.js 20 100 20 25 13-22
🔴   tenants.js 11.11 100 14.28 12.5 13-32
🔴   users.js 2.43 0 4.76 2.5 17-100
🔴  src/views 44.73 37.22 33.77 46.55
🔴   ...orterView.vue 25.58 51.85 18.18 26.82 ...,62,84,109-151
🟡   ...boardView.vue 50 17.64 50 52 89-98,114-125
🔴   ...tailsView.vue 32.97 14.28 17.07 33.69 ...96-435,441-462
🟢   GrantsView.vue 100 100 100 100
🟡   LoginView.vue 56 38.88 44.44 58.33 23,137-159
🟡   MyGrantsView.vue 77.77 66.66 66.66 77.77 1,69
🟡   ...ofileView.vue 78.26 80 42.85 78.26 1,32,63,136-140
🟢   NotFoundView.vue 100 100 100 100
🔴   ...tionsView.vue 47.05 57.14 41.66 53.33 ...97-100,114-118
🔴   ...ivityView.vue 46.42 23.8 43.75 46.42 ...01,114,120-134
🔴   TeamsView.vue 44.44 88.88 41.66 53.33 1,58,142,156-163
🟡   UsersView.vue 50 66.66 36.36 53.84 ...16-121,133-139
Coverage report for `packages/server`
St File % Stmts % Branch % Funcs % Lines Uncovered Line #s
🟡 All files 59.79 53.07 55.52 59.87
🟢  src 81.63 33.33 60 81.63
🟢   configure.js 81.63 33.33 60 81.63 42,61-68,97-99
🟢  src/arpa_reporter 98.75 66.66 100 98.75
🟢   configure.js 97.36 40 100 97.36 36
🟢   environment.js 100 100 100 100
🟢   use-request.js 100 100 100 100
🟡  src/arpa_reporter/db 50.73 45.88 50 51.9
🟡   arpa-subrecipients.js 53.19 50 38.46 54.54 23-60,101,113-122
🔴   reporting-periods.js 37.2 46.87 40 38.09 46,77-156
🟢   settings.js 100 83.33 100 100 13
🟡   uploads.js 50 28.57 52.38 51.42 18-29,84,99-124,141-150
🔴  src/arpa_reporter/lib 29.57 33.08 34.56 28.46
🟢   arpa-ec-codes.js 100 100 100 100
🔴   audit-report.js 21.44 19.35 24.19 21.32 ...28-529,554-684,732-758
🟡   ensure-async-context.js 75 100 50 100
🟢   format.js 90.62 90 90 91.3 41-42
🟡   log.js 75 50 50 75 13,25
🟡   preconditions.js 66.66 33.33 100 66.66 3
🔴   spreadsheet.js 9.09 0 0 9.09 15-32
🟢   validation-error.js 85.71 100 50 85.71 16
🔴  src/arpa_reporter/routes 40 14.92 14.28 40.6
🔴   agencies.js 22.58 0 0 23.33 13-21,26-53
🟡   application_settings.js 75 100 0 75 10-11
🟡   audit-report.js 68.91 58.33 100 68.91 57-58,64-78,100-116
🟢   exports.js 81.42 83.33 100 81.42 61-75,98-99
🔴   reporting-periods.js 20 0 0 20.43 ...25-137,143-149,154-180
🔴   subrecipients.js 23.8 0 0 23.8 12-13,17-27,31-48,52-63
🔴   uploads.js 28.28 7.89 9.09 29.16 ...33-154,164-166,173-180
🔴   users.js 19.6 0 0 20 15-35,39-44,48-81
🔴  src/arpa_reporter/services 44.55 34.79 47.55 44.83
🔴   generate-arpa-report.js 36.86 2.77 50 37.24 ...-975,984-997,1071-1138
🔴   get-template.js 21.62 0 0 21.62 18-79
🟡   persist-upload.js 69.66 90 73.07 69.76 ...64-206,227-241,279-301
🔴   records.js 20.75 0 11.11 21.15 38-204,221-276
🔴   revalidate-uploads.js 37.5 100 0 37.5 5-14
🔴   validate-upload.js 43.56 58.51 37.03 44.2 ...52,371,393,411-688,703
🟢   validation-rules.js 98.18 90 90.9 100 157,173
🟡  src/db 74.26 71.42 68.42 74.29
🟢   connection.js 100 50 100 100 6
🟢   constants.js 100 100 100 100
🟡   helpers.js 75 83.33 50 75 5,21-22
🟢   index.js 82.4 78.54 82.08 82.35 ...56-1422,1604-1605,1612
🟢   saved_search_migration.js 92 88.23 71.42 93.61 5,69,134
🔴   tenant_creation.js 10.58 2.7 0 11.11 15-40,[48-210](https://githu...*[Comment body truncated]*

Copy link

github-actions bot commented Oct 8, 2024

Terraform Summary

Step Result
🖌 Terraform Format & Style
⚙️ Terraform Initialization
🤖 Terraform Validation
📖 Terraform Plan

Hint: If "Terraform Format & Style" failed, run terraform fmt -recursive from the terraform/ directory and commit the results.

Output

Validation Output
stdout:
Success! The configuration is valid.


-------------------------------------
stderr:

Plan Summary
CHANGE RESOURCE
update module.api.aws_ecs_service.default[0]
module.api.module.grant_digest_scheduled_task.aws_iam_role_policy.default[0]
module.api.module.grant_digest_scheduled_task.aws_scheduler_schedule.default[0]
module.arpa_audit_report.aws_ecs_service.default
module.arpa_treasury_report.aws_ecs_service.default
module.consume_grants.aws_ecs_service.default
module.website.aws_s3_object.deploy-config[0]
module.website.aws_s3_object.origin_dist_artifact["assets/style-Lb7S2aAW.js.map"]
recreate module.api.aws_ecs_task_definition.default[0]
module.arpa_audit_report.aws_ecs_task_definition.consumer
module.arpa_treasury_report.aws_ecs_task_definition.consumer
module.consume_grants.aws_ecs_task_definition.consume_grants

Pusher: @sushilrajeeva, Action: pull_request_target, Workflow: Continuous Integration

@sushilrajeeva sushilrajeeva enabled auto-merge (squash) October 8, 2024 05:03
Copy link
Member

@TylerHendrickson TylerHendrickson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sushilrajeeva This is looking good. I have a few comments related to error handling and atomicity of database writes for you to take a look at.

Additionally, it would be preferable for this PR to add tests for the new behavior in packages/server/__tests__/api/grants.test.js.

Comment on lines 442 to 446
// Query to check if the interestedCode corresponds to 'Interested'
const interestedStatus = await knex('interested_codes')
.select('status_code')
.where({ id: interestedCode })
.first();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's run this query first in order to validate that the interestedCode provided by the request is valid before doing anything else.

Suggested change
// Query to check if the interestedCode corresponds to 'Interested'
const interestedStatus = await knex('interested_codes')
.select('status_code')
.where({ id: interestedCode })
.first();
// Query to check if the interestedCode corresponds to 'Interested'
const interestedStatus = await knex('interested_codes')
.select('status_code')
.where({ id: interestedCode })
.first();
if (!interestedStatus) {
res.status(400).json({ error: 'Invalid interestedCode in request' });
return;
}

Comment on lines 448 to 453
// Follow or Unfollow the grant based on the interestedCode
if (interestedStatus?.status_code === 'Interested') {
await followGrant(knex, grantId, user.id);
} else {
await unfollowGrant(knex, grantId, user.id);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should come before the call to db.markGrantAsInterested(), and be wrapped in a transaction so that we can roll back in case db.markGrantAsInterested() fails:

// Follow  or Unfollow the grant based on the interestedCode
const trx = await knex.transaction();
if (interestedStatus?.status_code === 'Interested') {
    await followGrant(trx, grantId, user.id);
} else {
    await unfollowGrant(trx, grantId, user.id);
}
try {
    // Calling the existing function to mark the grant as interested
    await db.markGrantAsInterested({
        grantId,
        agencyId,
        userId: user.id,
        interestedCode,
    });
} catch (error) {
    // Roll back the follow/unfollow operation since marking grant as interested failed
    trx.rollback();
    console.error('Error in marking as interested:', error);
    throw error;
}
// Commit the follow/unfollow operation since everything succeeded
trx.commit();

Note: The order of db.markGrantAsInterested() and either followGrant() or unfollowGrant() wouldn't matter if we were able to pass the transaction to db.markGrantAsInterested(), since that would allow us to commit or rollback as a unit. Modifying that function is arguably out of scope for this issue, but just to illustrate what I mean:

async function markGrantAsInterested({
    trx = knex, grantId, agencyId, userId, interestedCode,
}) {
    const results = await trx(TABLES.grants_interested)
        .insert({
            agency_id: agencyId,
            grant_id: grantId,
            user_id: userId,
            interested_code_id: +interestedCode,
        });
    return results;
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TylerHendrickson adding transaction to rollback if markGrantAsInterested fails is a great suggestion, thanks for the detailed implementation. I have incorporated your suggestion, I will push the code with the suggested changes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TylerHendrickson I have added await to the transaction.commit() and transaction.rollback().

Comment on lines 457 to 460
} catch (error) {
console.error('Error in marking as interested:', error);
res.status(500).send({ success: false, error: 'Internal Server Error' });
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer to remove this wrapping try/catch block since we're not inspecting it or otherwise using it to provide any additional context to the user if something were to fail. Otherwise the try/catch will suppress the error and make this harder to monitor and debug (in Datadog).

Copy link
Contributor

@phm200 phm200 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this locally and both the migration and PUT behavior met expectations.

Specifically I tested the following locally:

  • w/feature flag off, set myself as interested in two grants, then ran this branch, ran migration and turned on feature flag -> was following those two grants as expected
  • turned off feature flag, marked as interested in a new grant, turned on feature flag -> was following the grant as expected
  • spot checked a few grants I had not marked as interested -> not following the grant as expected
  • with feature flag off, marked as interested, then un-marked/deleted interested, turn on feature flag -> not following the grant as expected

@ClaireValdivia
Copy link
Contributor

@sushilrajeeva just want to make sure you see Ty's comments above for change requests!

@sushilrajeeva
Copy link
Contributor Author

@TylerHendrickson I have updated the test case for the PUT function. Specifically, I modified the previous implementation by adding a GET operation after performing the PUT request. This allows me to validate whether the agency.agency_id returned is correct. If there are any additional checks or improvements you'd recommend for this test, please let me know.

Copy link
Member

@TylerHendrickson TylerHendrickson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sushilrajeeva Thanks for the updates, looks great!

@sushilrajeeva sushilrajeeva merged commit 8fb4734 into main Oct 17, 2024
19 checks passed
@sushilrajeeva sushilrajeeva deleted the feat/sushilrajeeva/data-migration-status-to-follow+note branch October 17, 2024 15:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
database-changes Includes schema migrations or other critical changes enhancement New feature or request javascript Pull requests that update Javascript code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants