Add management command and workflow for generating live test data from historical exports #2569

danswick · 2023-10-20T19:41:29Z

Adds a management command and workflow that:

Generates SACs from historical data.
Progresses those SACs through to dissemination.
Accepts optional lists of DBKEY and YEAR values.
Runs automatically on dev and staging deploys with a set of default DBKEY/YEAR values.

PR checklist: submitters

PR checklist: reviewers

Pull the branch to your local environment and run make docker-clean; make docker-first-run && docker compose up; then run docker compose exec web /bin/bash -c "python manage.py test"
Manually test out the changes locally, or check this box to verify that it wasn’t applicable in this case.
Check that the PR has appropriate tests. Look out for changes in HTML/JS/JSON Schema logic that may need to be captured in Python tests even though the logic isn’t in Python.
Verify that no Git surgery is necessary at any point (such as during a merge party), or, if it was, repeat the testing after it’s finished.

The larger the PR, the stricter we should be about these points.

For running in production-like environments.

github-actions · 2023-10-20T19:42:24Z

Terraform plan for dev

No changes. Your infrastructure matches the configuration.

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

Warning: Argument is deprecated

  with module.dev.module.database.cloudfoundry_service_instance.rds,
  on /tmp/terraform-data-dir/modules/dev.database/database/main.tf line 14, in resource "cloudfoundry_service_instance" "rds":
  14:   recursive_delete = var.recursive_delete

Since CF API v3, recursive delete is always done on the cloudcontroller side.
This will be removed in future releases

(and 2 more similar warnings elsewhere)

📝 Plan generated in Pull Request Checks #1423

github-actions · 2023-10-20T19:42:27Z

Terraform plan for meta

No changes. Your infrastructure matches the configuration.

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

Warning: Argument is deprecated

  with module.s3-backups.cloudfoundry_service_instance.bucket,
  on /tmp/terraform-data-dir/modules/s3-backups/s3/main.tf line 14, in resource "cloudfoundry_service_instance" "bucket":
  14:   recursive_delete = var.recursive_delete

Since CF API v3, recursive delete is always done on the cloudcontroller side.
This will be removed in future releases

📝 Plan generated in Pull Request Checks #1423

github-actions · 2023-10-20T19:54:19Z

File	Coverage		Missing
All files	`86%`	❌
api/serializers.py	`88%`	❌	`177-178` `183` `188`
api/test_views.py	`96%`	✅	`105`
api/uei.py	`87%`	❌	`17-18` `87` `119-120` `164` `168-169`
api/views.py	`97%`	✅	`196-197` `204-205` `226` `362-363`
audit/forms.py	`47%`	❌	`22-29` `142-149`
audit/intake_to_dissemination.py	`92%`	✅	`67-68` `201-207` `257`
audit/models.py	`86%`	❌	`57` `59` `64` `66` `213` `246` `419` `437-438` `446` `468` `544-545` `549` `557` `566` `572`
audit/test_commands.py	`87%`	❌
audit/test_mixins.py	`90%`	✅	`112-113` `117-119` `184-185` `189-191`
audit/test_validators.py	`95%`	✅	`436` `440` `608-609` `848` `855` `862` `869`
audit/test_workbooks_should_fail.py	`85%`	❌	`56` `83-84` `88`
audit/test_workbooks_should_pass.py	`90%`	✅	`56` `81`
audit/utils.py	`70%`	❌	`13` `21` `33-35` `38`
audit/validators.py	`94%`	✅	`137` `189` `288-289` `304-305` `486-490` `495-499` `515-524`
audit/views.py	`42%`	❌	`87-108` `131-132` `206-207` `252-253` `264-265` `267-271` `318-331` `334-348` `353-366` `383-389` `394-414` `441-445` `450-479` `522-526` `531-551` `578-582` `587-616` `659-663` `668-680` `683-693` `698-710` `737-738` `743-792` `795-835` `838-855`
audit/cross_validation/additional_ueis.py	`93%`	✅	`33`
audit/cross_validation/check_award_ref_declaration.py	`90%`	✅
audit/cross_validation/check_award_reference_uniqueness.py	`93%`	✅
audit/cross_validation/check_certifying_contacts.py	`87%`	❌
audit/cross_validation/check_findings_count_consistency.py	`91%`	✅
audit/cross_validation/check_ref_number_in_cap.py	`90%`	✅
audit/cross_validation/check_ref_number_in_findings_text.py	`90%`	✅
audit/cross_validation/errors.py	`78%`	❌	`30` `69`
audit/cross_validation/naming.py	`68%`	❌	`178-182`
audit/cross_validation/submission_progress_check.py	`92%`	✅	`62` `79`
audit/cross_validation/tribal_data_sharing_consent.py	`81%`	❌	`33` `36` `40`
audit/cross_validation/validate_general_information.py	`93%`	✅	`28-29`
audit/fixtures/single_audit_checklist.py	`55%`	❌	`146-183` `229-238`
audit/intakelib/exceptions.py	`71%`	❌	`7-9` `12`
audit/intakelib/intermediate_representation.py	`94%`	✅	`23-24` `87` `125` `158` `182-185`
audit/intakelib/mapping_audit_findings.py	`97%`	✅	`51`
audit/intakelib/mapping_audit_findings_text.py	`97%`	✅	`51`
audit/intakelib/mapping_federal_awards.py	`93%`	✅	`95`
audit/intakelib/mapping_util.py	`40%`	❌	`28` `32` `36` `70-99` `104` `124-135` `142-158` `162-167` `171-185` `190` `195-215` `220-237` `258` `263-264` `273-279` `289` `304` `309`
audit/intakelib/checks/check_all_unique_award_numbers.py	`79%`	❌	`24`
audit/intakelib/checks/check_aln_three_digit_extension_pattern.py	`73%`	❌	`27` `36`
audit/intakelib/checks/check_cardinality_of_passthrough_names_and_ids.py	`91%`	✅
audit/intakelib/checks/check_cluster_name_always_present.py	`82%`	❌	`21`
audit/intakelib/checks/check_cluster_total.py	`87%`	❌	`46`
audit/intakelib/checks/check_federal_award_passed_always_present.py	`82%`	❌	`18`
audit/intakelib/checks/check_findings_grid_validation.py	`84%`	❌	`57`
audit/intakelib/checks/check_is_a_workbook.py	`68%`	❌	`16`
audit/intakelib/checks/check_loan_guarantee.py	`90%`	✅	`51`
audit/intakelib/checks/check_look_for_empty_rows.py	`91%`	✅	`18`
audit/intakelib/checks/check_missing_award_numbers.py	`72%`	❌	`16` `22-23`
audit/intakelib/checks/check_no_major_program_no_type.py	`72%`	❌	`22` `31` `40`
audit/intakelib/checks/check_no_repeat_findings.py	`76%`	❌	`17` `26`
audit/intakelib/checks/check_other_cluster_names.py	`81%`	❌	`24` `34`
audit/intakelib/checks/check_passthrough_name_when_no_direct.py	`88%`	❌	`9` `47`
audit/intakelib/checks/check_sequential_award_numbers.py	`76%`	❌	`14` `22`
audit/intakelib/checks/check_start_and_end_rows_of_all_columns_are_same.py	`89%`	❌	`14`
audit/intakelib/checks/check_state_cluster_names.py	`65%`	❌	`23-24` `34`
audit/intakelib/checks/check_uei_exists.py	`65%`	❌	`17-18`
audit/intakelib/checks/runners.py	`90%`	✅	`120` `126`
audit/intakelib/checks/util.py	`84%`	❌	`16` `33` `38`
audit/management/commands/load_fixtures.py	`46%`	❌	`39-45`
audit/viewlib/submission_progress_view.py	`96%`	✅	`171-172`
audit/viewlib/tribal_data_consent.py	`34%`	❌	`23-41` `44-79`
audit/viewlib/upload_report_view.py	`26%`	❌	`32-35` `44` `91-117` `120-170` `178-209`
cms/views.py	`57%`	❌	`11-16` `29-30`
config/urls.py	`71%`	❌	`87`
dissemination/models.py	`99%`	✅	`460`
dissemination/migrations/0002_general_fac_accepted_date.py	`47%`	❌	`10-12`
djangooidc/backends.py	`78%`	❌	`32` `57-63`
djangooidc/exceptions.py	`66%`	❌	`19` `21` `23` `28`
djangooidc/oidc.py	`16%`	❌	`32-35` `45-51` `64-70` `92-149` `153-199` `203-226` `230-275` `280-281` `286`
djangooidc/views.py	`80%`	❌	`22` `43` `114`
djangooidc/tests/common.py	`96%`	✅
report_submission/forms.py	`92%`	✅	`35`
report_submission/views.py	`76%`	❌	`83` `215-216` `218` `240-241` `260-261` `287-396` `399-409`
report_submission/templatetags/get_attr.py	`76%`	❌	`8` `11-14` `18`
support/admin.py	`88%`	❌	`76` `79` `84` `91-97` `100-102`
support/cog_over.py	`90%`	✅	`30-33` `86` `93` `145`
support/signals.py	`66%`	❌	`23-24` `33-34`
support/test_cog_over.py	`98%`	✅	`134-135` `224`
support/management/commands/seed_cog_baseline.py	`98%`	✅	`20-21`
tools/update_program_data.py	`89%`	❌	`96`
users/auth.py	`95%`	✅	`40-41`
users/models.py	`97%`	✅	`51-52`
users/fixtures/user_fixtures.py	`91%`	✅

Minimum allowed coverage is 90%

Generated by 🐒 cobertura-action against 590832d

Runs defaults

…ds-e2e-test-data

backend/dissemination/management/commands/end_to_end_test_data_generator.py

jadudm

LGTM.

* Add management command and workflow for generating live test data from historical exports (#2569) * First commit of E2E improvements For running in production-like environments. * add e2e generator workflow and profile changes * lint * double os * more linting somehow * don't do generation via profile * Linting. Runs defaults * invoke via workflow instead of profile * Apply suggestions from code review * lint --------- Co-authored-by: Matt Jadud <[email protected]> * add workflow call to e2e data generator (#2571) * This makes sure all of the named ranges exist (#2586) * This makes sure all of the named ranges exist Updates the ALN code slightly, and adds a new check that looks at the workbook templates, and makes sure that all of the named ranges in the template are in the file being submitted by the user * Removed unused code and implemented logic to retrieve named range names * Fix for string parsing issue --------- Co-authored-by: Hassan D. M. Sambo <[email protected]> * Add barebones feature allowing users to send submissions back to in progress from ready for certification (#2572) * Add barebones feature allowing users to send submissions back to in progress from ready for certification. * Coverage appeasement offering. * More coverage appeasement. * More coverage appeasement; this does fix an actual minor bug, though. * Reduce test coverage pass threshold. * Improve lock icon. * Improve lock icon. * Certification Unlock Copy & Styling Updates (#2595) * Ready for certification - remove bolded & reformat * Submissions homepage - icon on own line & format * Certification unlock - format, copy, links. * upload-cancel -> form-cancel * "Straight quotation marks boo" Co-authored-by: Tadhg O'Higgins <[email protected]> --------- Co-authored-by: Tadhg O'Higgins <[email protected]> * Search MVP (#2419) * Add basic search form * Add dummy results table * move search to dissem app, impl basic form (#2431) * Basic styles for search sidebar * Style up results column * [DRAFT] Search Summary (#2480) * Search Summary * Some title info, make tables take up the full width of the page * Linting - py whitespace & imports, html closed tag * Summary view tests (#2512) * Flesh out Search view (#2494) * fleshing out search view * rm leftover list comprehension * cog-over search fix, only search public * more tests * quick pass at displaying search results * Add explainer text * Remove filter chicklets (for now) * Add pagination component * Rm download all results button * Add alt text * Search - Pre-populate Form, Summary Link, Link Icons, Formatting (#2538) * URLS - Add a backslash to the search url * Search - Link to summary, icon size, formatting * Summary - remove "back" button, formatting. * Search - prepopulate form after making a search * Remove usused import * Don't run methods on empty data, kids * Search - Move UEI and ALN above the name field * Djlint reformatting * Search - Audit Year (#2547) * URLS - Add a backslash to the search url * Search - Link to summary, icon size, formatting * Summary - remove "back" button, formatting. * Search - prepopulate form after making a search * Remove usused import * Don't run methods on empty data, kids * Search - Move UEI and ALN above the name field * Djlint reformatting * Search - Add audit year * search_general params default to none * Tests - test_audit_year, search_general cleanups * Revert "Merge branch 'mh/implement-mvp-search-form-2369' into jp/search-audit-year" This reverts commit 7c99f95, reversing changes made to 30b2743. * Remove unused Y/N choices * PDF downloads via Search (#2520) * first pass at pdf downloads via search * re-add AWS_S3_ENDPOINT_URL * re-add download link * linter * fix cog_or_over field name mismatch * fix search tests * check if file exists in s3, else 404 * log warning if file not found in S3 --------- Co-authored-by: Matt Henry <[email protected]> Co-authored-by: Tim Ballard <[email protected]> Co-authored-by: Tim Ballard <[email protected]> Co-authored-by: James Person <[email protected]> Co-authored-by: James Person <[email protected]> * fix cg internal endpoint setting (#2613) * Add mgiration file that should have been in 2572. (#2615) * API app cleanup (#2596) * rm IndexView * rm SACViewSet * Bump Actions (#2624) * Bump to v0.13.0 * Bump setup-node to v4 * Media Backups! (#2610) * pseudo code for s3 tar implementation * add awscli to allow list * File Rename * All commands Just need to sort creds now * update command file * Update proxy allow list for s3tar * Script for backing up the media * chmod +x * Remove mediabackup * add media backups * Bind service with additional instances * Removing allowlist modifications * Using this workflow to test/debug * Update * Update * File Rename * Reformat Media Backups Workflow * remove media backups from prod (for now) * Remove mediabackups folder as well * Remove from apply This will be handled in its own workflow * Database and Media Backups Commented out prod for future iteration * rename file * full path of aws (#2636) * Removing old workbook_generator tool (#2635) Co-authored-by: Phil Dominguez <“[email protected]”> --------- Co-authored-by: Dan Swick <[email protected]> Co-authored-by: Matt Jadud <[email protected]> Co-authored-by: Hassan D. M. Sambo <[email protected]> Co-authored-by: Tadhg O'Higgins <[email protected]> Co-authored-by: James Person <[email protected]> Co-authored-by: Matt Henry <[email protected]> Co-authored-by: Matt Henry <[email protected]> Co-authored-by: Tim Ballard <[email protected]> Co-authored-by: Tim Ballard <[email protected]> Co-authored-by: James Person <[email protected]> Co-authored-by: Alex Steel <[email protected]> Co-authored-by: Phil Dominguez <“[email protected]”>

jadudm and others added 2 commits October 20, 2023 15:36

First commit of E2E improvements

28c8f66

For running in production-like environments.

add e2e generator workflow and profile changes

74f55a9

danswick temporarily deployed to dev October 20, 2023 19:41 — with GitHub Actions Inactive

danswick temporarily deployed to meta October 20, 2023 19:41 — with GitHub Actions Inactive

lint

861378a

danswick temporarily deployed to dev October 20, 2023 19:47 — with GitHub Actions Inactive

danswick temporarily deployed to meta October 20, 2023 19:47 — with GitHub Actions Inactive

double os

853b2e8

danswick temporarily deployed to dev October 20, 2023 19:50 — with GitHub Actions Inactive

danswick temporarily deployed to meta October 20, 2023 19:50 — with GitHub Actions Inactive

more linting somehow

121e47e

danswick temporarily deployed to dev October 20, 2023 19:53 — with GitHub Actions Inactive

danswick temporarily deployed to meta October 20, 2023 19:53 — with GitHub Actions Inactive

don't do generation via profile

d8fe779

danswick temporarily deployed to dev October 20, 2023 19:59 — with GitHub Actions Inactive

danswick temporarily deployed to meta October 20, 2023 19:59 — with GitHub Actions Inactive

jadudm and others added 3 commits October 20, 2023 16:03

Linting.

01a1369

Runs defaults

Merge branch 'mj-ds-e2e-test-data' of github.com:GSA-TTS/FAC into mj-…

60f4353

…ds-e2e-test-data

invoke via workflow instead of profile

8e88eb1

danswick temporarily deployed to dev October 20, 2023 20:05 — with GitHub Actions Inactive

danswick temporarily deployed to meta October 20, 2023 20:05 — with GitHub Actions Inactive

Merge branch 'mj-ds-e2e-test-data' of github.com:GSA-TTS/FAC into mj-…

9ec0fb2

…ds-e2e-test-data

jadudm temporarily deployed to dev October 20, 2023 20:30 — with GitHub Actions Inactive

jadudm temporarily deployed to meta October 20, 2023 20:30 — with GitHub Actions Inactive

danswick commented Oct 20, 2023

View reviewed changes

backend/dissemination/management/commands/end_to_end_test_data_generator.py Outdated Show resolved Hide resolved

Apply suggestions from code review

2682bf1

danswick temporarily deployed to dev October 20, 2023 20:36 — with GitHub Actions Inactive

danswick temporarily deployed to meta October 20, 2023 20:36 — with GitHub Actions Inactive

danswick requested a review from jadudm October 20, 2023 20:37

lint

590832d

danswick temporarily deployed to dev October 20, 2023 20:41 — with GitHub Actions Inactive

danswick temporarily deployed to meta October 20, 2023 20:41 — with GitHub Actions Inactive

jadudm approved these changes Oct 20, 2023

View reviewed changes

jadudm added this pull request to the merge queue Oct 20, 2023

Merged via the queue into main with commit ae5c6e9 Oct 20, 2023
11 checks passed

jadudm deleted the mj-ds-e2e-test-data branch October 20, 2023 20:42

danswick mentioned this pull request Oct 20, 2023

Make e2e test generator workflow reusable #2571

Merged

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add management command and workflow for generating live test data from historical exports #2569

Add management command and workflow for generating live test data from historical exports #2569

danswick commented Oct 20, 2023 •

edited

Loading

github-actions bot commented Oct 20, 2023 •

edited

Loading

github-actions bot commented Oct 20, 2023 •

edited

Loading

github-actions bot commented Oct 20, 2023 •

edited

Loading

jadudm left a comment

Add management command and workflow for generating live test data from historical exports #2569

Add management command and workflow for generating live test data from historical exports #2569

Conversation

danswick commented Oct 20, 2023 • edited Loading

PR checklist: submitters

PR checklist: reviewers

github-actions bot commented Oct 20, 2023 • edited Loading

github-actions bot commented Oct 20, 2023 • edited Loading

github-actions bot commented Oct 20, 2023 • edited Loading

jadudm left a comment

Choose a reason for hiding this comment

danswick commented Oct 20, 2023 •

edited

Loading

github-actions bot commented Oct 20, 2023 •

edited

Loading

github-actions bot commented Oct 20, 2023 •

edited

Loading

github-actions bot commented Oct 20, 2023 •

edited

Loading