2024-10-17 | MAIN --> PROD | DEV (be611dc) --> STAGING #4395
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…4386) * Hardcode redirect path * Validate redirect_uri - Created new method in `Util` class to return a redirect based on whether or not the `url` is valid and safe. * Lint * Linting again * One more lint. Hopefully
* Sets thresholds for new Coverate Reporting Tool * Change Job Name * Version bump to v3.2 * Going back to cobertura * Testing CodeCoverageSummary * Update coverage
* Add check_tables command * Add check-tables workflow * Fix naming * fix command * Check Tables post deploy * Change file name * Prep v0.1.6 version change * Prep schedule for checking tables * Test v0.1.6 * Add check tables to dashboard * Add post deploy table check * Setup check table workflow * setup check table scheduler * Fix version reference * Add widget for Submissions within a backup window Testing dual query on the widget * Change backup log capture * Remove unnecessary columns * add row_count to check_tables operation * Version bump to v0.1.7 * Adds RowCount() prior to backup on the schedule * Version bump to v0.1.8 * Add widgets for Row Count
* Erroring on duplicate editor email * Lint * Normalizing method names * Lint * Comments * Lint
* Updating babel/eslint-parser and eslint-plugin-cypress * Updating zaproxy * Updating python to 3.13
jadudm
added
autogenerated
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
Terraform plan for staging Plan: 1 to add, 4 to change, 1 to destroy.Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~ update in-place
-/+ destroy and then create replacement
Terraform will perform the following actions:
# module.staging.cloudfoundry_app.postgrest will be updated in-place
!~ resource "cloudfoundry_app" "postgrest" {
id = "db50e549-24c7-4dcf-b30e-e9dee3512be8"
!~ memory = 1024 -> 512
name = "postgrest"
# (17 unchanged attributes hidden)
# (1 unchanged block hidden)
}
# module.staging.module.clamav.cloudfoundry_app.clamav_api will be updated in-place
!~ resource "cloudfoundry_app" "clamav_api" {
!~ docker_image = "ghcr.io/gsa-tts/fac/clamav@sha256:df51007d864a4dd6985e0c6a334936c5774e4facaa431b751fb014ae3e5c2ee8" -> "ghcr.io/gsa-tts/fac/clamav@sha256:ac1531c48759b6269bd89c23242443a5d9d82a0b770c56ff514d656d6bfc6f7d"
id = "d1bea029-d2d3-4b68-b16d-b216bcaea573"
!~ memory = 3072 -> 2048
name = "fac-av-staging"
# (16 unchanged attributes hidden)
# (1 unchanged block hidden)
}
# module.staging.module.cors.null_resource.cors_header must be replaced
-/+ resource "null_resource" "cors_header" {
!~ id = "******************" -> (known after apply)
!~ triggers = { # forces replacement
!~ "always_run" = "2024-10-16T10:57:04Z" -> (known after apply)
}
}
# module.staging.module.file_scanner_clamav.cloudfoundry_app.clamav_api will be updated in-place
!~ resource "cloudfoundry_app" "clamav_api" {
!~ docker_image = "ghcr.io/gsa-tts/fac/clamav@sha256:df51007d864a4dd6985e0c6a334936c5774e4facaa431b751fb014ae3e5c2ee8" -> "ghcr.io/gsa-tts/fac/clamav@sha256:ac1531c48759b6269bd89c23242443a5d9d82a0b770c56ff514d656d6bfc6f7d"
id = "e416bb0c-8b95-47f7-9cff-c4029ebbd2df"
!~ memory = 3072 -> 2048
name = "fac-av-staging-fs"
# (16 unchanged attributes hidden)
# (1 unchanged block hidden)
}
# module.staging.module.newrelic.newrelic_one_dashboard.log_review_dashboard will be updated in-place
!~ resource "newrelic_one_dashboard" "log_review_dashboard" {
id = "MzkxOTA3NnxWSVp8REFTSEJPQVJEfGRhOjYzNjI3NDM"
name = "FAC Log Review (staging)"
# (4 unchanged attributes hidden)
!~ page {
name = "FAC Log Review"
# (2 unchanged attributes hidden)
+ widget_billboard {
+ column = 9
+ height = 3
+ legend_enabled = true
+ row = 8
+ title = "staging Table Check Count - Pass"
+ width = 3
+ nrql_query {
+ query = "SELECT count(*) FROM Log WHERE `tags.space_name` = 'staging' AND allColumnSearch('CHECKTABLESPASS', insensitive: true) SINCE 7 days ago"
}
}
+ widget_billboard {
+ column = 9
+ height = 3
+ legend_enabled = true
+ row = 9
+ title = "staging Table Check Count - Fail"
+ width = 3
+ nrql_query {
+ query = "SELECT count(*) FROM Log WHERE `tags.space_name` = 'staging' AND allColumnSearch('DBMISSINGTABLES', insensitive: true) SINCE 7 days ago"
}
}
+ widget_log_table {
+ column = 1
+ height = 3
+ legend_enabled = true
+ row = 8
+ title = "staging Check Tables Logs"
+ width = 8
+ nrql_query {
+ query = "SELECT `message` FROM Log WHERE `tags.space_name` = 'staging' AND allColumnSearch('CHECKTABLESPASS', insensitive: true) SINCE 7 days ago"
}
}
+ widget_log_table {
+ column = 1
+ height = 3
+ legend_enabled = true
+ row = 9
+ title = "staging Missing Tables Logs"
+ width = 8
+ nrql_query {
+ query = "SELECT `message` FROM Log WHERE `tags.space_name` = 'staging' AND allColumnSearch('DBMISSINGTABLES', insensitive: true) SINCE 7 days ago"
}
}
+ widget_table {
+ column = 1
+ height = 3
+ legend_enabled = true
+ linked_entity_guids = (known after apply)
+ row = 10
+ title = "staging Backups and Submissions"
+ width = 8
+ nrql_query {
+ query = "SELECT `message` FROM Log WHERE allColumnSearch('POST', insensitive: true) AND allColumnSearch('/submission/', insensitive: true) AND `newrelic.source` = 'logs.APM' AND entity.name ='staging' SINCE 2 hours ago"
}
+ nrql_query {
+ query = "SELECT `message`,`timestamp` FROM Log WHERE allColumnSearch('STARTUP_CHECK', insensitive: true) AND `message` LIKE '%db_to_s3%' AND `message` LIKE '%PASS%' AND tags.space_name ='staging' SINCE 2 hours ago"
}
}
+ widget_table {
+ column = 1
+ height = 3
+ legend_enabled = true
+ linked_entity_guids = (known after apply)
+ row = 11
+ title = "staging Backup and Row Count"
+ width = 8
+ nrql_query {
+ query = "SELECT `message` FROM Log WHERE `tags.space_name` = 'staging' AND allColumnSearch('\"TABLEROWCOUNT\"', insensitive: true) SINCE 2 hours ago"
}
+ nrql_query {
+ query = "SELECT `message`,`timestamp` FROM Log WHERE allColumnSearch('STARTUP_CHECK', insensitive: true) AND `message` LIKE '%db_to_s3%' AND `message` LIKE '%PASS%' AND tags.space_name ='staging' SINCE 2 hours ago"
}
}
+ widget_table {
+ column = 1
+ height = 3
+ legend_enabled = true
+ linked_entity_guids = (known after apply)
+ row = 12
+ title = "staging Row Count"
+ width = 8
+ nrql_query {
+ query = "SELECT `message` FROM Log WHERE `tags.space_name` = 'staging' AND allColumnSearch('\"TABLEROWCOUNT\"', insensitive: true) SINCE 7 days ago"
}
}
# (13 unchanged blocks hidden)
}
}
Plan: 1 to add, 4 to change, 1 to destroy.
Warning: Argument is deprecated
with module.staging-backups-bucket.cloudfoundry_service_instance.bucket,
on /tmp/terraform-data-dir/modules/staging-backups-bucket/s3/main.tf line 14, in resource "cloudfoundry_service_instance" "bucket":
14: recursive_delete = var.recursive_delete
Since CF API v3, recursive delete is always done on the cloudcontroller side.
This will be removed in future releases
(and 6 more similar warnings elsewhere)📝 Plan generated in Pull Request Checks #3856 |
|
Terraform plan for production Plan: 1 to add, 3 to change, 1 to destroy.Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~ update in-place
-/+ destroy and then create replacement
Terraform will perform the following actions:
# module.production.module.clamav.cloudfoundry_app.clamav_api will be updated in-place
!~ resource "cloudfoundry_app" "clamav_api" {
!~ docker_image = "ghcr.io/gsa-tts/fac/clamav@sha256:df51007d864a4dd6985e0c6a334936c5774e4facaa431b751fb014ae3e5c2ee8" -> "ghcr.io/gsa-tts/fac/clamav@sha256:ac1531c48759b6269bd89c23242443a5d9d82a0b770c56ff514d656d6bfc6f7d"
id = "5d0afa4f-527b-472a-8671-79a60335417f"
name = "fac-av-production"
# (17 unchanged attributes hidden)
# (1 unchanged block hidden)
}
# module.production.module.cors.null_resource.cors_header must be replaced
-/+ resource "null_resource" "cors_header" {
!~ id = "*******************" -> (known after apply)
!~ triggers = { # forces replacement
!~ "always_run" = "2024-10-16T18:17:34Z" -> (known after apply)
}
}
# module.production.module.file_scanner_clamav.cloudfoundry_app.clamav_api will be updated in-place
!~ resource "cloudfoundry_app" "clamav_api" {
!~ docker_image = "ghcr.io/gsa-tts/fac/clamav@sha256:df51007d864a4dd6985e0c6a334936c5774e4facaa431b751fb014ae3e5c2ee8" -> "ghcr.io/gsa-tts/fac/clamav@sha256:ac1531c48759b6269bd89c23242443a5d9d82a0b770c56ff514d656d6bfc6f7d"
id = "6b0ce685-a80c-4feb-9ed6-49c6427efdb8"
name = "fac-av-production-fs"
# (17 unchanged attributes hidden)
# (1 unchanged block hidden)
}
# module.production.module.newrelic.newrelic_one_dashboard.log_review_dashboard will be updated in-place
!~ resource "newrelic_one_dashboard" "log_review_dashboard" {
id = "MzkxOTA3NnxWSVp8REFTSEJPQVJEfGRhOjY0MzQ5OTY"
name = "FAC Log Review (production)"
# (4 unchanged attributes hidden)
!~ page {
name = "FAC Log Review"
# (2 unchanged attributes hidden)
+ widget_billboard {
+ column = 9
+ height = 3
+ legend_enabled = true
+ row = 8
+ title = "production Table Check Count - Pass"
+ width = 3
+ nrql_query {
+ query = "SELECT count(*) FROM Log WHERE `tags.space_name` = 'production' AND allColumnSearch('CHECKTABLESPASS', insensitive: true) SINCE 7 days ago"
}
}
+ widget_billboard {
+ column = 9
+ height = 3
+ legend_enabled = true
+ row = 9
+ title = "production Table Check Count - Fail"
+ width = 3
+ nrql_query {
+ query = "SELECT count(*) FROM Log WHERE `tags.space_name` = 'production' AND allColumnSearch('DBMISSINGTABLES', insensitive: true) SINCE 7 days ago"
}
}
+ widget_log_table {
+ column = 1
+ height = 3
+ legend_enabled = true
+ row = 8
+ title = "production Check Tables Logs"
+ width = 8
+ nrql_query {
+ query = "SELECT `message` FROM Log WHERE `tags.space_name` = 'production' AND allColumnSearch('CHECKTABLESPASS', insensitive: true) SINCE 7 days ago"
}
}
+ widget_log_table {
+ column = 1
+ height = 3
+ legend_enabled = true
+ row = 9
+ title = "production Missing Tables Logs"
+ width = 8
+ nrql_query {
+ query = "SELECT `message` FROM Log WHERE `tags.space_name` = 'production' AND allColumnSearch('DBMISSINGTABLES', insensitive: true) SINCE 7 days ago"
}
}
+ widget_table {
+ column = 1
+ height = 3
+ legend_enabled = true
+ linked_entity_guids = (known after apply)
+ row = 10
+ title = "production Backups and Submissions"
+ width = 8
+ nrql_query {
+ query = "SELECT `message` FROM Log WHERE allColumnSearch('POST', insensitive: true) AND allColumnSearch('/submission/', insensitive: true) AND `newrelic.source` = 'logs.APM' AND entity.name ='production' SINCE 2 hours ago"
}
+ nrql_query {
+ query = "SELECT `message`,`timestamp` FROM Log WHERE allColumnSearch('STARTUP_CHECK', insensitive: true) AND `message` LIKE '%db_to_s3%' AND `message` LIKE '%PASS%' AND tags.space_name ='production' SINCE 2 hours ago"
}
}
+ widget_table {
+ column = 1
+ height = 3
+ legend_enabled = true
+ linked_entity_guids = (known after apply)
+ row = 11
+ title = "production Backup and Row Count"
+ width = 8
+ nrql_query {
+ query = "SELECT `message` FROM Log WHERE `tags.space_name` = 'production' AND allColumnSearch('\"TABLEROWCOUNT\"', insensitive: true) SINCE 2 hours ago"
}
+ nrql_query {
+ query = "SELECT `message`,`timestamp` FROM Log WHERE allColumnSearch('STARTUP_CHECK', insensitive: true) AND `message` LIKE '%db_to_s3%' AND `message` LIKE '%PASS%' AND tags.space_name ='production' SINCE 2 hours ago"
}
}
+ widget_table {
+ column = 1
+ height = 3
+ legend_enabled = true
+ linked_entity_guids = (known after apply)
+ row = 12
+ title = "production Row Count"
+ width = 8
+ nrql_query {
+ query = "SELECT `message` FROM Log WHERE `tags.space_name` = 'production' AND allColumnSearch('\"TABLEROWCOUNT\"', insensitive: true) SINCE 7 days ago"
}
}
# (13 unchanged blocks hidden)
}
}
Plan: 1 to add, 3 to change, 1 to destroy.
Warning: Argument is deprecated
with module.domain.cloudfoundry_service_instance.external_domain_instance,
on /tmp/terraform-data-dir/modules/domain/domain/main.tf line 45, in resource "cloudfoundry_service_instance" "external_domain_instance":
45: recursive_delete = var.recursive_delete
Since CF API v3, recursive delete is always done on the cloudcontroller side.
This will be removed in future releases
(and 6 more similar warnings elsewhere)📝 Plan generated in Pull Request Checks #3856 |
…ation (#4351) * Create move_to_disseminated.py New management command file for attempting to move a SAC to dissemination. This is the first commit on this feature. Command is: `python manage.py move_to_disseminated --report_id ID_GOES_HERE`. This will look for a SAC with the `report_id` that you entered, and attempt to move it to the `disseminated` status - ONLY if it is stuck as `submitted`. * Linting * Comments for potential enhancements to management command * Enhance management command - Replaced logs with print statements to avoid bloating NR. - Now checking for validation errors (similar to cross-validation step) before proceeding to attempt dissemination. - New viewflow routine for transitioning a SAC from `submitted` to `auditee_certified`. This is only ran in the event there are errors with validation OR dissemination. * Fix test - submission_status_transitions * Linting * Curation library and dissemination fixes - Updated some error codes and logging more responses in `move_to_disseminated`. - Brought in a new application for enabling/disabling audit tracking from `jadudm/curation-api`, which creates a table `curation.record_version` as well as functions to manipulate it, for tracking the changelog of SACs. - Now tracking the changelog of the SAC when running `move_to_dissemination`. - Added a condition in `remove_workbook_artifacts.py` when logging after bulk deletion of S3 files to ensure the key is not null. - Added a new routine in `IntakeToDissemination` which takes a status and finds the first occurring date of that status for a SAC. `fac_acceptance_date` moving forward will now be based off of the first time a SAC was `submitted`, rather than the most recent time a SAC was `submitted`. * Feedback from jadud - New class `CurationTracking`, which allows us to wrap the audit trailing using a `with`. See the changelog below for reference. - Tracking `sac_reverted_from_submitted` in our curation table. - Now using `else` in dissemination validation check. - Calling `--disable` on `curation_audit_tracking` on startup of the application. * Update curation_audit_tracking_init.sh * Attempted fix - disable audit tracking on startup fails * Move curation after migrations * Update init_curation_auditing.sql
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is an auto-generated pull request to merge main into prod for a staging release on 2024-10-17 with the last commit being merged as be611dc