Let's stay in touch with @cmungall or @pkalita-lbl because there is some enhanced LInkML validation tooling in the works.

/cc @PeopleMakeCulture

Sounds like this ticket and #401 are the top two priorities coming out of Thursday's db sync. My concerns are:

1. Making sure we're all aligned on one strategy for validation and referential integrity
2. Understanding who will be responsible for which parts, which will inform how @dwinston and I prioritize our time

@eecavanna raised the possibility of organizing a validation & referential integrity squad. Would that be the best way forward?

From Slack:

Additional validation during DB inserts:

• Validation is focused around the validate_json function in nmdc_runtime.util, which in turn dispatches to an OverlayDB (class in the same module) context manager that wraps the main pymongo.database.Database singleton.
• We can add stronger validation there. We will also want to refactor nmdc_runtime.api.core.metadata._validate_changesheet to dispatch to an OverlayDB so that the stronger validation covers changesheet submissions as well.
• I think it also makes sense to add a materialized-from-mongo SPARQL view specifically to aid with internal validation checks, separate from its potential external utility as an exposed endpoint (and I recall that improved queryability was another takeaway from the discussion).

@dwinston to draft a PR to add stronger mongo-change validation to the runtime.

Validation and referential integrity design doc: https://excalidraw.com/#json=3-hD1MOoBuabBr9eOJjOU,IrBWUbkD3ZN6Eksj7F3zHA

And roadmap: https://docs.google.com/spreadsheets/d/10UreFu0tnXxjNBaZA0pXvjQ9ZpLhQwp5AhogEnrLVTU/edit?usp=sharing

@eecavanna @dwinston Notebooks have been added in #521

These are the relevant entrypoints that need referential integrity added:

Sprint 1

• POST /metadata/json:validate
• POST /metadata/json:submit
• POST /workflows/workflow_executions

Sprint 2

• POST /metadata/changesheets:submit
• POST /metadata/changesheets:validate
• POST /queries:run

Sprint 3

• dagster job: apply_metadata_in

@dwinston what is the num of seconds POST /metadata/json:validate or POST /metadata/json:submit will wait before timing out?

These are the relevant entrypoints that need referential integrity added:

Sprint 1

* [ ]  `POST /metadata/json:validate`[ ]  `POST /metadata/json:submit`[ ]  `POST /workflows/workflow_executions`

Sprint 2

* [ ]  `POST /metadata/changesheets:submit`[ ]  `POST /metadata/changesheets:validate`[ ]  `POST /queries:run`

Sprint 3

* [ ]  dagster job: `apply_metadata_in`

@aclum this is our current working timeline, with Sprint 1 being now through the team retreat, and Sprint 2 being the two weeks following the retreat

I would like confirmation that there are no more false positives with this implementation, especially b/c there has been drift from what was done in the notebooks, before this check is added to existing endpoints. Would it be possible to run the code that will be implemented on the existing records in mongo? The current expected results is that 33 records mention an id which does not exist.