FHIR field naming conformance #201

dogversioning · 2024-03-29T18:46:30Z

This PR makes the following changes:

Column names now adhere more closely to FHIR naming conventions
core__encounter_type table removed, as it is now 100% redundant with the encounter table
core__count_encounter_type table renamed to core__count_encounter_all_types. Same for the month striated version.
Added information on exceptions to FHIR naming to the core study documentation.

Checklist

Consider if documentation (like in docs/) needs to be updated
Consider if tests should be added
Update template repo if there are changes to study configuration

dogversioning · 2024-03-29T19:03:07Z

cumulus_library/studies/core/patient_extensions.sql

this file was an older reference sql file that just never got cleaned up.

dogversioning · 2024-03-29T19:05:09Z

docs/core-study-details.md

+Unless otherwise noted, column names correspond to a FHIR path relative to the resource
+table in question, with resources delineated by underscores. So, 'category_code' in the 
+Condition table would correspond to the code element inside of the category element of
+the Condition resource.
+
+Common exceptions to this rule:
+
+- `cnt` is a Cumulus specific notation for a column that contains counts of resources.
+- `age_at_visit` is calculated from a Patient's birthDate and the period.start.day of an Encounter
+- `postalcode_3` is calculated from a Patient's address.postalCode
+- The Encounter resource includes several elements from the referenced Patient (`gender`, `postalcode_3`,
+  and the US Core race and ethnicity extensions) that are commonly used in informatics analysis
+
+The core tables include all FHIR required/must support fields noted in the
+[FHIR resource profiles(http://hl7.org/fhir/us/core/STU4/artifacts.html#structures-resource-profiles).
+Additionally, there are fields that are useful to informatics analysis that are commonly available
+from EHRs, but are not guaranteed to be populated, so consult with your research partners if
+you are authoring a study using some of these data elements.


@comorbidity drawing your attention to this text - not quite in db metadata, but would this work for external consumers, i.e. a DPH?

@dogversioning Which part are you referring to? If you mean ZipCode with 3 digits, that's a HIPAA constraint.
There has been an outstanding task to look at "census track data" but that has not yet been completed. ZipCode with 5 digits to my knowledge we do not have PHI clearance for, playing it safe here.

@comorbidity this was more directed at answering the question that source metadata was trying to answer, i.e. 'i see this field name in the dashboard, where does it come from'?

dogversioning · 2024-03-29T19:06:02Z

docs/core-study-details.md

@comorbidity this might be the easiest place to note the changes this PR makes to the column names from the perspective of a study author.

age_at_visit is a calculated view very, very commonly used in population health studies that involve patients over any timeperiod (or even patients at a single point in time).

this comment was more aimed at the general cutover to FHIR-like column names, not age at visit specifically.

dogversioning · 2024-03-29T19:07:10Z

tests/conftest.py

@@ -151,6 +159,52 @@ def update_nested_obj(id_path, obj, i):
                    f.write(json.dumps(row, default=str) + "\n")


+# Debugging aids


I threw these in here out of convenience, but could be convinced to put them into a test_utils if we feel strongly about that.

This is fine!

dogversioning · 2024-03-29T19:07:46Z

tests/test_core.py

        ("core__medication"),
        ("core__medicationrequest"),
        ("core__observation"),
        ("core__observation_lab"),
+        ("core__patient"),


this has been missing for awhile! 😬

Do you know the reason that core__patient has been removed?
FHIR Patient object exists. I think the reason was because of patient AGE or DOB, which we just truncated down to YYYY. Otherwise, I think this table "should" exist?

DPH active Cumulus studies have all been "over time" and thus relied more on core__encounter, but I can easily imagine a user (not me) being confused by the lack of a core__patient table.

This was removed, or not added, only from our unit tests, as an oversight. I have not done the work to trace this back

dogversioning · 2024-03-29T19:08:44Z

tests/test_core.py

+    except Exception as e:
+        conftest.debug_diff_tables(cols, table_rows, ref_table, pos=0)
+        raise e


i'm leaving this in permanently for error states because it is incredibly useful for troubleshooting data problems.

Fail Fast, love it!

dogversioning · 2024-03-29T19:31:40Z

cumulus_library/studies/core/study_period.sql

I wasn't as agressive with some of the renaming in this file, for a few reasons:

we already have a ticket to do some cleanup work (core__study_period defines "ed_note" but this should not be present #198 ) that i'd like to keep seperate from the many, many moving parts here

this straddles the border between the FHIR standard and the study interfaces currently, and i'm not quite ready to break that until i at least look at cutting the covid study over.

I sort of think we might want some infrastructure around this that studies could use.

mikix · 2024-04-01T12:49:18Z

cumulus_library/studies/core/core_templates/encounter.sql.jinja

+    LEFT JOIN core__encounter_dn_reasonCode AS edr ON e.id = edr.id
+    LEFT JOIN core__encounter_dn_dischargeDisposition AS edd ON e.id = edd.id


Hah, mixed case tables are just as fine as mixed case fields (casing is ignored), but it does hit me different - I'm less used to it.

i don't either - this might be a ctrl+f artifact, lemme touch this one.

mikix · 2024-04-01T13:07:13Z

docs/core-study-details.md

+Condition table would correspond to the code element inside of the category element of
+the Condition resource.
+
+Common exceptions to this rule:


Other things you could mention, but you don't need to, just thoughts:

DateTime and Period fields are usually chopped up.

Often References are represented with a toplevel xxx_ref field, and sometimes that field lives alongside a simple ID version of same.

mikix · 2024-04-01T13:10:38Z

tests/test_discovery.py

@@ -42,7 +42,7 @@ def test_discovery(tmp_path):
    )
    db = databases.DuckDatabaseBackend(f"{tmp_path}/duck.db")
    cursor = db.cursor()
-    table_rows = conftest.get_sorted_table_data(cursor, "discovery__code_sources")
+    table_rows, cols = conftest.get_sorted_table_data(cursor, "discovery__code_sources")


cols isn't used anywhere, yeah? Does ruff not complain about it? That feels like a missing check to my brain. Usually I'd prefix it with a sunder, like _cols to avoid warnings / signal to other devs that they can ignore it.

(Do you use the result anywhere? Is it just for manual debugging?)

It's in the except statement on line 68.

mikix · 2024-04-01T13:13:25Z

tests/test_core.py

+            assert row in ref_table
+    except Exception as e:
+        conftest.debug_diff_tables(cols, table_rows, ref_table, pos=0)
+        raise e


nit: You can simply say raise and it will re-raise the current exception. (which is nice, because then you don't even have to give the exception a name)

huh! i had no idea.

mikix · 2024-04-01T13:15:56Z

tests/conftest.py

@@ -151,6 +159,52 @@ def update_nested_obj(id_path, obj, i):
                    f.write(json.dumps(row, default=str) + "\n")


+# Debugging aids


This is fine!

FHIR field naming conformance

732a1be

dogversioning force-pushed the mg/FHIR_naming branch from 421629d to 732a1be Compare March 29, 2024 18:57

dogversioning commented Mar 29, 2024

View reviewed changes

mikix approved these changes Apr 1, 2024

View reviewed changes

dogversioning added 2 commits April 1, 2024 09:38

Light docs tweaks, styling nits

9d15484

regression

7875181

dogversioning merged commit bf37eec into main Apr 1, 2024
3 checks passed

dogversioning deleted the mg/FHIR_naming branch April 1, 2024 14:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FHIR field naming conformance #201

FHIR field naming conformance #201

dogversioning commented Mar 29, 2024

dogversioning Mar 29, 2024

dogversioning Mar 29, 2024

comorbidity Mar 29, 2024

dogversioning Mar 29, 2024

dogversioning Mar 29, 2024

comorbidity Mar 29, 2024

dogversioning Mar 29, 2024

dogversioning Mar 29, 2024

mikix Apr 1, 2024

dogversioning Mar 29, 2024

comorbidity Mar 29, 2024

comorbidity Mar 29, 2024

dogversioning Mar 29, 2024

dogversioning Mar 29, 2024

comorbidity Mar 29, 2024

dogversioning Mar 29, 2024

mikix Apr 1, 2024

dogversioning Apr 1, 2024

mikix Apr 1, 2024

mikix Apr 1, 2024

dogversioning Apr 1, 2024

mikix Apr 1, 2024

dogversioning Apr 1, 2024

mikix Apr 1, 2024

		@@ -151,6 +159,52 @@ def update_nested_obj(id_path, obj, i):
		f.write(json.dumps(row, default=str) + "\n")


		# Debugging aids

		LEFT JOIN core__encounter_dn_reasonCode AS edr ON e.id = edr.id
		LEFT JOIN core__encounter_dn_dischargeDisposition AS edd ON e.id = edd.id

FHIR field naming conformance #201

FHIR field naming conformance #201

Conversation

dogversioning commented Mar 29, 2024

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment