Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Actions/builders converted to running via config & manifest #256

Merged
merged 5 commits into from
Jul 2, 2024

Conversation

dogversioning
Copy link
Contributor

This PR makes the following changes:

Checklist

  • Consider if documentation (like in docs/) needs to be updated
  • Consider if tests should be added
  • Update template repo if there are changes to study configuration

Copy link

github-actions bot commented Jul 1, 2024

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines Covered Coverage Threshold Status
1930 1881 97% 90% 🟢

New Files

No new covered files...

Modified Files

File Coverage Status
cumulus_library/init.py 100% 🟢
cumulus_library/actions/builder.py 96% 🟢
cumulus_library/actions/cleaner.py 97% 🟢
cumulus_library/actions/exporter.py 97% 🟢
cumulus_library/actions/file_generator.py 98% 🟢
cumulus_library/actions/importer.py 100% 🟢
cumulus_library/base_table_builder.py 100% 🟢
cumulus_library/base_utils.py 97% 🟢
cumulus_library/cli.py 96% 🟢
cumulus_library/databases.py 95% 🟢
cumulus_library/log_utils.py 100% 🟢
cumulus_library/protected_table_builder.py 100% 🟢
cumulus_library/statistics/counts.py 100% 🟢
cumulus_library/statistics/psm.py 96% 🟢
cumulus_library/studies/core/builder_prereq_tables.py 100% 🟢
cumulus_library/studies/core/count_core.py 96% 🟢
cumulus_library/studies/vocab/vocab_icd_builder.py 100% 🟢
TOTAL 98% 🟢

updated for commit: a74e72d by action🐍

Copy link
Contributor

@mikix mikix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not do a very close read of all the renaming stuff - but looks good!

cumulus_library/actions/builder.py Outdated Show resolved Hide resolved
Comment on lines +291 to +292
if args.get("db_type") == "duckdb":
schema = "main"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels like a little bit of duckdb specific logic escaping from database.py, but it's not upsetting.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh thank you, i was like 'i need to mention something' and couldn't remember what it was.

If we want to get this cat back in the bag, we'll need to undo the overloading we're currently doing, where raw schema name and duckdb file path are going into the same location. So we'd basically add one new CLI arg and then split how that logic is handled in some places.

if we want to do this, i can create a ticket for it, but i don't want to do it on this PR given the density.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, sounds like you are talking a bigger change than I was - I'm not 100% on what your vision there is.

I was talking more like the DuckDB database backend simply ignoring the input schema arg and using main or a method like DatabaseBackend.decide_schema_name(args) or something - just anything to put this tiny business logic under the umbrella of the DuckDB-specific code. Ideally in my mind, nothing outside of databases.py every has to know DuckDB exists.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per offline discussion, leaving this as is for now, but logging a ticket: #258

"SELECT schema_name FROM information_schema.schemata"
).fetchall()
if (schema_name,) not in schemas:
self.connection.sql(f"CREATE SCHEMA {schema_name}")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rude for SQL to not have CREATE SCHEMA IF NOT EXISTS

cumulus_library/protected_table_builder.py Outdated Show resolved Hide resolved
@dogversioning dogversioning merged commit f4895f5 into main Jul 2, 2024
3 checks passed
@dogversioning dogversioning deleted the mg/config_manifest branch July 2, 2024 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants