Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Run table and unit table #124

Merged
merged 40 commits into from
Nov 17, 2024
Merged

Run table and unit table #124

merged 40 commits into from
Nov 17, 2024

Conversation

Vedant1
Copy link
Collaborator

@Vedant1 Vedant1 commented Nov 14, 2024

  • create unified unit table
  • optional second function to set file reader data dictionary to metadata ordered dictionary
  • updated put artifact to only commit changes if all data in a certain workflow is stable
  • updated er diagram writer to create dot object in python instead of external dot file (no linux command necessary)
  • created generic text file reader

Vedant P Iyer added 16 commits November 7, 2024 01:01
Moved erd test function to test_file_writer and deleted test_plugin.p…
7176760 
…y/CI file
created test functions for schema, yaml, toml readers
f874bd5
updated file writer CI test with extra package installs for erd test
8544983
created a unified dsi_units table for all ingested data and updated h…
56e9d15 
…andling for writers
Updated other files to handle unified units table and separate back-w…
17c2734 
…rite/read in core calls
Created run table to store metadata for every workflow run and handle…
10d0e54 
…d unified unit table in sqlite reader/writer classes
updated backend actions and split back-write and read actions
7eb97dd
created set_schema2 which sets file reader data dict to metadata dict
fdcb7a3
added inspect artifact handler function example workflow
5a68c19
updated file writers to handle unified unit table
bd205d9
updated schema, yaml, toml readers to use set_schema2 instead of old …
f638ee1 
…metadata functions
Only commit db insert if all data in workflow is stable/non repetitiv…
e4229f8 
…e, and updated inspect artifact function to generate Jupyter notebook
only execute sql statements if no insertion error else rollback all p…
bd3b2f8 
…revious
customized print error if duplicate/error data ingested to backend
592a218
updated er diagram writer to generate dot as python object and no ext…
4c3c8d6 
…ernal file generated
created generic text file reader -- assumes only one table in data so…
b41422d 
…urce
@jpulidojr jpulidojr self-requested a review November 14, 2024 23:04
Vedant P Iyer and others added 13 commits November 15, 2024 10:15
merged changes with main branch
e1afd9b
combined and tested sqlite read/write class
0b49b3e
Updated other files to handle unified units table and separate back-w…
6d458ef 
…rite/read in core calls
updated primary key handling in schema reader
dec406c
Created run table to store metadata for every workflow run and handle…
750931b 
…d unified unit table in sqlite reader/writer classes
updated backend actions and split back-write and read actions
0411470
Only commit db insert if all data in workflow is stable/non repetitiv…
3e97340 
…e, and updated inspect artifact function to generate Jupyter notebook
changed coreterminal to merge to main
3536990
updated sqlite to merge to main
0a149d6
@Vedant1
Merge branch 'main' into runTable_and_unitTable
c5126a5
added graphviz to pip install in CI file
402477b
updated tests to reflect units table in collections now
5895856
updated test file reader again
23252d1
Vedant P Iyer and others added 11 commits November 15, 2024 15:12
jpulidojr
jpulidojr approved these changes Nov 17, 2024
View reviewed changes
Copy link
Collaborator

@jpulidojr jpulidojr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested open changes, pushing upstream to test internally

@jpulidojr jpulidojr merged commit e7b0c79 into main Nov 17, 2024
3 checks passed
@jpulidojr jpulidojr deleted the runTable_and_unitTable branch November 17, 2024 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jpulidojr jpulidojr jpulidojr approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Vedant1 @jpulidojr