feat: File management for `fetch-binary` and `fetch-science` functions #17

mfacchinelli · 2024-08-09T09:39:15Z

Adds OutputManager class to manage output of files to requested output folder and (optionally) database. Currently only supported for fetch-science and fetch-binary methods.

Plenty of unit tests for new classes. See new coverage report in GitHub Actions.

BREAKING: some tests no longer pass

…for mocks

…e with `version=latest` argument

…oid issue with `version=latest` argument" This reverts commit bddeea8.

…e with `version=latest` argument

…oid issue with `version=latest` argument" This reverts commit bddeea8.

commit ee06eb9 Author: Michele Facchinelli <[email protected]> Date: Fri Aug 16 09:53:09 2024 +0000 fix: add packages permission commit d0bac86 Author: Michele Facchinelli <[email protected]> Date: Fri Aug 16 09:47:17 2024 +0000 fix: add checks permissions commit 80b3b92 Author: Michele Facchinelli <[email protected]> Date: Fri Aug 16 09:44:44 2024 +0000 fix: permissions for CI actions

…ndon/imap-pipeline-core into feat/hk_manage

mfacchinelli · 2024-08-22T16:05:10Z

@alastairtree, updated to version 0.9.0 of imap-data-access and tests successfully pass!

alastairtree

Looks great, thank you and sorry it took me ages. A couple of minor things but nothing major so merge when you are happy.

.github/workflows/ci.yml

pyproject.toml

alastairtree · 2024-08-16T10:53:05Z

...imap_db/migrations/versions/2024_08_09-669111c45c37_added_version_hash_date_and_software_.py

+
+def upgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column("files", sa.Column("version", sa.Integer(), nullable=False))


This upgrade will probably fail if the database contains any existing rows because you cannot add a non-nullable column without a default value to an existing row. Fine for this since we do not care about the data but just flagging this will need us to drop and recreate the data when we deploy this. A good practise is to always provide a default value when adding columns to existing tables, or to add as nullable, populate the data and then update it and make it non nullable.

That makes sense. I did not consider that!

Is this any better?

(sorry - will push changes soon!)

alastairtree · 2024-08-29T14:46:03Z

pyproject.toml

 docker = "^7.1.0"
 testcontainers = "^4.7.2"
+wiremock = {git = "https://github.com/ImperialCollegeLondon/python-wiremock.git", rev = "fix-test-containers-on-windows"}


I just chased on the wiremock Slack to try and get this PR bugfix we need merged but no luck so far.

alastairtree · 2024-08-29T14:50:31Z

src/imap_mag/DB.py

@@ -40,3 +68,61 @@ def insert_files(self, files: list[File]):
            raise e
        finally:
            session.close()
+
+
+class DatabaseOutputManager(IOutputManager):


I don't really like the name "output manager" because i can't really tell what it is but not sure if there is anything better. Basically just talking to myself here.

tests/test_database.py

tests/test_fetchBinary.py

tests/test_fetchScience.py

tests/test_outputManager.py

mfacchinelli added 30 commits August 6, 2024 12:52

feat: add class to manage outputs

fe26c60

refactor: version is an int not a str

231e0c2

refactor: replace custom logic with OutputManager

3741a1f

refactor: convert "pattern" methods to separate "provider" class

fdcd9ba

build: single source app version

8ca6ca3

style: replace - with _ in file names

b4087a2

task: add support for prefix and level in output metadata

da4c44b

feat: add more variables in database

99773b5

feat!: use output manager to write to output and database

7ede655

BREAKING: some tests no longer pass

dev: add SQLALCHEMY_URL to dev container environment

76afe1e

test(fix): fix output manager and calibration tests

f6927b1

fix: do not export to database in tests

3eb23f1

fix: HK and science tests

332cb24

fix: attempt to fix issue with database initialization

de14a6f

build: add pytest-mock to dev dependencies

fdb5882

test: add tests for FetchBinary class

8cbe6be

test: add unit tests for FetchScience

527bcc6

chore: update lock file

bc374d3

task: add hash to database and make path a unique constraint

c7c274f

feat: update database handler to check for error cases

e95ac5f

fix: remove logging of URL

87dce32

test: move create_test_file to utility and use assert_not_called …

050d8b4

…for mocks

test: add some tests for database output manager

fa82965

test: add coverage for database error

84770f3

Merge remote-tracking branch 'origin/main' into feat/hk_manage

3ae9fc4

build: change imap-mag version to initial version

739765b

test(fix): remove commented out text

94831b2

fix: temporarely use custom version of imap-data-access to avoid issu…

bddeea8

…e with `version=latest` argument

Revert "fix: temporarely use custom version of imap-data-access to av…

3239efb

…oid issue with `version=latest` argument" This reverts commit bddeea8.

build: add coverage report to CI action

164b61b

mfacchinelli added 24 commits August 16, 2024 10:34

test: add some tests for database output manager

c22489b

test: add coverage for database error

5dbf6dd

build: change imap-mag version to initial version

490a515

test(fix): remove commented out text

d4ac011

fix: temporarely use custom version of imap-data-access to avoid issu…

e4b545d

…e with `version=latest` argument

Revert "fix: temporarely use custom version of imap-data-access to av…

7114e0c

…oid issue with `version=latest` argument" This reverts commit bddeea8.

build: add coverage report to CI action

f786a87

test: skip failing test due to bug in imap-data-access

e9694c9

build(fix): use version instead of branch

3a82f36

test(fix): name of classes under test in comment

6c368f4

test: add more tests to increase coverage

01360fb

test(fix): actually use convertToDatetime

0a0a455

fix: CI action failure, packaging failure, test failure

91f0c34

fix: more fixes to permissions for action and packaging

ea59ccb

Merge branch 'feat/hk_manage' of https://github.com/ImperialCollegeLo…

921768a

…ndon/imap-pipeline-core into feat/hk_manage

fix: merge issues

9c86b45

build(fix): use existing permissions

36d8ecf

fix: install git on Docker image

6218f99

try: use git under wine

41decd0

try: force Docker run action to finish

844213b

try: more tests to make git work on Docker

a041109

fix: give up on git-based imap-data-access and skip failing test

37fa416

fix: update to latest imap-data-access and unfilter test

1bcb16a

alastairtree approved these changes Aug 29, 2024

View reviewed changes

mfacchinelli added 4 commits August 30, 2024 13:46

fix: partially address Alastair's comments

55e0838

fix: avoid circular dependency

49a8d22

fix: further reduce duplication with hash

003cf5a

task: multiple changes

df8f4ab

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: File management for `fetch-binary` and `fetch-science` functions #17

feat: File management for `fetch-binary` and `fetch-science` functions #17

mfacchinelli commented Aug 9, 2024 •

edited

Loading

mfacchinelli commented Aug 22, 2024

alastairtree left a comment

alastairtree Aug 16, 2024

mfacchinelli Aug 30, 2024

mfacchinelli Aug 30, 2024

alastairtree Aug 29, 2024

alastairtree Aug 29, 2024

feat: File management for fetch-binary and fetch-science functions #17

Are you sure you want to change the base?

feat: File management for fetch-binary and fetch-science functions #17

Conversation

mfacchinelli commented Aug 9, 2024 • edited Loading

mfacchinelli commented Aug 22, 2024

alastairtree left a comment

Choose a reason for hiding this comment

alastairtree Aug 16, 2024

Choose a reason for hiding this comment

mfacchinelli Aug 30, 2024

Choose a reason for hiding this comment

mfacchinelli Aug 30, 2024

Choose a reason for hiding this comment

alastairtree Aug 29, 2024

Choose a reason for hiding this comment

alastairtree Aug 29, 2024

Choose a reason for hiding this comment

feat: File management for `fetch-binary` and `fetch-science` functions #17

feat: File management for `fetch-binary` and `fetch-science` functions #17

mfacchinelli commented Aug 9, 2024 •

edited

Loading