Ngio projection

[lorenzocerrone]

• Add ngio to dependecies
• Refactor projection task to use ngio

Checklist before merging

• I added an appropriate entry to CHANGELOG.md

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
fractal_tasks_core/tables
v1.py
fractal_tasks_core/tasks
projection.py					29, 77
Project Total

This report was generated by python-coverage-comment-action

[jluethi]

Is the plan to generally transition to using ngio logger in any task that uses ngio?

[tcompa]

Is the plan to generally transition to using ngio logger in any task that uses ngio?

In my opinion we should not do this, and we should rather keep two different loggers - ref #867

[jluethi]

To discuss with @tcompa: When does run_fractal_task use the logger name? Is the ngio_logger also a reasonable choice here?

[jluethi]

We need to set roi.z = 0, to avoid getting IndexErrors if roi.z was not 0 before. It should be 0 before in all our cases though

[tcompa]

Follow-up: why is roi.z even needed here? That is, why not setting roi.z_length = 1?
Naively, I expect roi.z to be some kind of coordinate, which should not matter when assigning a value to a length.

[jluethi]

This is great @lorenzocerrone !

A few brief questions: Have you tried running this projection task on a zyx dataset (no channel or time dimension)? Could we easily add a synthetic test case for that using an ngio generated tiny zyx OME-Zarr. That then closes #840

Also, would we expect this task to already work with time-data, e.g. an tczyx dataset?

How much we should test in tasks-core?
We test separately in tasks core only when there is task logic specifically to those cases

Can we add an option to copy over additional tables (e.g. user can deactivate that, but we copy non-roi tables by default). We'd keep them & ignore edge-cases (like missing label images) for now. TODO Joel: provide example dataset of condition tables for this.
Approach: First copy them as they are (by adding a include_table flag to the derive function or by adding a copy_tables function). Then modify the ROI tables in place in the new Zarr => let's go with a copy_tables function

TODO for myself:

• Check how we need to provide the metadata update for Review image-plate attribute of MIP task #841. Likely a change to the image_list_update_dict
• Run example 01 with this branch, to check for Update zenodo dataset, setting channel.active to True #823

[tcompa]

Unclear to me: are we planning to keep support for python3.9 in fractal-tasks-core? If so, is there a specific reason for removing from the CI?

[jluethi]

Yes, we're planning to drop support for Python 3.9 in fractal-tasks-core

[jluethi]

Let's also drop it in pyproject.toml

[tcompa]

I am a bit confused by this approach. Any clarification is welcome.

For instance:

1. Why do we use Image, if this function always receives NgffImage objects? Now I got it, I was mixing up original_ngff_image and original_image
2. Independently on point 1, does importing Image have any side-effect? If not, is there a special reason for hiding it behind TYPE_CHECKING or is it "only" as a best practice? (e.g. because the import is slow, or if there are circular imports)

[jluethi]

TBD: Coming in a separate PR if we want this style across tasks core

[tcompa]

Minor readability detail:

table = something(roi_table) was very confusing at a first read, before checking what original_ngff_image.tables.list( actually does. I'd suggest renaming roi_table into e.g. roi_table_name or table_name.

[tcompa]

Naive question by someone who is not involved in ngio development/integration: where does this feature fit best?

Would it make sense to have something like

ngio.copy_tables(original_ngff_image, new_ngff_image, project_z=True)

or at least

ngio.copy_tables(original_ngff_image, new_ngff_image)
# and then set `z_length` manually

or would it be just additional complexity?

[tcompa]

Thanks @lorenzocerrone!
I added a few small comments, but the pattern is often the same (I am no expert in ngio, and then questions pop up frequently independently on the actual PR). I think we are at a stage where it would be useful to briefly discuss this live.