Non-RGB multichannels can be 3

[aeisenbarth]

In pull request #115 for issue #66, a logic was implemented that converts multichannels to RGB if they are three.

In our use case, we have non-RGB channels, that can be 2, 4 or sometimes 3. The problem is that independent channels have very different value ranges and are not necessarily normalized, so that one RGB component is dominating. Furthermore, the visualization does not represent physical colors and can be misleading to the viewer.

Is there a way to set up the SpatialData object to explicitly use colors from the metadata, and avoid interpretation as RGB?

I think Napari's logic is that c must be 3 and additionally the order provided by the user must be yxc to interpret an image as RGB.

[melonora]

Hi @aeisenbarth, at the moment it is the case that in case of 3 or 4 channels it is interpreted as RGB. I will implement an RGB argument similar to napari which should support your usecase.

[aeisenbarth]

I also found that in _get_affine_for_images_labels (on branch spatialdata), we could also inspect channel names, if these are always available for true RGB images:

{"r", "g", "b"} <= set(element["scale0"].coords.indexes["c"]) # for MultiscaleSpatialImage

[melonora]

These are not always available for true RGB images as far as I know. @LucaMarconato could you pitch in?

[LucaMarconato]

Thanks @aeisenbarth for reporting. I think that a quick workaround would be to implement the RGB argument mentioned by @melonora to mimic the napari's functionality. I also think that the clean solution would be to rely exclusively on channel name and ensure that they are always present when needed by:

1. add an argument rgb: bool = None in ImageModel.parse() to give a convenience function to the user to set the channel names. The behavior could be to set the names to r, g, b based on the number of channels when rgb is None or is True, but also to give a printed log to the user when rgb is None.
2. modifying the readers of spatialdata-io to use the rgb=True parameter when calling ImageModel.parse().

[aeisenbarth]

I could do a PR for this.

I didn't find ImageModel, I assume Image2DModel (and 3D). The parse function is implemented in RasterSchema, so instead of copying the full signature to Image2DModel just to add one argument, I could add the rgb argument in RasterSchema (but it is also used by labels) or even better in to_spatial_image because there is also c_coord.

If rgb is None and there are 3 or 4 channels and c_coords is not explicitly passed, I would set it to True and log it. Then if rgb is True, I would set c_coords to ["r", "g", "b"] (or "a").
In napari-spatialdata, change the RGB detection based on names in c_coords.

The behavior would be unchanged for rgb=None, I don't know the formats in spatialdata-io well enough to know whether its right to interprete them as RGB?

[melonora]

some like the HE in visium would be RGB

[aeisenbarth]

Ok, so I propose now four PRs: spatial-image/spatial-image#21, scverse/spatialdata#342, napari-spatialdata#153, scverse/spatialdata-io#78

[LucaMarconato]

Thanks a lot for the contribution @aeisenbarth, amazing work! I have merged the PR in spatialdata, but before the others I would wait for the one in spatial-image as otherwise it would probably lead to problems. But looks really good to me!

[aeisenbarth]

Images are still bright red.

Can someone please have a look at the PRs? The issue will continue to persist as long as not all of them are merged.
So far, only one is merged and these are still open:

• Add rgb argument to to_spatial_image spatial-image/spatial-image#21
• Change RGB detection to be based on channel names #153
• Set rgb argument to auto-detect whether image channels are RGB spatialdata-io#78

spatialdata-plot also has this problem, I'll have to check whether it also needs a PR.

[melonora]

yes we are at a hackathon currently, sorry for it taking so long! I asked @thewtex whether the PR in spatial-image can be merged.

[thewtex]

I asked @thewtex whether the PR in spatial-image can be merged.

Yes! Done and released.

[melonora]

Thanks @thewtex !

[LucaMarconato]

Thanks! Now the only blocker is this issue: scverse/spatialdata#395, I will try to address it soon.

[melonora]

I will take it up this evening

[LucaMarconato]

Thanksss! I'd expect that change to break many parts across all the repos, we can also split the work. Let's sync via Zulip.

[melonora]

@aeisenbarth just as an update, I opened scverse/spatialdata#587 to change to the new version of SpatialImage that includes your PR there. Waiting for it to be reviewed before continueing.

[LucaMarconato]

@aeisenbarth this issue can be closed right?

[LucaMarconato]

Tested, it works. Thanks @aeisenbarth @melonora.