Sort keys in metadata .json files

[ieivanov]

Keys in the metadata dump files used to be unordered, making them difficult to browse visually. This PR sorts the keys, resulting in metadata like this:

{
    "0/0/0": {
        "Axes": {
            "z": 0
        },
        "Binning": "2",
        "BitDepth": 12,
        "CameraChannelIndex": 0,
        "Channel": "Default",
        "ChannelIndex": 0,
        "Exposure": 100,
        "Frame": 0,
        "FrameIndex": 0,
        "Height": 1024,
        "PixelSizeAffine": "0.0;0.0;0.0;0.0;0.0;0.0",
        "PixelSizeUm": 0,
        "PixelSize_um": 0,
        "PixelType": "GRAY16",
        "Position": "Default",
        "PositionIndex": 0,
        "ROI": "0-0-1224-1024",
        "Slice": 0,
        "SliceIndex": 0,
        "Time": "2024-02-15 14:08:53 -",
        "Width": 1224,
        "ZPosition_um_Intended": -2,
        "AP Galvo-DA Device": "TS2_DAC03",
        "AP Galvo-Description": "ZStage controlled with voltage provided by a DA board",
        "AP Galvo-Name": "DA Z Stage",
        "AP Galvo-Position": "0.0000",
        "AP Galvo-Stage High Position(um)": "156.5000",
        "AP Galvo-Stage High Voltage": "5.0000",
        "AP Galvo-Stage Low Position(um)": "-156.5000",
        "AP Galvo-Stage Low Voltage": "-5.0000",
        "Blackfly BFP-ADC Bit Depth": "Bit12",
        "Blackfly BFP-Binning": "1",
        "Blackfly BFP-Black Level": "2.0000",
...

One problem is that ElapsedTime-ms is not sorted near the top, but I think that's not a big problem.

[ziw-liu]

Can you measure the overhead of doing this? I've been testing the ometiff-uapi branch and observed that converting metadata is already taking more time than converting the images themselves, as each JSON file (for each FOV) could be 300 MB or more.

[ieivanov]

Oh interesting - do you have infrastructure for testing this? We can't do without the metadata, but I agree that sorting it is optional.

[ziw-liu]

An easy way is to pick a large dataset and measure the time needed for sorting.

[ieivanov]

Not sure what happened to this PR - maybe it needs to be rebased?

I tested the conversion as you suggested. I picked this ~150 GB dataset:

/hpc/instruments/cm.mantis/2023_09_21_OpenCell_targets/opencell_hcs_1/opencell_hcs_lightsheet_1/

Format:			 ndtiff
FOVs:			 93
FOV shape:		 T=1, C=2, Z=593, Y=300, X=2048
Channel names:		 ['GFP EX488 EM525-45', 'mCherry EX561 EM600-37']
(Z, Y, X) scale (um):	 (0.313, 0.1161, 0.1161)

And copied it to /tmp to avoid some of the io overhead. It finished converting in 383 seconds using the unified_api branch and in 385 seconds using this branch. I think that's worthwhile. I confirmed that the metadata keys are unsorted in one version and sorted in the other.

[ieivanov]

@ziw-liu I think this PR is ready now