3D training with ValueError(msg)

[mccruz07]

Hi @kevinjohncutler,

I'm trying to train a new model using your new 3D implementation, but I'm having the follow error:

python -m omnipose --train --use_gpu --dir ./omni/omnipose_training --mask_filter _masks --n_epochs 4000 --pretrained_model None --learning_rate 0.1 --save_every 50 --save_each --verbose --dim 3 --RAdam --batch_size 4 --diameter 0

2022-11-29 13:26:02,817 [INFO] WRITING LOG OUTPUT TO /Users/mcruz/.cellpose/run.log
log file /Users/mcruz/.cellpose/run.log
2022-11-29 13:26:02,820 [INFO] ** TORCH GPU version installed and working. **
2022-11-29 13:26:02,820 [INFO] >>>> using GPU
Omnipose enabled. See Omnipose repo for licencing details.
2022-11-29 13:26:02,820 [INFO] Training omni model. Setting nclasses=4, RAdam=True
2022-11-29 13:26:02,824 [INFO] not all flows are present, will run flow generation for all images
2022-11-29 13:26:02,826 [INFO] training from scratch
2022-11-29 13:26:02,826 [INFO] median diameter set to 0 => no rescaling during training
2022-11-29 13:26:02,996 [INFO] No precomuting flows with Omnipose. Computed during training.
2022-11-29 13:26:03,021 [INFO] >>> Using RAdam optimizer
2022-11-29 13:26:03,021 [INFO] >>>> training network with 120 channel input <<<<
2022-11-29 13:26:03,021 [INFO] >>>> LR: 0.10000, batch_size: 4, weight_decay: 0.00001
2022-11-29 13:26:03,021 [INFO] >>>> ntrain = 5
2022-11-29 13:26:03,021 [INFO] >>>> nimg_per_epoch = 5
Traceback (most recent call last):
  File "/Users/mcruz/opt/anaconda3/envs/omnipose/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/mcruz/opt/anaconda3/envs/omnipose/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/mcruz/opt/anaconda3/envs/omnipose/lib/python3.10/site-packages/omnipose/__main__.py", line 3, in <module>
    main(omni_CLI=True)
  File "/Users/mcruz/opt/anaconda3/envs/omnipose/lib/python3.10/site-packages/cellpose/__main__.py", line 476, in main
    cpmodel_path = model.train(images, labels, train_files=image_names,
  File "/Users/mcruz/opt/anaconda3/envs/omnipose/lib/python3.10/site-packages/cellpose/models.py", line 1045, in train
    model_path = self._train_net(train_data, train_labels,
  File "/Users/mcruz/opt/anaconda3/envs/omnipose/lib/python3.10/site-packages/cellpose/core.py", line 1001, in _train_net
    imgi, lbl, scale = transforms.random_rotate_and_resize(
  File "/Users/mcruz/opt/anaconda3/envs/omnipose/lib/python3.10/site-packages/cellpose/transforms.py", line 839, in random_rotate_and_resize
    return omnipose.core.random_rotate_and_resize(X, Y=Y, scale_range=scale_range, gamma_range=gamma_range,
  File "/Users/mcruz/opt/anaconda3/envs/omnipose/lib/python3.10/site-packages/omnipose/core.py", line 1387, in random_rotate_and_resize
    imgi[n], lbl[n], scale[n] = random_crop_warp(img, y, nt, tyx, nchan, scale[n],
  File "/Users/mcruz/opt/anaconda3/envs/omnipose/lib/python3.10/site-packages/omnipose/core.py", line 1543, in random_crop_warp
    I = do_warp(img[k], M, tyx, offset=offset, mode=mode)
  File "/Users/mcruz/opt/anaconda3/envs/omnipose/lib/python3.10/site-packages/omnipose/core.py", line 1606, in do_warp
    return scipy.ndimage.affine_transform(A, np.linalg.inv(M), offset=offset, output_shape=tyx, order=order, mode=mode)
  File "/Users/mcruz/opt/anaconda3/envs/omnipose/lib/python3.10/site-packages/scipy/ndimage/_interpolation.py", line 586, in affine_transform
    raise ValueError(msg)
ValueError: Expected homogeneous transformation matrix with shape (3, 3) for image shape (1, 288), but bottom row was not equal to [0, 0, 1]

[kevinjohncutler]

@mccruz07 I think I have seen others with this error in the past, it came down to incompatible dependencies. Did you install in a fresh conda environment? Can you post your conda list and pip list?

[mccruz07]

Sure, here is the conda list:

> conda list
# packages in environment at /Users/mcruz/opt/anaconda3/envs/omnipose:
#
# Name                    Version                   Build  Channel
aom                       3.5.0                h7ea286d_0    conda-forge
appnope                   0.1.3                    pypi_0    pypi
asttokens                 2.2.0                    pypi_0    pypi
backcall                  0.2.0                    pypi_0    pypi
blosc                     1.21.1               hd414afc_3    conda-forge
brotli                    1.0.9                h1a8c8d9_8    conda-forge
brotli-bin                1.0.9                h1a8c8d9_8    conda-forge
brunsli                   0.1                  h9f76cd9_0    conda-forge
bzip2                     1.0.8                h3422bc3_4    conda-forge
c-ares                    1.18.1               h3422bc3_0    conda-forge
c-blosc2                  2.4.3                h303ed30_0    conda-forge
ca-certificates           2022.9.24            h4653dfc_0    conda-forge
cachetools                5.2.0                    pypi_0    pypi
cellpose-omni             0.7.2                    pypi_0    pypi
certifi                   2022.9.24                pypi_0    pypi
cfitsio                   4.1.0                hd4f5c17_0    conda-forge
charls                    2.3.4                hbdafb3b_0    conda-forge
charset-normalizer        2.1.1                    pypi_0    pypi
colour                    0.1.5                    pypi_0    pypi
comm                      0.1.1                    pypi_0    pypi
contourpy                 1.0.6                    pypi_0    pypi
cycler                    0.11.0                   pypi_0    pypi
darkdetect                0.7.1                    pypi_0    pypi
dav1d                     1.0.0                he4db4b2_1    conda-forge
debugpy                   1.6.4                    pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
edt                       2.3.0                    pypi_0    pypi
entrypoints               0.4                      pypi_0    pypi
executing                 1.2.0                    pypi_0    pypi
fastremap                 1.13.3                   pypi_0    pypi
fonttools                 4.38.0                   pypi_0    pypi
giflib                    5.2.1                h27ca646_2    conda-forge
google-api-core           2.10.2                   pypi_0    pypi
google-auth               2.14.1                   pypi_0    pypi
google-cloud-core         2.3.2                    pypi_0    pypi
google-cloud-storage      2.6.0                    pypi_0    pypi
google-crc32c             1.5.0                    pypi_0    pypi
google-resumable-media    2.4.0                    pypi_0    pypi
googleapis-common-protos  1.57.0                   pypi_0    pypi
idna                      3.4                      pypi_0    pypi
imagecodecs               2022.9.26       py310h24fba19_3    conda-forge
imageio                   2.22.4                   pypi_0    pypi
ipykernel                 6.18.2                   pypi_0    pypi
ipython                   8.7.0                    pypi_0    pypi
jedi                      0.18.2                   pypi_0    pypi
joblib                    1.2.0                    pypi_0    pypi
jpeg                      9e                   he4db4b2_2    conda-forge
jupyter-client            7.4.7                    pypi_0    pypi
jupyter-core              5.1.0                    pypi_0    pypi
jxrlib                    1.1                  h27ca646_2    conda-forge
kiwisolver                1.4.4                    pypi_0    pypi
krb5                      1.19.3               he492e65_0    conda-forge
lcms2                     2.14                 h8193b64_0    conda-forge
lerc                      4.0.0                h9a09cb3_0    conda-forge
libaec                    1.0.6                hbdafb3b_0    conda-forge
libavif                   0.11.1               h3d80962_0    conda-forge
libblas                   3.9.0           16_osxarm64_openblas    conda-forge
libbrotlicommon           1.0.9                h1a8c8d9_8    conda-forge
libbrotlidec              1.0.9                h1a8c8d9_8    conda-forge
libbrotlienc              1.0.9                h1a8c8d9_8    conda-forge
libcblas                  3.9.0           16_osxarm64_openblas    conda-forge
libcurl                   7.86.0               h1c293e1_1    conda-forge
libcxx                    14.0.6               h2692d47_0    conda-forge
libdeflate                1.14                 h1a8c8d9_0    conda-forge
libedit                   3.1.20191231         hc8eb9b7_2    conda-forge
libev                     4.33                 h642e427_1    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libgfortran               5.0.0           11_3_0_hd922786_26    conda-forge
libgfortran5              11.3.0              hdaf2cc0_26    conda-forge
liblapack                 3.9.0           16_osxarm64_openblas    conda-forge
libnghttp2                1.47.0               h519802c_1    conda-forge
libopenblas               0.3.21          openmp_hc731615_3    conda-forge
libpng                    1.6.38               h76d750c_0    conda-forge
libsqlite                 3.39.4               h76d750c_0    conda-forge
libssh2                   1.10.0               h7a5bd25_3    conda-forge
libtiff                   4.4.0                hfa0b094_4    conda-forge
libwebp-base              1.2.4                h57fd34a_0    conda-forge
libzlib                   1.2.13               h03a7124_4    conda-forge
libzopfli                 1.0.3                h9f76cd9_0    conda-forge
llvm-openmp               15.0.4               h7cfbb63_0    conda-forge
llvmlite                  0.39.1                   pypi_0    pypi
lz4-c                     1.9.3                hbdafb3b_1    conda-forge
mahotas                   1.4.13                   pypi_0    pypi
matplotlib                3.6.2                    pypi_0    pypi
matplotlib-inline         0.1.6                    pypi_0    pypi
mgen                      1.2.0                    pypi_0    pypi
mpmath                    1.2.1                    pypi_0    pypi
natsort                   8.2.0                    pypi_0    pypi
ncolor                    1.1.8                    pypi_0    pypi
ncurses                   6.3                  h07bb92c_1    conda-forge
nest-asyncio              1.5.6                    pypi_0    pypi
networkx                  2.8.8                    pypi_0    pypi
numba                     0.56.4                   pypi_0    pypi
numpy                     1.23.4          py310h5d7c261_1    conda-forge
omnipose                  0.3.4.dev9+g88ccc8d          pypi_0    pypi
omnipose-theme            1.1.2                    pypi_0    pypi
opencv-python-headless    4.6.0.66                 pypi_0    pypi
openjpeg                  2.5.0                h5d4e404_1    conda-forge
openssl                   3.0.7                h03a7124_0    conda-forge
packaging                 21.3                     pypi_0    pypi
parso                     0.8.3                    pypi_0    pypi
pbr                       5.11.0                   pypi_0    pypi
pexpect                   4.8.0                    pypi_0    pypi
pickleshare               0.7.5                    pypi_0    pypi
pillow                    9.3.0                    pypi_0    pypi
pip                       22.3.1             pyhd8ed1ab_0    conda-forge
platformdirs              2.5.4                    pypi_0    pypi
prompt-toolkit            3.0.33                   pypi_0    pypi
protobuf                  4.21.9                   pypi_0    pypi
psutil                    5.9.4                    pypi_0    pypi
ptyprocess                0.7.0                    pypi_0    pypi
pure-eval                 0.2.2                    pypi_0    pypi
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pygments                  2.13.0                   pypi_0    pypi
pyparsing                 3.0.9                    pypi_0    pypi
pyqt6                     6.4.0                    pypi_0    pypi
pyqt6-qt6                 6.4.0                    pypi_0    pypi
pyqt6-sip                 13.4.0                   pypi_0    pypi
pyqtgraph                 0.13.1                   pypi_0    pypi
python                    3.10.4          h14b404e_0_cpython    conda-forge
python-dateutil           2.8.2                    pypi_0    pypi
python_abi                3.10                    2_cp310    conda-forge
pytorch                   1.14.0.dev20221116        py3.10_0    pytorch-nightly
pytorch-ranger            0.1.1                    pypi_0    pypi
pywavelets                1.4.1                    pypi_0    pypi
pyzmq                     24.0.1                   pypi_0    pypi
qtpy                      2.3.0                    pypi_0    pypi
readline                  8.1.2                h46ed386_0    conda-forge
requests                  2.28.1                   pypi_0    pypi
rsa                       4.9                      pypi_0    pypi
scikit-image              0.19.3                   pypi_0    pypi
scikit-learn              1.1.3                    pypi_0    pypi
scipy                     1.9.3                    pypi_0    pypi
setuptools                65.5.1             pyhd8ed1ab_0    conda-forge
six                       1.16.0                   pypi_0    pypi
snappy                    1.1.9                h17c5cce_2    conda-forge
sqlite                    3.39.4               h2229b38_0    conda-forge
stack-data                0.6.2                    pypi_0    pypi
superqt                   0.4.0                    pypi_0    pypi
sympy                     1.11.1                   pypi_0    pypi
threadpoolctl             3.1.0                    pypi_0    pypi
tifffile                  2022.10.10               pypi_0    pypi
tk                        8.6.12               he1e0b03_0    conda-forge
torch                     1.13.0                   pypi_0    pypi
torch-optimizer           0.3.0                    pypi_0    pypi
tornado                   6.2                      pypi_0    pypi
tqdm                      4.64.1                   pypi_0    pypi
traitlets                 5.6.0                    pypi_0    pypi
typing_extensions         4.4.0              pyha770c72_0    conda-forge
tzdata                    2022f                h191b570_0    conda-forge
urllib3                   1.26.12                  pypi_0    pypi
wcwidth                   0.2.5                    pypi_0    pypi
wheel                     0.38.4             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h57fd34a_0    conda-forge
zfp                       1.0.0                hb6e4faa_3    conda-forge
zlib-ng                   2.0.6                he4db4b2_0    conda-forge
zstd                      1.5.2                h8128057_4    conda-forge

and the pip list:

> pip list
Package                  Version
------------------------ -------------------
appnope                  0.1.3
asttokens                2.2.0
backcall                 0.2.0
cachetools               5.2.0
cellpose-omni            0.7.2
certifi                  2022.9.24
charset-normalizer       2.1.1
colour                   0.1.5
comm                     0.1.1
contourpy                1.0.6
cycler                   0.11.0
darkdetect               0.7.1
debugpy                  1.6.4
decorator                5.1.1
edt                      2.3.0
entrypoints              0.4
executing                1.2.0
fastremap                1.13.3
fonttools                4.38.0
google-api-core          2.10.2
google-auth              2.14.1
google-cloud-core        2.3.2
google-cloud-storage     2.6.0
google-crc32c            1.5.0
google-resumable-media   2.4.0
googleapis-common-protos 1.57.0
idna                     3.4
imagecodecs              2022.9.26
imageio                  2.22.4
ipykernel                6.18.2
ipython                  8.7.0
jedi                     0.18.2
joblib                   1.2.0
jupyter_client           7.4.7
jupyter_core             5.1.0
kiwisolver               1.4.4
llvmlite                 0.39.1
mahotas                  1.4.13
matplotlib               3.6.2
matplotlib-inline        0.1.6
mgen                     1.2.0
mpmath                   1.2.1
natsort                  8.2.0
ncolor                   1.1.8
nest-asyncio             1.5.6
networkx                 2.8.8
numba                    0.56.4
numpy                    1.23.4
omnipose                 0.3.4.dev9+g88ccc8d
omnipose-theme           1.1.2
opencv-python-headless   4.6.0.66
packaging                21.3
parso                    0.8.3
pbr                      5.11.0
pexpect                  4.8.0
pickleshare              0.7.5
Pillow                   9.3.0
pip                      22.3.1
platformdirs             2.5.4
prompt-toolkit           3.0.33
protobuf                 4.21.9
psutil                   5.9.4
ptyprocess               0.7.0
pure-eval                0.2.2
pyasn1                   0.4.8
pyasn1-modules           0.2.8
Pygments                 2.13.0
pyparsing                3.0.9
PyQt6                    6.4.0
PyQt6-Qt6                6.4.0
PyQt6-sip                13.4.0
pyqtgraph                0.13.1
python-dateutil          2.8.2
pytorch-ranger           0.1.1
PyWavelets               1.4.1
pyzmq                    24.0.1
QtPy                     2.3.0
requests                 2.28.1
rsa                      4.9
scikit-image             0.19.3
scikit-learn             1.1.3
scipy                    1.9.3
setuptools               65.5.1
six                      1.16.0
stack-data               0.6.2
superqt                  0.4.0
sympy                    1.11.1
threadpoolctl            3.1.0
tifffile                 2022.10.10
torch                    1.14.0.dev20221116
torch-optimizer          0.3.0
tornado                  6.2
tqdm                     4.64.1
traitlets                5.6.0
typing_extensions        4.4.0
urllib3                  1.26.12
wcwidth                  0.2.5
wheel                    0.38.4

Thank you!

[Reardonlab]

I'm also having the exact same issue. Clean environment to start. Anyone figured out a solution or work around?

[audreyeternal]

I also have the same problem :(

[gouthamr321]

Hi @kevinjohncutler, I seem to get this same issue when trying to run training in 3D on a custom dataset that I prepared. My images are saved as ZYX and I started off with a clean conda environment. This is the error message I am seeing:

I am running on the CLI:
python -m omnipose --train --use_gpu --dir /combined_dataset_omnipose --mask_filter _mask --n_epochs 10 --pretrained_model None --verbose --dim 3 --save_each

Has anyone figured out any solutions for this?

[kevinjohncutler]

@gouthamr321 @Reardonlab @mccruz07 @audreyeternal Sorry for the delay, I wasn't able to reproduce this issue but it dawned on me that none of the commands posted are using --all_channels. This is the source of the error. See the training docs page showing the use of this command on 3D data. I apologize that this was not emphasized. but this argument is required for mono-channel images in 3D. Arguably, the necessity of this for mono-channel images should be considered a bug, and I will work on a fix. For now, please let me know if this solves your issue. I have not tried training on 3D multichannel images either, so I am curious if anyone has attempted this.

[mccruz07]

Hey @kevinjohncutler

Sorry to bother you again, but the initial issue was resolved using the --all_channels flag, but now I'm seeing another error:

> python -m omnipose --train --use_gpu --dir ./omni/new --mask_filter _masks --n_epochs 400 --pretrained_model None  --learning_rate 0.1 --save_every 50 --save_each  --verbose  --look_one_level_down --all_channels --dim 3 --RAdam --batch_size 4
2023-02-16 10:07:34,799 [INFO] WRITING LOG OUTPUT TO /Users/mcruz/.cellpose/run.log
log file /Users/mcruz/.cellpose/run.log
2023-02-16 10:07:34,801 [INFO] ** TORCH GPU version installed and working. **
2023-02-16 10:07:34,801 [INFO] >>>> using GPU
Omnipose enabled. See Omnipose repo for licencing details.
2023-02-16 10:07:34,801 [INFO] Training omni model. Setting nclasses=4, RAdam=True
2023-02-16 10:07:34,814 [INFO] not all flows are present, will run flow generation for all images
2023-02-16 10:07:34,819 [INFO] training from scratch
2023-02-16 10:07:34,819 [INFO] during training rescaling images to fixed diameter of 30.0 pixels
2023-02-16 10:07:34,940 [INFO] Training with rescale = 1.00
2023-02-16 10:07:35,430 [INFO] No precomuting flows with Omnipose. Computed during training.
2023-02-16 10:07:35,455 [WARNING] channels is set to None, input must therefore have nchan channels (default is 2)
2023-02-16 10:07:35,462 [INFO] >>> Using RAdam optimizer
Traceback (most recent call last):
  File "/Users/mcruz/opt/anaconda3/envs/omnipose1/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/mcruz/opt/anaconda3/envs/omnipose1/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/mcruz/opt/anaconda3/envs/omnipose1/lib/python3.9/site-packages/omnipose/__main__.py", line 3, in <module>
    main(omni_CLI=True)
  File "/Users/mcruz/opt/anaconda3/envs/omnipose1/lib/python3.9/site-packages/cellpose_omni/__main__.py", line 480, in main
    cpmodel_path = model.train(images, labels, links, train_files=image_names,
  File "/Users/mcruz/opt/anaconda3/envs/omnipose1/lib/python3.9/site-packages/cellpose_omni/models.py", line 1071, in train
    model_path = self._train_net(train_data, train_labels, train_links,
  File "/Users/mcruz/opt/anaconda3/envs/omnipose1/lib/python3.9/site-packages/cellpose_omni/core.py", line 935, in _train_net
    if links is not None:
NameError: name 'links' is not defined