2023.08.0 release

[m-albert]

We got quite some things clarified and merged in the last days and probably we'd not wait for #306 (despite its quick progress). Are we ready for the next release?

[GenevieveBuckley]

Sure 😄

I agree that #306 should not block the release. It's a maintenance PR, users will not care.

[jakirkham]

Yeah releasing sounds like a good idea

We can always release again if we want to get pyproject.toml or other things out into the wild

[jakirkham]

Do you want to try running the release Marvin?

[m-albert]

Great. Then I'll try tomorrow (European Thursday), or Friday depending on how it goes.

[m-albert]

Just realised I don't have upload permissions for https://pypi.org/project/dask-image yet. Could you add me? My pypi user is https://pypi.org/user/m-albert/. Thank you!

Added an entry about this to Genevieve's "add new maintainer checklist" #195.

[jakirkham]

Done. Please check for an invite (maybe in email?)

Thanks Marvin! 🙏

Side note: Would consider adding 2FA to PyPI

[jakirkham]

Once the release is on PyPI, we can use the @conda-forge-admin, please update version bot request in a dask-image feedstock issue to start a conda-forge release

Might be worth adding this to the release docs (if it is not already there)

[m-albert]

Thank you very much John 🙏 Got and accepted the invite (both over email and visible in the account) and enabled 2FA for my pypi account as you suggested.

Once the release is on PyPI, we can use the @conda-forge-admin, please update version bot request in a dask-image feedstock issue to start a conda-forge release

Might be worth adding this to the release docs (if it is not already there)

Thanks a lot for the suggestion, I'll have a proper look at it tomorrow :)

[jakirkham]

Great! 😄

Also wrote up some thoughts on how we might automate more of the release procedure ( #331 )

[m-albert]

I'm currently going through the release process (you might have gotten some email 😁).

I created the v2023.08.0 release and everything up to the conda feedstock including the pypi upload worked well. However, I'm currently stuck in the conda-forge dask-image-feedstock PR.

After updating the dependencies in recipe/meta.yaml, the tests in the PR started to fail.

1. First, there was a mamba solver problem. I could reproduce the problem locally and realised that it disappears when removing the scikit-image >=0.12.3,<0.17.0 requirement, which in the feedstock is a test requirement for some reason. Since there is no reference to this anywhere in the repository (especially not in setup.py), I wanted to try what happens if I assume the requirement is there by mistake. Indeed, with this change the build works.
2. However, now there's a failing test related to an error appearing in find_objects. Interestingly, when installing the same requirements as the feedstock CI locally, I can reproduce the error both on linux and mac. When installing dependencies using pip, the error does not occur.

Any input is welcome and I'll further investigate this later tonight.

So currently there's a divide between the versions available on pypi and conda.

[jakirkham]

Could you please run pip list for both types of installs (working & failing) and include them here?

Also pro-tip the HTML tags <details></details> can be helpful for collapsing large amounts of output

[m-albert]

Failing install

Installed with:

• mamba create -n dask_image_release_mamba_py311 python=3.11
• mamba install -c conda-forge "pytest >=3.0.5" "tifffile >=2018.10.18" "pims >=0.4.1" "scipy >=0.19.1" "python >=3.9" "numpy >=1.18" "pandas>=2.0.0" "dask >=2023.2.0"
• pip install -e . (on dask-image main)

pip list

bokeh              3.2.1
Brotli             1.0.9
certifi            2023.7.22
charset-normalizer 3.2.0
click              8.1.6
cloudpickle        2.2.1
colorama           0.4.6
contourpy          1.1.0
cytoolz            0.12.2
dask               2023.7.1
dask-image         2023.3.0+14.g67540af.dirty /Users/malbert/software/dask-image
distributed        2023.7.1
exceptiongroup     1.1.2
fsspec             2023.6.0
idna               3.4
imagecodecs        2023.7.10
imageio            2.31.1
importlib-metadata 6.8.0
iniconfig          2.0.0
Jinja2             3.1.2
locket             1.0.0
lz4                4.3.2
MarkupSafe         2.1.3
msgpack            1.0.5
numpy              1.25.2
packaging          23.1
pandas             2.0.3
partd              1.4.0
Pillow             10.0.0
PIMS               0.6.1
pip                23.2.1
platformdirs       3.10.0
pluggy             1.2.0
pooch              1.7.0
psutil             5.9.5
pyarrow            12.0.1
PySocks            1.7.1
pytest             7.4.0
python-dateutil    2.8.2
pytz               2023.3
PyYAML             6.0
requests           2.31.0
scipy              1.11.1
setuptools         68.0.0
six                1.16.0
slicerator         1.1.0
sortedcontainers   2.4.0
tblib              1.7.0
tifffile           2023.7.18
tomli              2.0.1
toolz              0.12.0
tornado            6.3.2
typing_extensions  4.7.1
tzdata             2023.3
urllib3            2.0.4
wheel              0.38.4
xyzservices        2023.7.0
zict               3.0.0
zipp               3.16.2

Working install

Installed with:

• mamba create -n dask_image_release_pip_py311 python=3.11
• pip install -e . (on dask-image main)

pip list

click              8.1.6
cloudpickle        2.2.1
dask               2023.7.1
dask-image         2023.3.0+14.g67540af.dirty /Users/malbert/software/dask-image
fsspec             2023.6.0
imageio            2.31.1
importlib-metadata 6.8.0
iniconfig          2.0.0
locket             1.0.0
numpy              1.25.2
packaging          23.1
pandas             2.0.3
partd              1.4.0
Pillow             10.0.0
PIMS               0.6.1
pip                23.2.1
pluggy             1.2.0
pytest             7.4.0
python-dateutil    2.8.2
pytz               2023.3
PyYAML             6.0.1
scipy              1.11.1
setuptools         68.0.0
six                1.16.0
slicerator         1.1.0
tifffile           2023.7.18
toolz              0.12.0
tzdata             2023.3
wheel              0.38.4
zipp               3.16.2

[m-albert]

Digging some more I found that

1. Replacing pandas=2.0.3 by pandas<2 in the failing mamba install fixes the problem.
2. The error and the different behaviors between the installs can be tracked down to the following:

dask-image/dask_image/ndmeasure/_utils/_find_objects.py

Lines 68 to 70 in 67540af

	meta = dd.utils.make_meta([(i, object) for i in range(ndim)])
	if isinstance(df1, Delayed):
	df1 = dd.from_delayed(df1, meta=meta)

In both installs:
type(df1.compute()[0].iloc[0])

<class 'slice'>

While
type(dd.from_delayed(df1, meta=meta).compute()[0].iloc[0])

in the working install gives:

<class 'slice'>

and in the failing install:

<class 'str'>

Error tracing conclusion: It seems that dd.from_delayed is behaving differently for the two installs in the case of pandas>=2.0.0.

[jakirkham]

Wonder if this is related to Pandas Arrow support

There were some changes in Pandas 2.0

IIUC Dask is looking at defaulting to these (not sure if that is affecting us here). The dask package requires in pyarrow. There is an issue for feedback ( dask/dask#10139 ). Think we might want to leave a note there

Think we can turn that off, which is probably what we should do in conda-forge

We can file a follow up issue here to investigate after the release

[m-albert]

That might be it!

dd.from_delayed(df1, meta=meta).compute().dtypes

Working install:

0 object
1 object
2 object
dtype: object

Failing install:

0 string[pyarrow]
1 string[pyarrow]
2 string[pyarrow]
dtype: object

The dask package requires in pyarrow.

While in https://github.com/dask/dask/blob/main/pyproject.toml the dependency is just part of the [complete] target, explaining the difference between the pip and conda environments.

[m-albert]

Think we can turn that off, which is probably what we should do in conda-forge

Will check for ways to do that.

[jakirkham]

Already added one :)

xref: conda-forge/dask-image-feedstock#14 (comment)

Edit: Though if it needs more work, feel free to revise as needed ;)

[GenevieveBuckley]

...this sounds like it's been a really painful release process. Sorry you're going through it @m-albert and thank you for your perseverance!

[m-albert]

Might have been a bit of a bumpy process but actually this a lot of fun with you guys :) Thank you!

[m-albert]

2023.08.1 has been released and uploaded to pypi. The conda feedstock PR release has been merged and the new version should appear soon is available.

Should we remove 2023.08.0 from pypi as discussed here?

I've left some notes about my experiences with the release process here: #339.

[jakirkham]

Thanks Marvin! 🙏

Have yanked the release on PyPI: https://pypi.org/project/dask-image/2023.8.0

Appreciate you taking time to write notes on the experience 🙂

[m-albert]

Great, thank you John!

Now the availability of dask-image on pypi and conda-forge is the same again.

[jakirkham]

Ofc! Thank you for running the release 😀

[GenevieveBuckley]

I'll close this issue since dask-image 2023.8.1 is now available on pypi and conda-forge.
Thanks for all your hard work @m-albert 🎉