Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incompatible with python 3.12 #606

Closed
3 tasks done
clbarnes opened this issue Jan 9, 2024 · 11 comments · May be fixed by navis-org/navis#140
Closed
3 tasks done

Incompatible with python 3.12 #606

clbarnes opened this issue Jan 9, 2024 · 11 comments · May be fixed by navis-org/navis#140
Labels
installation Issue affecting installation.

Comments

@clbarnes
Copy link

clbarnes commented Jan 9, 2024

Some dependencies cannot be installed on python 3.12, mainly because those packages do not list their build-time dependencies. Listing here as I hit them:

  • compresso
  • dracopy
    • pinned to <v2, which doesn't include a pyproject.toml and so does not specify that it needs skbuild
  • pysimdjson
clbarnes added a commit to clbarnes/navis that referenced this issue Jan 9, 2024
cloud-volume is still incompatible.
See here seung-lab/cloud-volume#606
@william-silversmith
Copy link
Contributor

Hmm, for compresso it seems that I have released wheels. I think you meant that this is fixed by the checkmark?

Yes, DracoPy needs a new release. Thanks for reminding me.

@clbarnes
Copy link
Author

clbarnes commented Jan 9, 2024

Yes, compresso's fixed, thanks for doing that!

@william-silversmith
Copy link
Contributor

forgot to mention, new DracoPy was released last week!

@clbarnes
Copy link
Author

clbarnes commented Jan 17, 2024

Unfortunately pysimdjson won't install either! Do you have a CI setup where the 3.12 build could be tested?

@william-silversmith
Copy link
Contributor

It looks like there's a PR for that with a workaround while it waits to be merged: TkTech/pysimdjson#113

I think I need to overhaul the CloudVolume tests. They've only been able to be run locally for a while.

@clbarnes
Copy link
Author

clbarnes commented Jan 18, 2024

By these benchmarks, orjson beats pysimdjson by some margin, has wheels up to and including 3.12, and if the wheels don't exist, the build story is "have a rust compiler released in the last year" rather than the nightmare that building C++ packages from source often is.

I can put together a github actions workflow if that would be helpful, if the limiting factor is just travis - but if it's about the test configuration itself I won't be of much use.

@william-silversmith
Copy link
Contributor

We use both orjson and pysimdjson unfortunately... We use orjson for decoding JSON rapidly as a whole, but pysimdjson has special methods for partial decoding that are necessary for managing very large and numerous json files. This is necessary for processing the spatial index, which we use for creating sharded mesh and skeleton types. It saves hours or days of computation.

@william-silversmith
Copy link
Contributor

william-silversmith commented Jan 18, 2024

It's also worth noting that the decoder amounts to about 5% of the compute time. Most of the time is spent marshaling the decoded objects into python objects in my experience.

@william-silversmith
Copy link
Contributor

I posted asking the maintainer of pysimdjson to update their repo. In the meantime, there may be a workaround. I haven't quite gotten this to work yet, but looking at the setup.py file, it seems that if you have cython installed and set BUILD_WITH_CYTHON=1 in your environment, you can force recythonization (which should fix the issue if your cython is up-to-date).

https://github.com/TkTech/pysimdjson/blob/master/setup.py#L32

@william-silversmith william-silversmith added the installation Issue affecting installation. label Feb 1, 2024
@william-silversmith
Copy link
Contributor

Looks like py3.12 wheels are available for pysimdjson. https://pypi.org/project/pysimdjson/#files

@william-silversmith
Copy link
Contributor

Afaik, this problem is solved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
installation Issue affecting installation.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants