It seems that all of the ML related dependencies are not being published through PyPI's JSON API so that Poetry can inspect them:

• https://github.com/isl-org/Open3D-ML/blob/8ddb67206e4fef55b39eea691ff00d49cef18be5/requirements.txt

Having a minimal non-ML package would also really help this overall situation:

• Provide a non-ML "open3d-lite" package to minimize Linux wheel size #5124
• pip: Why does Open3D pull in Open3D-ML/requirements.txt? #2991

To work around this, I had to mirror all of the ML requirements that are missing from 0.16.1 JSON API:

[tool.poetry.dependencies]
python = "^3.10, <3.11"
open3d = "0.16.1"
# See: https://github.com/isl-org/Open3D-ML/blob/8ddb67206e4fef55b39eea691ff00d49cef18be5/requirements.txt
addict = "*"
pillow = "*"
matplotlib = "*"
numpy = "*"
pandas = "*"
pyyaml = "*"
scikit-learn = "*"
tqdm = "*"
pyquaternion = "*"

The Poetry devs pointed out that they are pulling the information from the PyPI JSON API. This currently is populated by the first wheel that is uploaded. If you look at:

• https://pypi.org/pypi/open3d/0.16.1/json

    "requires_dist": [
      "numpy (>=1.18.0)",
      "dash (>=2.6.0)",
      "nbformat (==5.5.0)",
      "configargparse"
    ],

• But 0.16.0: https://pypi.org/pypi/open3d/0.16.0/json

    "requires_dist": [
      "numpy (>=1.18.0)",
      "dash (>=2.6.0)",
      "nbformat (==5.5.0)",
      "configargparse",
      "addict",
      "pillow (>=8.2.0)",
      "matplotlib (>=3)",
      "numpy (>1.15)",
      "pandas (>=1.0)",
      "pyyaml (>=5.4.1)",
      "scikit-learn (>=0.21)",
      "tqdm",
      "pyquaternion"
    ],

This is related to how this issue was solved:

• open3d 0.16.0: arm64 conda python unable to import due to '_quadmath_snprintf' not found #5607 (comment)

Okay, I've discovered the root cause of this issue. The ARM macOS wheels do not have the full list of requirements. If you unzip:

• https://files.pythonhosted.org/packages/dd/2e/11bfade1f5438a874d6978edd98978427b528404664b281f08b350f251c3/open3d-0.16.1-cp310-cp310-macosx_12_0_arm64.whl

And look at the METADATA file:

Requires-Dist: numpy (>=1.18.0)
Requires-Dist: dash (>=2.6.0)
Requires-Dist: nbformat (==5.5.0)
Requires-Dist: configargparse

Since these were the first wheels uploaded, the JSON API stores those, and the x86 wheel dependencies are not locked properly by Poetry.

Hi @johnthagen as far as I can tell, your debugging looks correct. The ARM macOS wheels do not support 3DML (yet) and so do not have the extra Open3D-ML requirements. From the next release, we will add 3DML support to macOS ARM and this issue should not occur. This is not related to the v0.16.1 bugfixes.

@ssheorey Do you also think we could for the next release make the ML dependencies optional?

Something like:

pip install open3d[ml]

Similar for Dash:

pip install open3d[dash]

• Make dash an optional dependency #5740

To get everything:

pip install open3d[all]

setuptools handles it this way:

• https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#optional-dependencies

This would make it much nicer to use Open3D in constrained, headless environments.

@johnthagen thanks for the suggestion - we will consider it, but note that it does need significant effort in updating packaging and documentation.

I was wondering if there was any way to use poetry + open3d at the moment? My company has been stuck on 0.16.0 since 0.16.1, 0.17.0, and now 0.18.0 all fail to have dependencies picked up correctly by poetry. This is reproducible on Ubuntu 22.04.

Hi @apockill based on @johnthagen 's debugging this seems to be a strange issue / race condition in the PyPI json API / poetry dependency checker, so I'm not really sure what we can do about this. If there is a fix for Open3D we would be happy to merge it. Otherwise, perhaps this should be addressed as a bug in poetry or PyPI json API.

The way to make open3d work with poetry is for open3d to use the same set of requirements in all distributions, with variations expressed through pep508 markers eg

not-arm-64-dependency==1.0.0 ; platform_machine != "arm64"

@ssheorey Here is a link the section of PEP 508 that describes environment markers:

• https://peps.python.org/pep-0508/#environment-markers

If we follow this line:

Here are the missing requirements:

• https://github.com/isl-org/Open3D-ML/blob/main/requirements.txt

So I think the improvement needed would be to unconditionally include the ML dependencies, but put environment markers on them such that they are only installed for the wheels that include ML functionality on their platform (e.g. platform_machine != "arm64")

Related to this, it would be convenient if the user could opt-in using extras to these various sub parts of Open3D, as discussed in

• Provide a non-ML "open3d-lite" package to minimize Linux wheel size #5124
• Make dash an optional dependency #5740
• Support for Python Optional Dependencies #5853
• Basic support for Python Optional Dependencies #5940

I think that by using PEP 508 environment markers and extras together, Open3D's dependencies could work a lot nicer together.

not-arm-64-dependency==1.0.0

Thank you for the quick response! I'm excited to get this working. I'm having a bit of trouble figuring out how to get this environment marker to work. I tried a few lines in my pyproject.toml, but had various syntax errors. I'm not sure if I'm missing something super obvious 😅

open3d = { version = "0.18.0", markers = "not-arm-64-dependency==1.0.0 and platform_machine != 'arm64'" }

It seems like no matter where I put it, the not-arm-64-dependency trips up my poetry lock command (poetry==1.7.1):

No terminal matches 'n' in the current parser context, at line 1 col 1

not-arm-64-dependency==1.0.0 platform_ma
^
Expected one of: 
        * L_PAREN
        * ESCAPED_STRING
        * SINGLE_QUOTED_STRING
        * MARKER_NAME

I wasn't able to find documentation on this marker, so my trail ran cold. When I tried using just the platform_machine != 'arm64' I was able to poetry lock successfully, but the dependencies were still not being locked correctly. Thank you again for you help!

@apockill This is something that the Open3D team would need to change about their built wheels, not something that an end user can fix externally.

@dimbleby Was just showing an example with not-arm-64-dependency as an example of one of Open3D's dependencies that are x86-only (it's not a real package, he was just showing how PEP 508 markers work).

Hi @dimbleby thanks for the suggestion - I think we can make that work with some effort. The way dependencies are set are:

• there are different requirement files for specific components (e.g. Jupyter support or Open3D-ML).
• wheel building is controlled by cmake. Depending on which build components are enabled, the specific requirements files are used.

The change would be to add all requirement files, but update the environment markers based on the enabled components. setup.py looks to be the right place for this change.

We would appreciate it if someone wants to create a PR for this. I can provide help as needed.

I inspected the 0.18 wheels, and it looks like the matrix of ML dependencies is a little more complex than just "not ARM":

• windows x86_64: no
• mac universal: yes
• mac arm64: yes
• linux x86_64: yes
• linux arm64: no

@ssheorey Does this sound accurate?

So the environment marker we'd need to apply to each ML dependency would look something like:

platform_system == "Darwin" or (platform_system == "Linux" and platform_machine == "x86_64")