Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feedback on the new resolution mechanism #230

Closed
krassowski opened this issue Apr 3, 2024 · 10 comments · Fixed by #233
Closed

Feedback on the new resolution mechanism #230

krassowski opened this issue Apr 3, 2024 · 10 comments · Fixed by #233
Labels
enhancement New feature or request

Comments

@krassowski
Copy link
Member

  1. It looks like this is going to break a few more packages. I just tried running CI on jupyter-server and see this:
+ '[' Linux == Windows ']'
+ '[' -f pyproject.toml ']'
+ uv pip compile --resolution=lowest-direct pyproject.toml -o /home/runner/constraints.txt
error: Failed to download and build: overrides==0.1
  Caused by: Failed to build: overrides==0.1
  Caused by: Build backend failed to determine extra requires with `build_wheel()` with exit status: 1
--- stdout:

Also many other jobs (e.g. https://github.com/krassowski/jupyter_server/actions/runs/8542681567/job/23404855225) seem to be failing with:

E       TypeError: websocket_connect() got an unexpected keyword argument 'body'

but this might be unrelated

  1. It installs the lowest version possible when no floor version is defined:
dependencies = [
    "async_lru>=1.0.0",
    "httpx>=0.25.0",
    "importlib-metadata>=4.8.3;python_version<\"3.10\"",
    "importlib-resources>=1.4;python_version<\"3.9\"",
    "ipykernel",
    "jinja2>=3.0.3",
    "jupyter_core",
    "jupyter_server>=2.4.0,<3",
    "jupyter-lsp>=2.0.0",
    "jupyterlab_server>=2.19.0,<3",
    "notebook_shim>=0.2",
    "packaging",
    "tomli;python_version<\"3.11\"",
    "tornado>=6.2.0",
    "traitlets",
]

Results in:

+ uv pip compile --resolution=lowest-direct pyproject.toml -o /home/runner/constraints.txt
error: Failed to download and build: traitlets==0.0.1
  Caused by: Failed to build: traitlets==0.0.1
  Caused by: Build backend failed to determine extra requires with `build_wheel()` with exit status: 1
  1. It pins to oldest available version even if other dependencies of dependencies have a pin (after pinning traitlets to something modern):
+ uv pip compile --resolution=lowest-direct pyproject.toml -o /home/runner/constraints.txt
Resolved 92 packages in 1.07s
# This file was autogenerated by uv via the following command:
#    uv pip compile --resolution=lowest-direct pyproject.toml -o /home/runner/constraints.txt
anyio==4.3.0
    # via
    #   httpcore
    #   jupyter-server
argon2-cffi==23.1.0
    # via jupyter-server
argon2-cffi-bindings==21.2.0
    # via argon2-cffi
arrow==1.3.0
    # via isoduration
asttokens==2.4.1
    # via stack-data
async-lru==1.0.0
attrs==23.2.0
    # via
    #   jsonschema
    #   referencing
babel==2.14.0
    # via jupyterlab-server
backcall==0.2.0
    # via ipython
beautifulsoup4==4.12.3
    # via nbconvert
bleach==6.1.0
    # via nbconvert
certifi==2024.2.2
    # via
    #   httpcore
    #   httpx
    #   requests
cffi==1.16.0
    # via argon2-cffi-bindings
charset-normalizer==3.3.2
    # via requests
decorator==5.1.1
    # via ipython
defusedxml==0.7.1
    # via nbconvert
exceptiongroup==1.2.0
    # via anyio
executing==2.0.1
    # via stack-data
fastjsonschema==2.19.1
    # via nbformat
fqdn==1.5.1
    # via jsonschema
h11==0.14.0
    # via httpcore
httpcore==0.18.0
    # via httpx
httpx==0.25.0
idna==3.6
    # via
    #   anyio
    #   httpx
    #   jsonschema
    #   requests
importlib-metadata==4.8.3
    # via
    #   jupyter-client
    #   jupyter-lsp
    #   jupyterlab-server
    #   nbconvert
importlib-resources==1.4.0
    # via
    #   jsonschema
    #   jsonschema-specifications
ipykernel==4.0.1
ipython==8.12.3
    # via ipykernel
isoduration==20.11.0
    # via jsonschema
jedi==0.19.1
    # via ipython
jinja2==3.0.3
    # via
    #   jupyter-server
    #   jupyterlab-server
    #   nbconvert
json5==0.9.24
    # via jupyterlab-server
jsonpointer==2.4
    # via jsonschema
jsonschema==4.21.1
    # via
    #   jupyter-events
    #   jupyterlab-server
    #   nbformat
jsonschema-specifications==2023.12.1
    # via jsonschema
jupyter-client==8.6.1
    # via
    #   ipykernel
    #   jupyter-server
    #   nbclient
jupyter-core==4.12.0
    # via
    #   jupyter-client
    #   jupyter-server
    #   nbclient
    #   nbconvert
    #   nbformat
jupyter-events==0.10.0
    # via jupyter-server
jupyter-lsp==2.0.0
jupyter-server==2.4.0
    # via
    #   jupyter-lsp
    #   jupyterlab-server
    #   notebook-shim
jupyter-server-terminals==0.5.3
    # via jupyter-server
jupyterlab-pygments==0.3.0
    # via nbconvert
jupyterlab-server==2.19.0
markupsafe==2.1.5
    # via
    #   jinja2
    #   nbconvert
matplotlib-inline==0.1.6
    # via ipython
mistune==3.0.2
    # via nbconvert
nbclient==0.10.0
    # via nbconvert
nbconvert==7.16.3
    # via jupyter-server
nbformat==5.10.3
    # via
    #   jupyter-server
    #   nbclient
    #   nbconvert
notebook-shim==0.2.0
packaging==21.3
    # via
    #   jupyter-server
    #   jupyterlab-server
    #   nbconvert
pandocfilters==1.5.1
    # via nbconvert
parso==0.8.3
    # via jedi
pexpect==4.9.0
    # via ipython
pickleshare==0.7.5
    # via ipython
pkgutil-resolve-name==1.3.10
    # via jsonschema
prometheus-client==0.20.0
    # via jupyter-server
prompt-toolkit==3.0.43
    # via ipython
ptyprocess==0.7.0
    # via
    #   pexpect
    #   terminado
pure-eval==0.2.2
    # via stack-data
pycparser==2.22
    # via cffi
pygments==2.17.2
    # via
    #   ipython
    #   nbconvert
pyparsing==3.1.2
    # via packaging
python-dateutil==2.9.0.post0
    # via
    #   arrow
    #   jupyter-client
python-json-logger==2.0.7
    # via jupyter-events
pytz==2024.1
    # via babel
pyyaml==6.0.1
    # via jupyter-events
pyzmq==25.1.2
    # via
    #   jupyter-client
    #   jupyter-server
referencing==0.34.0
    # via
    #   jsonschema
    #   jsonschema-specifications
    #   jupyter-events
requests==2.31.0
    # via jupyterlab-server
rfc3339-validator==0.1.4
    # via
    #   jsonschema
    #   jupyter-events
rfc3986-validator==0.1.1
    # via
    #   jsonschema
    #   jupyter-events
rpds-py==0.18.0
    # via
    #   jsonschema
    #   referencing
send2trash==1.8.2
    # via jupyter-server
six==1.16.0
    # via
    #   asttokens
    #   bleach
    #   python-dateutil
    #   rfc3339-validator
sniffio==1.3.1
    # via
    #   anyio
    #   httpcore
    #   httpx
soupsieve==2.5
    # via beautifulsoup4
stack-data==0.6.3
    # via ipython
terminado==0.18.1
    # via
    #   jupyter-server
    #   jupyter-server-terminals
tinycss2==1.2.1
    # via nbconvert
tomli==0.2.0
tornado==6.2
    # via
    #   jupyter-client
    #   jupyter-server
    #   terminado
traitlets==5.6.0
    # via
    #   ipykernel
    #   ipython
    #   jupyter-client
    #   jupyter-core
    #   jupyter-events
    #   jupyter-server
    #   matplotlib-inline
    #   nbclient
    #   nbconvert
    #   nbformat
types-python-dateutil==2.9.0.20240316
    # via arrow
typing-extensions==4.10.0
    # via
    #   anyio
    #   ipython
uri-template==1.3.0
    # via jsonschema
urllib3==2.2.1
    # via requests
wcwidth==0.2.13
    # via prompt-toolkit
webcolors==1.13
    # via jsonschema
webencodings==0.5.1
    # via
    #   bleach
    #   tinycss2
websocket-client==1.7.0
    # via jupyter-server
zipp==3.18.1
    # via importlib-metadata
/scripts/ci_install.sh: line 19: yarn: command not found
+ mkdir -p /home/runner/.jupyter
+ git config --global user.name foo
+ git config --global user.email [email protected]
+ pip install -q --upgrade pip --user
+ pip --version
pip 24.0 from /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pip (python 3.8)
+ pip install -e '.[dev,test]'
Obtaining file:///home/runner/work/jupyterlab/jupyterlab
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [[29](https://github.com/krassowski/jupyterlab/actions/runs/8542572841/job/23404523285#step:4:30) lines of output]
      Collecting hatchling>=1.21.1
        Using cached hatchling-1.22.4-py3-none-any.whl.metadata (3.8 kB)
      Collecting packaging>=21.3 (from hatchling>=1.21.1)
        Downloading packaging-21.3-py3-none-any.whl.metadata (15 kB)
      Collecting pathspec>=0.10.1 (from hatchling>=1.21.1)
        Using cached pathspec-0.12.1-py3-none-any.whl.metadata (21 kB)
      Collecting pluggy>=1.0.0 (from hatchling>=1.21.1)
        Using cached pluggy-1.4.0-py3-none-any.whl.metadata (4.3 kB)
      INFO: pip is looking at multiple versions of hatchling to determine which version is compatible with other requirements. This could take a while.
      Collecting hatchling>=1.21.1
        Downloading hatchling-1.22.3-py3-none-any.whl.metadata (3.8 kB)
        Downloading hatchling-1.22.2-py3-none-any.whl.metadata (3.8 kB)
        Using cached hatchling-1.21.1-py3-none-any.whl.metadata (3.8 kB)
      Collecting editables>=0.3 (from hatchling>=1.21.1)
        Using cached editables-0.5-py3-none-any.whl.metadata (3.1 kB)
      ERROR: Cannot install hatchling==1.21.1, hatchling==1.22.2, hatchling==1.22.3 and hatchling==1.22.4 because these package versions have conflicting dependencies.
      
      The conflict is caused by:
          hatchling 1.22.4 depends on tomli>=1.2.2; python_version < "3.11"
          hatchling 1.22.3 depends on tomli>=1.2.2; python_version < "3.11"
          hatchling 1.22.2 depends on tomli>=1.2.2; python_version < "3.11"
          hatchling 1.21.1 depends on tomli>=1.2.2; python_version < "3.11"
          The user requested (constraint) tomli==0.2.0
      
      To fix this you could try to:
      1. loosen the range of package versions you've specified
      2. remove package versions to allow pip attempt to solve the dependency conflict
      
      ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
+ pip install -v -e '.[dev,test]'
Using pip 24.0 from /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pip (python 3.8)
Obtaining file:///home/runner/work/jupyterlab/jupyterlab
  Installing build dependencies: started
  Running command pip subprocess to install build dependencies
  Collecting hatchling>=1.21.1
    Using cached hatchling-1.22.4-py3-none-any.whl.metadata (3.8 kB)
  Collecting packaging>=21.3 (from hatchling>=1.21.1)
    Using cached packaging-21.3-py3-none-any.whl.metadata (15 kB)
  Collecting pathspec>=0.10.1 (from hatchling>=1.21.1)
    Using cached pathspec-0.12.1-py3-none-any.whl.metadata (21 kB)
  Collecting pluggy>=1.0.0 (from hatchling>=1.21.1)
    Using cached pluggy-1.4.0-py3-none-any.whl.metadata (4.3 kB)
  INFO: pip is looking at multiple versions of hatchling to determine which version is compatible with other requirements. This could take a while.
  Collecting hatchling>=1.21.1
    Using cached hatchling-1.22.3-py3-none-any.whl.metadata (3.8 kB)
    Using cached hatchling-1.22.2-py3-none-any.whl.metadata (3.8 kB)
    Using cached hatchling-1.21.1-py3-none-any.whl.metadata (3.8 kB)
  Collecting editables>=0.3 (from hatchling>=1.21.1)
    Using cached editables-0.5-py3-none-any.whl.metadata (3.1 kB)
  ERROR: Cannot install hatchling==1.21.1, hatchling==1.22.2, hatchling==1.22.3 and hatchling==1.22.4 because these package versions have conflicting dependencies.
  The conflict is caused by:
      hatchling 1.22.4 depends on tomli>=1.2.2; python_version < "3.11"
      hatchling 1.22.3 depends on tomli>=1.2.2; python_version < "3.11"
      hatchling 1.22.2 depends on tomli>=1.2.2; python_version < "3.11"
      hatchling 1.21.1 depends on tomli>=1.2.2; python_version < "3.11"
      The user requested (constraint) tomli==0.2.0
  To fix this you could try to:
  1. loosen the range of package versions you've specified
  2. remove package versions to allow pip attempt to solve the dependency conflict
  ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
  error: subprocess-exited-with-error
@krassowski krassowski added the enhancement New feature or request label Apr 3, 2024
@krassowski
Copy link
Member Author

Also on notebook in jupyter/notebook#7312:

+ '[' standard == pre ']'
+ '[' standard == minimum ']'
+ uv pip compile pyproject.toml -o /home/runner/constraints.txt
Resolved 91 packages in 559ms
# This file was autogenerated by uv via the following command:
#    uv pip compile pyproject.toml -o /home/runner/constraints.txt
anyio==4.3.0
    # via
    #   httpx
    #   jupyter-server
argon2-cffi==23.1.0
    # via jupyter-server
argon2-cffi-bindings==21.2.0
    # via argon2-cffi
arrow==1.3.0
    # via isoduration
asttokens==2.4.1
    # via stack-data
async-lru==2.0.4
    # via jupyterlab
attrs==23.2.0
    # via
    #   jsonschema
    #   referencing
babel==2.14.0
    # via jupyterlab-server
beautifulsoup4==4.12.3
    # via nbconvert
bleach==6.1.0
    # via nbconvert
certifi==2024.2.2
    # via
    #   httpcore
    #   httpx
    #   requests
cffi==1.16.0
    # via argon2-cffi-bindings
charset-normalizer==3.3.2
    # via requests
comm==0.2.2
    # via ipykernel
debugpy==1.8.1
    # via ipykernel
decorator==5.1.1
    # via ipython
defusedxml==0.7.1
    # via nbconvert
exceptiongroup==1.2.0
    # via
    #   anyio
    #   ipython
executing==2.0.1
    # via stack-data
fastjsonschema==2.19.1
    # via nbformat
fqdn==1.5.1
    # via jsonschema
h11==0.14.0
    # via httpcore
httpcore==1.0.5
    # via httpx
httpx==0.27.0
    # via jupyterlab
idna==3.6
    # via
    #   anyio
    #   httpx
    #   jsonschema
    #   requests
ipykernel==6.29.4
    # via jupyterlab
ipython==8.23.0
    # via ipykernel
isoduration==20.11.0
    # via jsonschema
jedi==0.19.1
    # via ipython
jinja2==3.1.3
    # via
    #   jupyter-server
    #   jupyterlab
    #   jupyterlab-server
    #   nbconvert
json5==0.9.24
    # via jupyterlab-server
jsonpointer==2.4
    # via jsonschema
jsonschema==4.21.1
    # via
    #   jupyter-events
    #   jupyterlab-server
    #   nbformat
jsonschema-specifications==2023.12.1
    # via jsonschema
jupyter-client==8.6.1
    # via
    #   ipykernel
    #   jupyter-server
    #   nbclient
jupyter-core==5.7.2
    # via
    #   ipykernel
    #   jupyter-client
    #   jupyter-server
    #   jupyterlab
    #   nbclient
    #   nbconvert
    #   nbformat
jupyter-events==0.10.0
    # via jupyter-server
jupyter-lsp==2.2.4
    # via jupyterlab
jupyter-server==2.13.0
    # via
    #   jupyter-lsp
    #   jupyterlab
    #   jupyterlab-server
    #   notebook-shim
jupyter-server-terminals==0.5.3
    # via jupyter-server
jupyterlab==4.2.0b0
jupyterlab-pygments==0.3.0
    # via nbconvert
jupyterlab-server==2.25.4
    # via jupyterlab
markupsafe==2.1.5
    # via
    #   jinja2
    #   nbconvert
matplotlib-inline==0.1.6
    # via
    #   ipykernel
    #   ipython
mistune==3.0.2
    # via nbconvert
nbclient==0.10.0
    # via nbconvert
nbconvert==7.16.3
    # via jupyter-server
nbformat==5.10.3
    # via
    #   jupyter-server
    #   nbclient
    #   nbconvert
nest-asyncio==1.6.0
    # via ipykernel
notebook-shim==0.2.4
    # via jupyterlab
overrides==7.7.0
    # via jupyter-server
packaging==24.0
    # via
    #   ipykernel
    #   jupyter-server
    #   jupyterlab
    #   jupyterlab-server
    #   nbconvert
pandocfilters==1.5.1
    # via nbconvert
parso==0.8.3
    # via jedi
pexpect==4.9.0
    # via ipython
platformdirs==4.2.0
    # via jupyter-core
prometheus-client==0.20.0
    # via jupyter-server
prompt-toolkit==3.0.43
    # via ipython
psutil==5.9.8
    # via ipykernel
ptyprocess==0.7.0
    # via
    #   pexpect
    #   terminado
pure-eval==0.2.2
    # via stack-data
pycparser==2.22
    # via cffi
pygments==2.17.2
    # via
    #   ipython
    #   nbconvert
python-dateutil==2.9.0.post0
    # via
    #   arrow
    #   jupyter-client
python-json-logger==2.0.7
    # via jupyter-events
pyyaml==6.0.1
    # via jupyter-events
pyzmq==25.1.2
    # via
    #   ipykernel
    #   jupyter-client
    #   jupyter-server
referencing==0.34.0
    # via
    #   jsonschema
    #   jsonschema-specifications
    #   jupyter-events
requests==2.31.0
    # via jupyterlab-server
rfc3339-validator==0.1.4
    # via
    #   jsonschema
    #   jupyter-events
rfc3986-validator==0.1.1
    # via
    #   jsonschema
    #   jupyter-events
rpds-py==0.18.0
    # via
    #   jsonschema
    #   referencing
send2trash==1.8.2
    # via jupyter-server
six==1.16.0
    # via
    #   asttokens
    #   bleach
    #   python-dateutil
    #   rfc3339-validator
sniffio==1.3.1
    # via
    #   anyio
    #   httpx
soupsieve==2.5
    # via beautifulsoup4
stack-data==0.6.3
    # via ipython
terminado==0.18.1
    # via
    #   jupyter-server
    #   jupyter-server-terminals
tinycss2==1.2.1
    # via nbconvert
tomli==2.0.1
    # via jupyterlab
tornado==6.4
    # via
    #   ipykernel
    #   jupyter-client
    #   jupyter-server
    #   jupyterlab
    #   terminado
traitlets==5.14.2
    # via
    #   comm
    #   ipykernel
    #   ipython
    #   jupyter-client
    #   jupyter-core
    #   jupyter-events
    #   jupyter-server
    #   jupyterlab
    #   matplotlib-inline
+ echo PIP_CONSTRAINT=/home/runner/constraints.txt
    #   nbclient
    #   nbconvert
    #   nbformat
types-python-dateutil==2.9.0.20240[316](https://github.com/jupyter/notebook/actions/runs/8539292958/job/23396548527?pr=7312#step:3:330)
    # via arrow
typing-extensions==4.10.0
    # via
    #   anyio
    #   async-lru
    #   ipython
uri-template==1.3.0
    # via jsonschema
urllib3==2.2.1
    # via requests
wcwidth==0.2.13
    # via prompt-toolkit
webcolors==1.13
    # via jsonschema
webencodings==0.5.1
    # via
    #   bleach
    #   tinycss2
websocket-client==1.7.0
    # via jupyter-server
+ hatch run cov:test
Creating environment: cov
Installing project in development mode
ERROR: Cannot install jupyter-server, jupyterlab-server and openapi-core because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

https://github.com/jupyter/notebook/actions/runs/8539292958/job/23396548527?pr=7312

@krassowski
Copy link
Member Author

I guess there is nothing to do here. With bumping a few pins the job is now passing for me. I just think that maybe this should have been a new release given how many downstreams are affected? But maybe it is ok if it does find at least one true positive where lack of a pin made a difference (so far I have not seen it though).

@jtpio
Copy link
Member

jtpio commented Apr 4, 2024

Yes, and it's also a bit tedious to fix. For example in the case of the notebook issue reported above, the default hatch run cov:test does not give enough information:

+ hatch run cov:test
Creating environment: cov
Installing project in development mode
ERROR: Cannot install jupyter-server, jupyterlab-server and openapi-core because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

While doing a plain pip install -ve ".[dev,test]" gives more pointers:

image

In this case, it looks like it's not even fixable in notebook, but requires a fix in jupyterlab_server first?

@jtpio
Copy link
Member

jtpio commented Apr 4, 2024

In this case, it looks like it's not even fixable in notebook, but requires a fix in jupyterlab_server first?

OK the jupyterlab_server CI is also failing: jupyterlab/jupyterlab_server#446

So it looks like that would mean fixing the dependencies in jupyterlab_server, and then making a new release of jupyterlab_server to fix the CI of other downstreams (after bumping to the latest jupyterlab_server)?

@krassowski
Copy link
Member Author

Yes, the thing worries me is that even if we do some magic with the pins the final resolution may not represent the actual resolution users get when installing using pip. I think using the new constraints file generation method for minimum resolution mode is fine, but by default we should not use it.

@krassowski krassowski reopened this Apr 4, 2024
@blink1073
Copy link
Contributor

  1. That was a separate bug that I fixed.
  2. I think that is actually a good thing, and I've been updating packages to be more accurate.

I think the errors we're seeing in practice will be fixed by using --extra test in the resolution, doing that now.

@blink1073
Copy link
Contributor

I think using the new constraints file generation method for minimum resolution mode is fine, but by default we should not use it.

I disagree, this is an "advanced" check, and we're making it a bit stricter and more correct.

@blink1073
Copy link
Contributor

Ah, I do see the point about affecting regular installs. If the next release doesn't fix that in most cases, we can make it opt-in for regular installs.

@blink1073
Copy link
Contributor

Okay, no, now jupyter_server doesn't install. Let's call this a failed experiment...

@krassowski
Copy link
Member Author

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants