How do I support pip editable (-e) installs with pyproject.toml + setup.py (or setup.cfg)?

[fndrplayer39]

I have an old Python project that I am trying to update to work on Python 3.10. In the process, I have swapped out my setup.py for a pyproject.toml, as that seems to be the recommended practice moving forward.

Everything works except for editable installs: (i.e. pip install -e ./mypkg). With the pyproject.toml, modules are not being discovered, which results in AttributeError: module 'mypkg' has no attribute 'foo' when I go to run my app. For whatever reason, standard installation (without the -e flag) works fine.

The "Important" admonition box on this page and footnote [1] indicate that this might be a limitation of the current beta status, and that I can support editable installs by including either a simple setup.py, or setup.cfg, along with my pyproject.toml. Unfortunately, I haven't been able to make that work either.

If compatibility with legacy builds or versions of tools that don’t support certain packaging standards (e.g. PEP 517 or PEP 660), a simple setup.py script can be added to your project [1] (while keeping the configuration in pyproject.toml):

from setuptools import setup

setup()

[1] pip may allow editable install only with pyproject.toml and setup.cfg. However, this behavior may not be consistent over various pip versions and other packaging-related tools (setup.py is more reliable on those scenarios).

How would I adapt my project (below) to support editable installs?

• If I have to provide a setup.py anyway, what is the advantage of including a pyproject.toml?
• What configuration data goes in setup.py vs. pyproject.toml?
• If configuration variables are specified in both files, which takes precedence?

Other References:

• https://stackoverflow.com/questions/69711606/how-to-install-a-package-using-pip-in-editable-mode-with-pyproject-toml

---

Minimal Example:

• Python 3.10 in a virtual environment on Windows 10
• Latest versions of pip and setuptools.

The project is organized with a flat-laout.

.
├── .venv
└── mypkg/
    ├── mypkg/
    │   └── __init__.py
    ├── pyproject.toml
    └── setup.py

Virtual Environment:

py -3.10 -m venv .venv
.venv\scripts\activate
(.venv) python -m pip install --upgrade pip
(.venv) pip install --upgrade setuptools

Package    Version
---------- -------
pip        23.0
setuptools 67.2.0

__init__.py

foo = "bar"

setup.py

from setuptools import setup, find_packages

setup(
    name="mypkg",
    version="1.2.3",
    packages=find_packages(
        where=".",
        include=["mypkg*"],
    )
)

Alternate setup.py (verbatim from setuptools documentation):

from setuptools import setup

setup()

pyproject.toml

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "mypkg"
version = "1.2.3"

[tool.setuptools.packages.find]
where = ["."]
include = ["mypkg*"]

Install / Test:

(.venv) pip install --force-reinstall -e ./mypkg
(.venv) python -c "import mypkg;print(mypkg.foo)"

Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: module 'mypkg' has no attribute 'foo'

[abravalheri]

Hi @fndrplayer39, I think what is happening here is that you are hitting a limitation of editable installations for flat-layouts, due to the where you are running your test commands from. This limitation is described in the docs:

File system entries in the current working directory whose names coincidentally match installed packages may take precedence in Python’s import system. Users are encouraged to avoid such scenarios [2].

If you change directory before running your test script, it should be fine:

> docker run --rm -it python:3.10 /bin/bash


mkdir -p /tmp/mypkg/mypkg
cd /tmp/mypkg
python3.10 -m venv /tmp/.venv
cat <<EOF > pyproject.toml
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "mypkg"
version = "1.2.3"

[tool.setuptools.packages.find]
where = ["."]
include = ["mypkg*"]
EOF
echo 'foo = "bar"' > mypkg/__init__.py
/tmp/.venv/bin/python -m pip install -U pip
/tmp/.venv/bin/python -m pip install  --force-reinstall -e .
cd /tmp/.venv
./bin/python -c "import mypkg;print(mypkg.foo)"
# => bar

(Just a friendly reminder that for the Python interpreter any folder can be a namespace package, even if you don't install it:

mkdir /tmp/folder
cd /tmp
python -c 'import folder; print(folder)'
# => <module 'folder' (<_frozen_importlib_external._NamespaceLoader object at 0x7ff834b19cf0>)>

)

[fndrplayer39]

Thank you so much, @abravalheri, for taking the time to put this answer together, including your additional remarks below. I believe you’re right, that my local project folder is maybe getting confused with the installed package of the same name. Things do seem to work a bit more smoothly with the src-layout, in general, so I’ll proceed with that.

[abravalheri]

If I have to provide a setup.py anyway, what is the advantage of including a pyproject.toml?

You don't need a setup.py, but the editable installation has limitations and is not a silver bullet for all layouts/all projects/all circumstances.
The advantages of using pyproject.toml is that your project's metadata can be easily obtained without running custom scripts and with less overhead.

What configuration data goes in setup.py vs. pyproject.toml?

So far, I recommend only using setup.py if you need ext_modules or you want to customize cmdclass (If you don't need those, you can skip setup.py). You can keep metadata in pyproject.toml.

If configuration variables are specified in both files, which takes precedence?

pyproject.toml is processed after setup.py, but setuptools might complain if you have divergent information or you forgot to set pyproject.toml's dynamic appropriately...