Run custom code to generate package_data before install

[CareF]

Hi all,

I want the installation process to run some code to generate some data for the package, but I find my solutions show different behavior for pip and setup.py. So here is how I'm making it work:

from setuptools.command.install import install

class MyInstallCMD(install):
    def run(self):
        my_func()
        super().run()

setup(
    ...,
    package_data={'': ['GENERATED_DATA']},
    cmdclass={
        'install': MyInstallCMD
    },
)

This works for python setup.py install but does not work for pip install .
The following is the opposite: works for pip but not setup.py

from wheel.bdist_wheel import bdist_wheel

class MyInstallCMD(bdist_wheel):
    def run(self):
        my_func()
        super().run()

setup(
    ...,
    package_data={'': ['GENERATED_DATA']},
    cmdclass={
        'bdist_wheel': MyInstallCMD
    },
)

Combining both, and a global flag to avoid running my_func twice works now, but it looks like a dirty hack that will not last for future versions.

I also tried build and build_py and they behave just like install: not working for pip.

Any comments would be very appreciate!

---

P.S. I also have used pyproject.toml as suggested by the doc.

[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

[webknjaz]

@CareF pip install only falls back to setup.py install internally if it fails to build a wheel. And I think, this workaround will probably be deprecated at some point now that PEP 517 is a thing. Also, I wouldn't recommend using setup.py install without a compelling reason when you are already using pip.

For non-editable installs, pip will use PEP 517, if pyproject.toml is present. It will import and invoke the build-backend you set after pre-provisioning requires in an isolated (by default) env. From that point, it'll call hooks defined by PEP 517 to build sdist (a tarball) followed by a wheel (since you have wheel in requires, hence in that isolated env). Then, it'll cache the wheel if you install from PyPI and use it for any subsequent pip installs by just unpacking that zip archive. In your case, you target . so it won't use cached wheels, though.
Back to how an sdist and a wheel, these two hooks call code that is equivalent to commands setup.py sdist and setup.py bdist_wheel.

And so I believe you're looking to redefine those. I saw you attempted to change bdist_wheel but the thing is that setup.py install probably doesn't run it and uses setup.py sdist to get the metadata — you can verify what subcommands are being executed by just carefully looking at the command output.
It's also worth noticing that bdist_wheel is running on top of an sdist usually, so you probably want to explore changing the sdist command since, if you don't include files in your tarball, your wheel won't get those either.

[CareF]

Thank you for the reply!
I'm not redefining sdist because I don't want the generated meta-date to be included in the source install on PyPI. They are expected to be generated on the user side (and being platform dependent, including binary files).
Currently, the only reason I'm still testing setup.py install is for users to have consistent result with pip and setup.py.

So if I understand it correctly, pip install does not call setup.py install because it has its own installing logic, and will use PEP 517, but yet still, it calls setup.py sdist and setup.py bdist_wheel to build for the installation. It seems that, having the customized code piece in bdist_wheel will not be deprecated in the foreseeable versions, will it?

In my test runs, setup.py install is still executed by calling pip install . though. However, its generated data is not copied to the installation directory (.../python3.9/site-packages/...). By looking at the output of pip install . -vvv we can see the following (running blahblah lines):

  running egg_info
    running dist_info
  running bdist_wheel
  running build
  running build_py
  running install
  running install_lib
  running install_egg_info
  running egg_info
  running install_scripts

There is running install and I can confirm that the redefined install script runs, but the generated data is not included in the installation. So is the case if I redefine build or build_py. My interpretation is after bdist_wheel it creates a tarball that is going to the installation, and any dates generated later will not be transport to the installation directory.

[webknjaz]

So if I understand it correctly, pip install does not call setup.py install because it has its own installing logic

Yes, it's one of the deeply nested fallbacks that it may call under very specific conditions when everything else fails.

and will use PEP 517, but yet still, it calls setup.py sdist and setup.py bdist_wheel to build for the installation

It won't call those directly, it's what the pep517 will do but not through subprocesses — it'll invoke the equivalent imported hooks: https://github.com/pypa/setuptools/blob/07eda18/setuptools/build_meta.py#L219-L227.

It seems that, having the customized code piece in bdist_wheel will not be deprecated in the foreseeable versions, will it?

No, it shouldn't be.

There is running install and I can confirm that the redefined install script runs, but the generated data is not included in the installation. So is the case if I redefine build or build_py.

Note that the order is important here. Also, I don't know why build runs after bdist_wheel but I think maybe it's a fallback branch — pip failed to build a wheel probably (there's not enough logs to judge this).

My interpretation is after bdist_wheel it creates a tarball that is going to the installation, and any dates generated later will not be transport to the installation directory.

First of all, bdist_wheel usually produces a whl file with is a regular zip archive, not a tarball. Tarballs are produced by sdist.
I also think that maybe install skips building dists and just emits the results based on egg_info, I don't know off the top of my head. If I were you I'd just stick a debugger in some places and check what's actually happening there...

@pradyunsg should know better how this works.