how to execute a script that will modify my package at build time?

[12rambau]

in ipyvuetify, we have the following package structure:

.
├── genreate_source/
│   ├── generate_schema.py
│   ├── generate_source.py
│   └── __init__.py
├── ipyvuetify/
│   ├── template.py
│   ├── Html.py
│   ├── Themes.py
│   ├── Template.py
│   └── _init__.py
├── pyproject.toml
└── setup.py

The main package is ipyvuetify but the content is generated from the vuetify.js API. To buid the source, we need to execute a function that live in generate_source.py. Until now we were adding these source at release time but that makes developping in the lib complicated. We try to make the build of the widget automatic and included in the python build.

When we try to importe "genreate_source" module in the setup.py it doesn't work (moduleNotFound). Which prevent us from updating the cmdclass Is it normal and can we call this script from septup ?

in case it's an xy issue, is there a better way to include this content in the distrib at build time without storing it in the distant repository ?

[abravalheri]

Hi @12rambau maybe a custom build sub-command would work for you?

Something like:

# setup.py
import os
import sys
from setuptools immport Command, setup
from setuptools.command.build import build

sys.path.append(os.path.dirname(__file__))

from genreate_source import generate_schema, generate_source


class custom_build(build):
    sub_commands = [("build_vuetify", None), *build.sub_commands]


class build_vuetify(build):
    # See interface in https://setuptools.pypa.io/en/latest/userguide/extension.html

    def initialize_options(self):
        self.editable_mode = False
        self.build_lib = None
        ...

    def finalize_options(self):
        self.set_undefined_options('build', ('build_lib', 'build_lib'))
        ...

    def get_source_files(self): ...
    def get_outputs(self): ...
    def get_output_mapping(self): ...
    def run(self): ...


setup(
   ...
   cmdclass={"build": custom_build, "build_vuefy": build_vuetify},
)

# Untested code, written from memory, please review :P

[12rambau]

thanks @abravalheri. This is exactly what we want to do. the only issue for us is that the "generate" method lives in another folder (not something we can change because of the webpack binding). So the main difference between your code and mine (https://github.com/12rambau/ipyvuetify/blob/install/setup.py) is that I try to call a script from another file.

When doing so the error is "moduleNotFound" even though the package is just here next to setup.py. Is there a way to import it to setup.py or is there something I misendurstood about installation steps ?

[abravalheri]

Hi @12rambau, have you trying modifying sys.path before importing the extra code as in my example above? Probably that is the main point in terms of importing.

The other thing I think the example differs is that when using sub_commands the build step will be called automatically during the build. In your example I see that you are using a decorator instead (which is also a valid approach).

[12rambau]

your suggestion seems to work. Thanks for point it out.

I think your solution would be better as I would get more control on the output. The initial code was also overwritting build_py, sdist and egg_info wich seems to much to me.

Just to better understand what is happening here:

• build command is called by python -m build to produce a wheel
• when loading a wheel from say pypi, the build step is ignored ? (as everything should already be available)
• in -e is there a recommanded way to call these command and update the deps ?

[abravalheri]

Hi @12rambau

• build command is called by python -m build to produce a wheel
• when loading a wheel from say pypi, the build step is ignored ? (as everything should already be available)

Yes, this is how it works.

in -e is there a recommended way to call these command and update the deps ?

Not sure if I understood this correctly, but if you add a dependency you probably will have to run pip install -e . again.

In general the following is true:

• Pre-existing .py files that are modified should be automatically picked up (if you restart the console).
• Pre-existing data files inside the package directory should be automatically picked up.

Other more complex stuff might require you to run pip install -e . again.

If you want to "re-generate" some files on the fly you might run pip install -e . again or have some form of background process that modify existing files.

[12rambau]

cristal clear, thanks a lot.
I'll read carefully this documentation page https://setuptools.pypa.io/en/latest/userguide/extension.html to better grasp how I should set the created files and that should do it.

[12rambau]

sorry for asking again about my specific issue. I implemented your suggestions and I'm still facing a challenge. Now I manage to execute NPM packaging operations but I failed to complete the build command. One of the genering operation is populating a js/generated folder.
Upon running python -m build the folder is effectively created but in the command error with the following:

error: [Errno 2] No such file or directory: '/tmp/build-via-sdist-b78q40kt/ipyvuetify-1.8.10/js'

Diving into my commands it's normal because files where built in /home/ not in tmp/. How can I move these files to the tmp/ build directory after they are created ?

[abravalheri]

Hi @12rambau I am not sure if I understood what is going on... Do you have a branch that you have been working with that I could have a look? Or could you share your build logs?

Are you using something like self.build_lib to find out during runtime the build directory?

[12rambau]

sure tthing, I'm working on this branch: https://github.com/12rambau/ipyvuetify/tree/install

Are you using something like self.build_lib to find out during runtime the build directory?

nope, I don't even know how I should

[abravalheri]

Thanks @12rambau.

There are a few things to have in mind about the build process with python -m build:

1. By default, python -m build will create a sdist first, and then create a wheel from that sdist.
   This means that you need to make sure that all the files needed to build the wheel are present in the sdist.
2. python -m build not necessarily uses the project directory to store intermediate/generated files.
   It is very likely it will use a temporary directory.

So in this case I would double check that the existing MANIFEST.in is configured correctly to include all the necessary folders for the build to take place (and it seems that the js folder is not included).

[abravalheri]

If sorting out MANIFEST.in is not enough to solve this problem, I would also consider using the build_js.build_lib attribute, that tells which directory is being used for the build. Maybe you could compute destination_js and destionation_python directories that you are using in your generate function? Another approach would be copying the files over to the build_js.build_lib directory.

[12rambau]

sorting the manifest.in seems to do the trick. But if I understood your point correctly, I should only decorate the sdist command right ?