Skip to content

Latest commit

 

History

History
176 lines (119 loc) · 9.41 KB

Developing.md

File metadata and controls

176 lines (119 loc) · 9.41 KB
Raw

Developing thonny-ev3dev plugin

Setup plugin project with multiple context projects (each a git project)

To develop a project we sometimes need to have several related projects also set up.

For example, the thonny-ev3dev plugin for thonny uses the ev3devcmd library to implement most of its functionality. But being a plugin for thonny, it of course also requires the Thonny package which implements the Thonny IDE for which this project is a plugin. In this case, the plugin depends on the IDE.

The related projects can be a project your project depends on, or can be a project your project needs as a dependency. Both cases are in context of your project, so therefore we call them 'context projects'.

When you are developing in a project, a change in the project can require also a change in a context project. Therefore it would be convenient as a developer that you can edit in both projects in your setup.

To allow developing in the thonny-ev3dev plugin and its context projects we set up the thonny-ev3dev project as the main project and setup the context projects within the main project. Then we can work in the main and its context projects in parallel, each having its own .git folder to do commits to its own repository. All modern ide's recognise the .git folder of the main project and the context projects! So we can easily commit changes done at each git repo from the IDE also.

Instructions to manually setup:

  • checkout from git the main project (thonny-ev3dev)

  • In python we do not have project support which set ups a sandbox environment for you like in languages like Rust. Instead you need to create this environment for your project self with an virtual env tool.

     python3 -m venv .venv 
 source .venv/bin/activate

    Our project is now isolated/sandboxed. We only install the dependencies of this project in this virtual environment. We don't add the virtual env environment to the git project because it can become very large, and it can easy be replicated.

  • For every new bash shell we need to load this environment with:

     source .venv/bin/activate

  • within the main project make a subdir context/
    we choose to have the context projects as subdirs within the main project.

  • add context/ to .gitignore of the main project

    • NOTE: context/ shouldn't be in git project. The context projects in the context/ folder are installed only locally because we want to work on the main and its context projects at the same time in a single project!! The main project, and the context projects all have their own git project independent of each other.

  • within context/ checkout from git each context project:

    Note: we clone each subproject ourselves because that is easier and less confusing than using the submodules feature in git itself.

  • configure context projects to be used by main project:

    1. all projects must be on same PYTHONPATH so that they are available to each other in the same python runtime.
    2. all projects dependencies must be installed

  • there are several ways to do this, but currently the best method is using editable installs:

    • install each context project as an editable install
    • also install the main module as an editable install
    • source code of the editable install, which is local in our folder, is by using this special install made available on the PYTHONPATH. So all projects are available in the same python runtime!
    • ADVANTAGE above conventional method: because an editable install also installs its dependencies
    • ADVANTAGE: this way is python's own way and is not dependent of any IDE
    • all modern IDE's support editable installs
    • note: when using the IDEA IDE do the editable install with the IDE and not with pip, because somehow editable installs done by pip are not picked up by the IDE. vscode however works fine with editable installs done by pip.

  • We use pip-tools to automatically install all dependencies, including the editable installs, for us. The 'requirements.txt' file in the repository is generated by running:

     pip-compile pyproject.toml context.txt

    Where 'pyproject.toml' contains the dependencies of the project in production. The file 'context.txt' contains our editable installs. For each project X it contains a line '-e ./dir/to/project/X'. Finally the file 'context.txt' needs a line '-e .' to also install the main project as editable install. Editable install of the main project is required if your project uses the src-layout. Without an editable install your python files are in the src/ subdirectory and therefore won't be available on the PYTHONPATH automatically. The editable install of your main project solves that!

  • An editable install X in context.txt overrules a normal install X in pyproject.toml. The 'pip-compile' command generates a list of all dependencies of the project where each dependency is locked to a specific version. Using the lock file we can install the project with the exact same dependencies on different develop machines.

  • On the first time we setup the project on a new machine we have to create a new virtual environment for the machine:

     python3 -m venv .venv 
 source .venv/bin/activate                   
 pip install pip-tools build
 pip-sync

    Here 'pip-sync', using the requirements.txt file in your project, automatically installs all normal and editable installs in our virtual environment.

  • If during development we change our project dependencies we have to run the pip-compile and pip-sync commands again to get the virtual environment updated with the new depencies setup. If some dependency is not needed anymore, then 'pip-sync' will delete it for us from the virtual environment.

  • Finally we must remark that the requirements.txt lockfile is specific per platform! So if you want to develop on multiple different platforms, then you have to create a requirements.txt per platform.

    How can we create multiple lockfiles which have the same versions for cross-platform packages, and only differ in platform specific packages? The answer is by stripping all platform specific packages from a specific platform's requirements.txt file, and then use the resulting file as a constraints file for pip-compile to create a new requirements.txt lockfile on a different platform, which then only differs in platform specific packages.

    But how do we know which packages are platform specific and must be stripped? The answer is just by running pip-sync on the requirements.txt from the different platform, then all packages giving errors when installing, are packages which are specific for the different platform. So by stripping all these error packages we stripped away all platform specific packages. The resulting file we can then use as the constraints file.

    UPDATE: using the constraints option of pip-compile we can more easily create a requirements.txt file for platform NEW from platform OLD using the command:

      pip-compile -c requirements.OLD.txt  pyproject.toml context.txt

    All dependencies which both platforms use, have the same versions, because of the constraint. Only for dependencies specific for the NEW platform the pip-compile will add new entries. Dependencies specific for the OLD platform are not added, because they are not needed for the new platform. Their entries in the contstraint file are just ignored.

Automated setup:

The repository contains a pyproject.bash script from the project https://github.com/harcokuppens/pyproject.bash.git which helps you running above commands more easily. Instead of requirements.txt the pyproject.bash script uses as default lockfile lockfile.txt. This name better describes it purpose. Using the name requirements.txt is confusing, because it is used by pip-sync as a lockfile, whereas the real requirements are in the pyproject.toml file.

Hence, we can easily setup the developing environment for thonny-ev3dev by running in a bash shell:

  1. clone main project 'thonny-ev3dev'

      git clone https://github.com/ev3dev-python-tools/thonny-ev3dev.git
  cd thonny-ev3dev

  2. clone all git repositories for context projects

     source git_clone_all_projects.bash

  3. reactivate project environment with context projects note: the git repo already has a 'lockfile.txt' lockfile

     source pyproject.bash reactivate

We now have an active virtual environment using the same python and package versions as describe in the lockfile. We can then start debugging the plugin by running in an IDE:

	python context/thonny/thonny/__main__.py

It then uses the source in your main project and all the sources of your context projects.

Note: above we assume that we are only developing on a specific operating system/platform where we always can use the same lockfile.txt file as lockfile.