Maintainers also can use Breeze for other purposes (those are commands that regular contributors likely do not need or have no access to run). Those are usually connected with releasing Airflow:
The outline for this document in GitHub is available at top-right corner button (with 3-dots and 3 lines).
Those are all of the available release management commands:
Running airflow release commands is part of the release procedure performed by the release managers and it is described in detail in dev .
You can prepare airflow packages using Breeze:
breeze release-management prepare-airflow-package
This prepares airflow .whl package in the dist folder.
Again, you can specify optional --package-format
flag to build selected formats of airflow packages,
default is to build both
type of packages sdist
and wheel
.
breeze release-management prepare-airflow-package --package-format=wheel
You can prepare airflow source tarball using Breeze:
breeze release-management prepare-airflow-tarball
This prepares airflow -source.tar.gz package in the dist folder.
You must specify --version
flag which is a pre-release version of Airflow you are preparing the
tarball for.
breeze release-management prepare-airflow-tarball --version 2.8.0rc1
When we create a new minor branch of Airflow, we need to perform a few maintenance tasks. This command automates it.
breeze release-management create-minor-branch
When we prepare release candidate, we automate some of the steps we need to do.
breeze release-management start-rc-process
When we prepare final release, we automate some of the steps we need to do.
breeze release-management start-release
You can use Breeze to generate a airflow core issue when you release new airflow.
The Python client source code can be generated and Python client packages could be built. For that you need to have python client's repository checked out
breeze release-management prepare-python-client --python-client-repo ~/code/airflow-client-python
You can also generate python client with custom security schemes.
These are all of the available flags for the command:
The Production image can be released by release managers who have permissions to push the image. This happens only when there is an RC candidate or final version of Airflow released.
You release "regular" and "slim" images as separate steps.
Releasing "regular" images:
breeze release-management release-prod-images --airflow-version 2.4.0
Or "slim" images:
breeze release-management release-prod-images --airflow-version 2.4.0 --slim-images
By default when you are releasing the "final" image, we also tag image with "latest" tags but this
step can be skipped if you pass the --skip-latest
flag.
These are all of the available flags for the release-prod-images
command:
This command can be utilized to manage git tags for providers within the airflow remote repository during provider releases. Sometimes in cases when there is a connectivity issue to Github, it might be possible that local tags get created and lead to annoying errors. The default behaviour would be to clean such local tags up.
The flag --clean-local-tags
can be used to delete the local tags.
However, If you want to disable this behaviour, set the envvar CLEAN_LOCAL_TAGS to false or use the
--no-clean-local-tags
flag.
breeze release-management tag-providers
These are all of the available flags for the tag-providers
command:
You can prepare helm chart source tarball using Breeze:
breeze release-management prepare-helm-chart-tarball
This prepares helm chart -source.tar.gz package in the dist folder.
You must specify --version
and --version-suffix
flags that specify
which version of Helm Chart you are preparing the tarball for.
breeze release-management prepare-helm-chart-tarball --version 1.12.0 --version-suffix rc1
You can prepare helm chart package and optionally sign it using Breeze:
breeze release-management prepare-helm-chart-package
This prepares helm chart .tar.gz package in the dist folder.
breeze release-management prepare-helm-chart-package --sign [email protected]
You can use Breeze to generate a helm chart issue when you release new helm chart.
Preparing provider release is part of the release procedure by the release managers and it is described in detail in dev .
You can use Breeze to prepare provider documentation.
The below example perform documentation preparation for provider packages.
breeze release-management prepare-provider-documentation
You can also add --answer yes
to perform non-interactive build.
You can use Breeze to prepare provider packages.
The packages are prepared in dist
folder. Note, that this command cleans up the dist
folder
before running, so you should run it before generating airflow package below as it will be removed.
The below example builds provider packages in the wheel format.
breeze release-management prepare-provider-packages
If you run this command without packages, you will prepare all packages, you can however specify
providers that you would like to build. By default both
types of packages are prepared (
wheel
and sdist
, but you can change it providing optional --package-format flag.
breeze release-management prepare-provider-packages google amazon
You can see all providers available by running this command:
breeze release-management prepare-provider-packages --help
In some cases we want to just see if the provider packages generated can be installed with airflow without
verifying them. This happens automatically on CI for sdist pcackages but you can also run it manually if you
just prepared provider packages and they are present in dist
folder.
breeze release-management install-provider-packages
You can also run the verification with an earlier airflow version to check for compatibility.
breeze release-management install-provider-packages --use-airflow-version 2.4.0
All the command parameters are here:
Breeze can also be used to verify if provider classes are importable and if they are following the
right naming conventions. This happens automatically on CI but you can also run it manually if you
just prepared provider packages and they are present in dist
folder.
breeze release-management verify-provider-packages
You can also run the verification with an earlier airflow version to check for compatibility.
breeze release-management verify-provider-packages --use-airflow-version 2.4.0
All the command parameters are here:
The release manager can generate providers metadata per provider version - information about provider versions including the associated Airflow version for the provider version (i.e first airflow version released after the provider has been released) and date of the release of the provider version.
These are all of the available flags for the generate-providers-metadata
command:
You can use Breeze to generate a provider issue when you release new providers.
During the provider releases, we need to clean up the older provider versions in the SVN release folder.
Earlier this was done using a script, but now it is being migrated to a breeze command to ease the life of
release managers for providers. This can be achieved using breeze release-management clean-old-provider-artifacts
command.
These are all available flags of clean-old-provider-artifacts
command:
Whenever pyproject.toml
gets modified, the CI main job will re-generate constraint files. Those constraint
files are stored in separated orphan branches: constraints-main
, constraints-2-0
.
Those are constraint files as described in detail in the ../../../contributing-docs/12_airflow_dependencies_and_extras.rst#pinned-constraint-files contributing documentation.
You can use breeze release-management generate-constraints
command to manually generate constraints for
all or selected python version and single constraint mode like this:
Warning
In order to generate constraints, you need to build all images with --upgrade-to-newer-dependencies
flag - for all python versions.
breeze release-management generate-constraints --airflow-constraints-mode constraints
Constraints are generated separately for each python version and there are separate constraints modes:
- 'constraints' - those are constraints generated by matching the current airflow version from sources
- and providers that are installed from PyPI. Those are constraints used by the users who want to install airflow with pip.
- "constraints-source-providers" - those are constraints generated by using providers installed from current sources. While adding new providers their dependencies might change, so this set of providers is the current set of the constraints for airflow and providers from the current main sources. Those providers are used by CI system to keep "stable" set of constraints.
- "constraints-no-providers" - those are constraints generated from only Apache Airflow, without any providers. If you want to manage airflow separately and then add providers individually, you can use those.
These are all available flags of generate-constraints
command:
In case someone modifies pyproject.toml
, the scheduled CI Tests automatically upgrades and
pushes changes to the constraint files, however you can also perform test run of this locally using
the procedure described in the
Manually generating image cache and constraints
which utilises multiple processors on your local machine to generate such constraints faster.
This bumps the constraint files to latest versions and stores hash of pyproject.toml
. The generated constraint
and pyproject.toml
hash files are stored in the files
folder and while generating the constraints diff
of changes vs the previous constraint files is printed.
Sometimes (very rarely) we might want to update individual packages in constraints that we generated and
tagged already in the past. This can be done using breeze release-management update-constraints
command.
These are all available flags of update-constraints
command:
You can read more details about what happens when you update constraints in the Manually generating image cache and constraints
To publish the documentation generated by build-docs
in Breeze to airflow-site
,
use the release-management publish-docs
command:
breeze release-management publish-docs
The publishing documentation consists of the following steps:
- checking out the latest
main
of clonedairflow-site
- copying the documentation to
airflow-site
- running post-docs scripts on the docs to generate back referencing HTML for new versions of docs
breeze release-management publish-docs <provider id>
Where provider id
is a short form of provider name.
breeze release-management publish-docs amazon
The flag --package-filter
can be used to selectively publish docs during a release. The filters are glob
pattern matching full package names and can be used to select more than one package with single filter.
breeze release-management publish-docs "apache-airflow-providers-microsoft*"
breeze release-management publish-docs --override-versioned
The flag --override-versioned
is a boolean flag that is used to override the versioned directories
while publishing the documentation.
breeze release-management publish-docs --airflow-site-directory
You can also use shorthand names as arguments instead of using the full names for airflow providers. To find the short hand names, follow the instructions in :ref:`generating_short_form_names`.
The flag --airflow-site-directory
takes the path of the cloned airflow-site
. The command will
not proceed if this is an invalid path.
When you have multi-processor machine docs publishing can be vastly sped up by using --run-in-parallel
option when
publishing docs for multiple providers.
These are all available flags of release-management publish-docs
command:
To add back references to the documentation generated by build-docs
in Breeze to airflow-site
,
use the release-management add-back-references
command. This is important to support backward compatibility
the airflow documentation.
You have to specify which packages you run it on. For example you can run it for all providers:
release-management add-back-references --airflow-site-directory DIRECTORY all-providers
The flag --airflow-site-directory
takes the path of the cloned airflow-site
. The command will
not proceed if this is an invalid path.
You can also run the command for apache-airflow (core documentation):
breeze release-management publish-docs --airflow-site-directory DIRECTORY apache-airflow
Also for helm-chart package:
breeze release-management publish-docs --airflow-site-directory DIRECTORY helm-chart
You can also manually specify (it's auto-completable) list of packages to run the command for including individual providers - you can mix apache-airflow, helm-chart and provider packages this way:
breeze release-management publish-docs --airflow-site-directory DIRECTORY apache.airflow apache.beam google
These are all available flags of release-management add-back-references
command:
Maintainers also can use Breeze for SBOM generation:
In order to generate SBOM information for providers, we need to generate requirements for them. This is
done by the generate-providers-requirements
command. This command generates requirements for the
selected provider and python version, using the airflow version specified.
Thanks to our constraints captured for all versions of Airflow we can easily generate SBOM information for
Apache Airflow. SBOM information contains information about Airflow dependencies that are possible to consume
by our users and allow them to determine whether security issues in dependencies affect them. The SBOM
information is written directly to docs-archive
in airflow-site repository.
These are all of the available flags for the update-sbom-information
command:
In order to generate providers requirements, we need docker images with all airflow versions pre-installed,
such images are built with the build-all-airflow-images
command.
This command will build one docker image per python version, with all the airflow versions >=2.0.0 compatible.
The SBOM information published on our website can be converted into a spreadsheet that we are using to analyse security
properties of the dependencies. This is done by the export-dependency-information
command.
Next step: Follow the Advanced Breeze topics to learn more about Breeze internals.
You can prepare airflow packages using Breeze:
breeze release-management prepare-task-sdk-package
This prepares airflow Task SDK .whl package in the dist folder.
Again, you can specify optional --package-format
flag to build selected formats of the Task SDK packages,
default is to build both
type of packages sdist
and wheel
.
breeze release-management prepare-task-sdk-package --package-format=wheel