Feature/devops ci (#3799)

* initial CI tutorial * wip * moved * wip * wip * wip * wip * moved default versioning * products pipeline * wip * final draft * Update devops/versioning/default.rst Co-authored-by: Abril Rincón Blanco <[email protected]> * Update devops/package_promotions.rst Co-authored-by: Abril Rincón Blanco <[email protected]> * Update devops/package_promotions.rst Co-authored-by: Abril Rincón Blanco <[email protected]> * Update devops/devops.rst Co-authored-by: Abril Rincón Blanco <[email protected]> * Update ci_tutorial/tutorial.rst Co-authored-by: Abril Rincón Blanco <[email protected]> * Update ci_tutorial/tutorial.rst Co-authored-by: Abril Rincón Blanco <[email protected]> * Update ci_tutorial/packages_pipeline.rst Co-authored-by: Carlos Zoido <[email protected]> * Update ci_tutorial/packages_pipeline/multi_configuration_lockfile.rst Co-authored-by: Carlos Zoido <[email protected]> * Update ci_tutorial/tutorial.rst Co-authored-by: Abril Rincón Blanco <[email protected]> * Update ci_tutorial/products_pipeline/distributed_build.rst Co-authored-by: Abril Rincón Blanco <[email protected]> * Apply suggestions from code review Co-authored-by: Carlos Zoido <[email protected]> Co-authored-by: Michael Farrell <[email protected]> Co-authored-by: Abril Rincón Blanco <[email protected]> Co-authored-by: Artalus <[email protected]> * review * multiline cmdlines -> singleline * Update devops/package_promotions.rst Co-authored-by: Michael Farrell <[email protected]> * review * review * Update ci_tutorial/products_pipeline/multi_product.rst Co-authored-by: Abril Rincón Blanco <[email protected]> * lockfile storing * final remarks * Update ci_tutorial/products_pipeline/full_pipeline.rst Co-authored-by: Abril Rincón Blanco <[email protected]> * Update ci_tutorial/products_pipeline/full_pipeline.rst Co-authored-by: Abril Rincón Blanco <[email protected]> --------- Co-authored-by: Abril Rincón Blanco <[email protected]> Co-authored-by: Carlos Zoido <[email protected]> Co-authored-by: Michael Farrell <[email protected]> Co-authored-by: Artalus <[email protected]>
conan-io · Dec 10, 2024 · 43cc8b9 · 43cc8b9
1 parent 23ed5d6
commit 43cc8b9
Show file tree

Hide file tree

Showing 21 changed files with 1,779 additions and 25 deletions.
diff --git a/ci_tutorial/packages_pipeline.rst b/ci_tutorial/packages_pipeline.rst
@@ -0,0 +1,41 @@
+Packages pipeline
+==================
+
+
+The **packages pipeline** will build, create and upload the package binaries for the different configurations and platforms, when some
+developer is submitting some changes to one of the organization repositories source code. For example if a developer is doing some changes
+to the ``ai`` package, improving some of the library functionality, and bumping the version to ``ai/1.1.0``. If the organization needs to
+support both Windows and Linux platforms, then the package pipeline will build the new ``ai/1.1.0`` both for Windows and Linux, before
+considering the changes are valid. If some of the configurations fail to build under a specific platform, it is common to consider the
+changes invalid and stop the processing of those changes, until the code is fixed.
+
+
+For the ``package pipeline`` we will start with a simple source code change in the ``ai`` recipe, simulating some improvements
+in the ``ai`` package, providing some better algorithms for our game.
+
+✍️ **Let's do the following changes in the ai package**:
+
+- Let's change the implementation of the ``ai/src/ai.cpp`` function and change the message from ``Some Artificial`` to ``SUPER BETTER Artificial``
+- Let's change the default ``intelligence=0`` value in ``ai/include/ai.h`` to a new ``intelligence=50`` default.
+- Finally, let's bump the version. As we did some changes to the package public headers, it would be adviced to bump the ``minor`` version,
+  so let`s edit the ``ai/conanfile.py`` file and define ``version = "1.1.0"`` there (instead of the previous ``1.0``). Note that if we
+  did some breaking changes to the ``ai`` public API, the recommendation would be to change the major instead and create a new ``2.0`` version.
+
+
+The **packages pipeline** will take care of building the different packages binaries for the new ``ai/1.1.0`` and upload them to the ``packages``
+binary repository to avoid disrupting or causing potential issues to other developers and CI jobs. 
+If the pipeline succeed it will promote (copy) them to the ``products`` binary repository, and stop otherwise.
+
+There are different aspects that need to be taken into account when building these binary packages for ``ai/1.1.0``. The following tutorial sections do the same
+job, but under different hypothesis. They are explained in increasing complexity.
+
+Note all of the commands can be found in the repository ``run_example.py`` file. This file is mostly intended for maintainers and testing,
+but it might be useful as a reference in case of issues.
+
+
+.. toctree::
+   :maxdepth: 1
+
+   packages_pipeline/single_configuration
+   packages_pipeline/multi_configuration
+   packages_pipeline/multi_configuration_lockfile
diff --git a/ci_tutorial/packages_pipeline/multi_configuration.rst b/ci_tutorial/packages_pipeline/multi_configuration.rst
@@ -0,0 +1,198 @@
+Package pipeline: multi configuration
+=====================================
+
+In the previous section we were building just 1 configuration. This section will cover the case in which we need to build more
+than 1 configuration. We will use the ``Release`` and ``Debug`` configurations here for convenience, as it is easier to 
+follow, but in real case these configurations will be more like Windows, Linux, OSX, building for different architectures, 
+cross building, etc.
+
+Let's begin cleaning our cache:
+
+.. code-block:: bash
+
+    $ conan remove "*" -c  # Make sure no packages from last run
+
+We will create the packages for the 2 configurations sequentially in our computer, but note these will typically run
+in different computers, so it is typical for CI systems to launch the builds of different configurations in parallel.
+
+.. code-block:: bash
+    :caption: Release build
+
+    $ cd ai  # If you were not inside "ai" folder already
+    $ conan create . --build="missing:ai/*" -s build_type=Release --format=json > graph.json
+    $ conan list --graph=graph.json --graph-binaries=build --format=json > built.json
+
+    $ conan remote enable packages
+    $ conan upload -l=built.json -r=packages -c --format=json > uploaded_release.json
+    $ conan remote disable packages
+
+We have done a few changes and extra steps:
+
+- First step is similar to the one in the previous section, a ``conan create``, just making it explicit our configuration
+  ``-s build_type=Release`` for clarity, and capturing the output of the ``conan create`` in a ``graph.json`` file.
+- The second step is create from the ``graph.json`` a ``built.json`` **package list** file, with the packages that needs to be uploaded,
+  in this case, only the packages that have been built from source (``--graph-binaries=build``) will be uploaded. This is
+  done for efficiency and faster uploads.
+- Third step is to enable the ``packages`` repository. It was not enabled to guarantee that al possible dependencies came from ``develop``
+  repo only.
+- Then, we will upload the ``built.json`` package list to the ``packages`` repository, creating the ``uploaded_release.json``
+  package list with the new location of the packages (the server repository).
+- Finally, we will disable again the ``packages`` repository
+
+Likewise, the Debug build will do the same steps:
+
+
+.. code-block:: bash
+    :caption: Debug build
+
+    $ conan create . --build="missing:ai/*" -s build_type=Debug --format=json > graph.json
+    $ conan list --graph=graph.json --graph-binaries=build --format=json > built.json
+
+    $ conan remote enable packages
+    $ conan upload -l=built.json -r=packages -c --format=json > uploaded_debug.json
+    $ conan remote disable packages
+
+
+When both Release and Debug configuration finish successfully, we would have these packages in the repositories:
+
+.. graphviz::
+    :align: center
+
+    digraph repositories {
+        node [fillcolor="lightskyblue", style=filled, shape=box]
+        rankdir="LR"; 
+        subgraph cluster_0 {
+                label="Packages server";
+                style=filled;
+                color=lightgrey;
+                subgraph cluster_1 {
+                label = "packages\n repository" 
+                shape = "box";
+                style=filled;
+                color=lightblue;
+                "packages" [style=invis];
+                "ai/1.1.0\n (Release)";
+                "ai/1.1.0\n (Debug)";
+                }
+                subgraph cluster_2 {
+                label = "products\n repository" 
+                shape = "box";
+                style=filled;
+                color=lightblue;
+                "products" [style=invis];
+                } 
+                subgraph cluster_3 {
+                rankdir="BT";
+                shape = "box";
+                label = "develop repository";
+                color=lightblue;
+                rankdir="BT";
+
+                node [fillcolor="lightskyblue", style=filled, shape=box]
+                "game/1.0" -> "engine/1.0" -> "ai/1.0" -> "mathlib/1.0";
+                "engine/1.0" -> "graphics/1.0" -> "mathlib/1.0";
+                "mapviewer/1.0" -> "graphics/1.0";
+                "game/1.0" [fillcolor="lightgreen"];
+                "mapviewer/1.0" [fillcolor="lightgreen"];
+                }
+                {
+                edge[style=invis];
+                "packages" -> "products" -> "game/1.0" ; 
+                rankdir="BT";    
+                }
+        }
+    }
+
+
+When all the different binaries for ``ai/1.1.0`` have been built correctly, the ``package pipeline`` can consider its job succesfull and decide
+to promote those binaries. But further package builds and checks are necessary, so instead of promoting them to the ``develop`` repository,
+the ``package pipeline`` can promote them to the ``products`` binary repository. As all other developers and CI use the ``develop`` repository,
+no one will be broken at this stage either:
+
+.. code-block:: bash
+    :caption: Promoting from packages->product
+
+    # aggregate the package list
+    $ conan pkglist merge -l uploaded_release.json -l uploaded_debug.json --format=json > uploaded.json
+
+    $ conan remote enable packages
+    $ conan remote enable products
+    # Promotion using Conan download/upload commands 
+    # (slow, can be improved with art:promote custom command)
+    $ conan download --list=uploaded.json -r=packages --format=json > promote.json
+    $ conan upload --list=promote.json -r=products -c
+    $ conan remote disable packages
+    $ conan remote disable products
+
+
+The first step uses the ``conan pkglist merge`` command to merge the package lists from the "Release" and "Debug" configurations and 
+merge it into a single ``uploaded.json`` package list.
+This list is the one that will be used to run the promotion.
+
+In this example we are using a slow ``conan download`` + ``conan upload`` promotion. This can be way more efficient with 
+the ``conan art:promote`` extension command.
+
+After running the promotion we will have the following packages in the server:
+
+.. graphviz::
+    :align: center
+
+    digraph repositories {
+        node [fillcolor="lightskyblue", style=filled, shape=box]
+        rankdir="LR"; 
+        subgraph cluster_0 {
+                label="Packages server";
+                style=filled;
+                color=lightgrey;
+                subgraph cluster_1 {
+                label = "packages\n repository" 
+                shape = "box";
+                style=filled;
+                color=lightblue;
+                "packages" [style=invis];
+                "ai/1.1.0\n (Release)";
+                "ai/1.1.0\n (Debug)";
+                }
+                subgraph cluster_2 {
+                label = "products\n repository" 
+                shape = "box";
+                style=filled;
+                color=lightblue;
+                "products" [style=invis];
+                "ai/promoted release" [label="ai/1.1.0\n (Release)"];
+                "ai/promoted debug" [label="ai/1.1.0\n (Debug)"];
+                } 
+                subgraph cluster_3 {
+                rankdir="BT";
+                shape = "box";
+                label = "develop repository";
+                color=lightblue;
+                rankdir="BT";
+
+                node [fillcolor="lightskyblue", style=filled, shape=box]
+                "game/1.0" -> "engine/1.0" -> "ai/1.0" -> "mathlib/1.0";
+                "engine/1.0" -> "graphics/1.0" -> "mathlib/1.0";
+                "mapviewer/1.0" -> "graphics/1.0";
+                "game/1.0" [fillcolor="lightgreen"];
+                "mapviewer/1.0" [fillcolor="lightgreen"];
+                }
+                {
+                edge[style=invis];
+                "packages" -> "products" -> "game/1.0" ; 
+                rankdir="BT";    
+                }
+        }
+    }
+
+
+To summarize:
+
+- We built 2 different configurations, ``Release`` and ``Debug`` (could have been Windows/Linux or others), and uploaded them
+  to the ``packages`` repository.
+- When all package binaries for all configurations were successfully built, we promoted them from the ``packages`` to the
+  ``products`` repository, to make them available for the ``products pipeline``.
+- **Package lists** were captured in the package creation process and merged into a single one to run the promotion.
+
+
+There is still an aspect that we haven't considered yet, the possibility that the dependencies of ``ai/1.1.0`` change
+during the build. Move to the next section to see how to use lockfiles to achieve more consistent multi-configuration builds.
diff --git a/ci_tutorial/packages_pipeline/multi_configuration_lockfile.rst b/ci_tutorial/packages_pipeline/multi_configuration_lockfile.rst
@@ -0,0 +1,148 @@
+Package pipeline: multi configuration using lockfiles
+=====================================================
+
+In the previous example, we built both ``Debug`` and ``Release`` package binaries for ``ai/1.1.0``. In real world scenarios the binaries to build would be different platforms (Windows, Linux, embedded), different architectures, and very often it will not be possible to build them in the same machine, requiring different computers.
+
+The previous example had an important assumption: the dependencies of ``ai/1.1.0`` do not change at all during the building process. In many scenarios, this assumption will not hold, for example if there are any other concurrent CI jobs, and one succesfull job publishes a new ``mathlib/1.1`` version in the ``develop`` repo. 
+
+Then it is possible that one build of ``ai/1.1.0``, for example, the one running in the Linux servers starts earlier and uses the previous ``mathlib/1.0`` version as dependency, while the Windows servers start a bit later, and then their build will use the recent ``mathlib/1.1`` version as dependency. This is a very undesirable situation, having binaries for the same ``ai/1.1.0`` version using different dependencies versions. This can lead in later graph resolution problems, or even worse, get to the release with different behavior for different platforms.
+
+The way to avoid this discrepancy in dependencies is to force the usage of the same dependencies versions and revisions, something that can be done with :ref:`lockfiles<tutorial_versioning_lockfiles>`.
+
+Creating and applying lockfiles is relatively straightforward. The process of creating and promoting the configurations will be identical to the previous section, but just applying the lockfiles.
+
+Creating the lockfile
+---------------------
+
+Let's make sure as usual that we start from a clean state:
+
+.. code-block:: bash
+
+    $ conan remove "*" -c  # Make sure no packages from last run
+
+
+Then we can create the lockfile ``conan.lock`` file:
+
+.. code-block:: bash
+
+    # Capture a lockfile for the Release configuration
+    $ conan lock create . -s build_type=Release --lockfile-out=conan.lock
+    # extend the lockfile so it also covers the Debug configuration
+    # in case there are Debug-specific dependencies
+    $ conan lock create . -s build_type=Debug --lockfile=conan.lock --lockfile-out=conan.lock
+
+Note that different configurations, using different profiles or settings could result in different dependency graphs. A lockfile file can be used to lock the different configurations, but it is important to iterate the different configurations/profiles and capture their information in the lockfile.
+
+.. note::
+
+    The ``conan.lock`` is the default argument, and if a ``conan.lock`` file exists, it might be automatically used by ``conan install/create`` and other graph commands. This can simplify many of the commands, but this tutorial is showing the full explicit commands for clarity and didactical reasons.
+
+The ``conan.lock`` file can be inspected, it will be something like:
+
+.. code-block:: json
+
+    {
+        "version": "0.5",
+        "requires": [
+            "mathlib/1.0#f2b05681ed843bf50d8b7b7bdb5163ea%1724319985.398"
+        ],
+        "build_requires": [],
+        "python_requires": [],
+        "config_requires": []
+    }
+    
+As we can see, it is locking the ``mathlib/1.0`` dependency version and revision.
+
+
+With the lockfile, creating the different configurations is exactly the same, but providing the ``--lockfile=conan.lock`` argument to the ``conan create`` step, it will guarantee that ``mathlib/1.0#f2b05681ed843bf50d8b7b7bdb5163ea`` will always be the exact dependency used, irrespective if there exist new ``mathlib/1.1`` versions or new revisions available. The following builds could be launched in parallel but executed at different times, and still they will always use the same ``mathlib/1.0`` dependency:
+
+
+.. code-block:: bash
+    :caption: Release build
+
+    $ cd ai  # If you were not inside "ai" folder already
+    $ conan create . --build="missing:ai/*" --lockfile=conan.lock -s build_type=Release --format=json > graph.json
+    $ conan list --graph=graph.json --graph-binaries=build --format=json > built.json
+    $ conan remote enable packages
+    $ conan upload -l=built.json -r=packages -c --format=json > uploaded_release.json
+    $ conan remote disable packages
+
+.. code-block:: bash
+    :caption: Debug build
+
+    $ conan create . --build="missing:ai/*" --lockfile=conan.lock -s build_type=Debug --format=json > graph.json
+    $ conan list --graph=graph.json --graph-binaries=build --format=json > built.json
+    $ conan remote enable packages
+    $ conan upload -l=built.json -r=packages -c --format=json > uploaded_debug.json
+    $ conan remote disable packages
+
+Note the only modification to the previous example is the addition of ``--lockfile=conan.lock``. The promotion will also be identical to the previous one:
+
+.. code-block:: bash
+    :caption: Promoting from packages->product
+
+    # aggregate the package list
+    $ conan pkglist merge -l uploaded_release.json -l uploaded_debug.json --format=json > uploaded.json
+    
+    $ conan remote enable packages
+    $ conan remote enable products
+    # Promotion using Conan download/upload commands 
+    # (slow, can be improved with art:promote custom command)
+    $ conan download --list=uploaded.json -r=packages --format=json > promote.json
+    $ conan upload --list=promote.json -r=products -c
+    $ conan remote disable packages
+    $ conan remote disable products
+
+And the final result will be the same as in the previous section, but this time just with the guarantee that both ``Debug`` and ``Release`` binaries were built using exactly the same ``mathlib`` version:
+
+.. graphviz::
+    :align: center
+
+    digraph repositories {
+        node [fillcolor="lightskyblue", style=filled, shape=box]
+        rankdir="LR"; 
+        subgraph cluster_0 {
+                label="Packages server";
+                style=filled;
+                color=lightgrey;
+                subgraph cluster_1 {
+                label = "packages\n repository" 
+                shape = "box";
+                style=filled;
+                color=lightblue;
+                "packages" [style=invis];
+                "ai/1.1.0\n (Release)";
+                "ai/1.1.0\n (Debug)";
+                }
+                subgraph cluster_2 {
+                label = "products\n repository" 
+                shape = "box";
+                style=filled;
+                color=lightblue;
+                "products" [style=invis];
+                "ai/promoted release" [label="ai/1.1.0\n (Release)"];
+                "ai/promoted debug" [label="ai/1.1.0\n (Debug)"];
+                } 
+                subgraph cluster_3 {
+                rankdir="BT";
+                shape = "box";
+                label = "develop repository";
+                color=lightblue;
+                rankdir="BT";
+
+                node [fillcolor="lightskyblue", style=filled, shape=box]
+                "game/1.0" -> "engine/1.0" -> "ai/1.0" -> "mathlib/1.0";
+                "engine/1.0" -> "graphics/1.0" -> "mathlib/1.0";
+                "mapviewer/1.0" -> "graphics/1.0";
+                "game/1.0" [fillcolor="lightgreen"];
+                "mapviewer/1.0" [fillcolor="lightgreen"];
+                }
+                {
+                edge[style=invis];
+                "packages" -> "products" -> "game/1.0" ; 
+                rankdir="BT";    
+                }
+        }
+    }
+
+Now that we have the new ``ai/1.1.0`` binaries in the ``products`` repo, we can consider the ``packages pipeline`` finished and move to the next section, and build and check our products to see if this new ``ai/1.1.0`` version integrates correctly.