diff --git a/source/conf.py b/source/conf.py index 6496a3b0a..5a5a87b0d 100644 --- a/source/conf.py +++ b/source/conf.py @@ -336,6 +336,11 @@ # Config for sphinx-reredirects, maps source: target, target path is relative to source. # TODO: troubleshooting sections redirecting to lmp-customization do not appear to be functional redirects = { + "reference-manual/docker/compose-apps": "../../user-guide/containers-and-docker/compose-apps.html", + "reference-manual/docker/configure-docker-helper": "../../user-guide/containers-and-docker/configure-docker-helper.html", + "reference-manual/docker/containers": "../../user-guide/containers-and-docker/containers.html", + "user-guide/containers-preloading/container-preloading": "../containers-and-docker/container-preloading.html", + "user-guide/multi-stage-container/multi-stage-container": "../containers-and-docker/multi-stage-container", "reference-manual/factory/team-based-access": "../../user-guide/account-management/team-based-access.html", "reference-manual/factory/factory-keys": "../../reference-manual/security/factory-keys.html", "reference-manual/security/secure-boot": "security.html#secure-boot-hardware-root-of-trust", diff --git a/source/index.rst b/source/index.rst index 5575362d6..f54dc596d 100644 --- a/source/index.rst +++ b/source/index.rst @@ -38,7 +38,7 @@ OE/Yocto Project, the Linux microPlatform™ and Docker®. :name: sec-user-guide user-guide/account-management/account-management - user-guide/container-preloading/container-preloading + user-guide/containers-and-docker/index user-guide/custom-ci/custom-ci user-guide/lmp-customization/lmp-customization user-guide/lmp-auto-hostname/lmp-auto-hostname @@ -47,7 +47,6 @@ OE/Yocto Project, the Linux microPlatform™ and Docker®. user-guide/mirror-action/mirror-action user-guide/submodule/submodule user-guide/custom-sota-client - user-guide/multi-stage-container/multi-stage-container user-guide/fioctl/index user-guide/cert-rotation user-guide/device-gateway-pki/device-gateway-pki diff --git a/source/reference-manual/docker/caching.rst b/source/reference-manual/docker/caching.rst index 9c418b7cd..79a151cfe 100644 --- a/source/reference-manual/docker/caching.rst +++ b/source/reference-manual/docker/caching.rst @@ -3,31 +3,26 @@ Caching ======= -Each container built in a FoundriesFactory will publish the build cache layers -to our private registry, hub.foundries.io. Subsequent builds will pull from this -cache, importing it for the current build. This provides efficient incremental -container builds for our FoundriesFactory users. Exporting and importing -these build cache layers uses the built-in features of -`Docker Buildx. `_ +Each container built in a Factory publishes the build cache layers to our private registry, ``hub.foundries.io``. +Subsequent builds pull/import from this cache. +This provides efficient, incremental container builds. +This exporting and importing of build cache layers uses built-in features of `Docker Buildx `_. -While the cache is very helpful, there are a few things to note: +While the cache is useful, there are a few things to note: - * The build cache for each container image is branch specific - * It is architecture specific - * It can be invalidated for a few reasons, such as source files changing - or a base image update. + * The build cache for each container image is *branch specific* + * It is *architecture specific* + * It can be invalidated for a few reasons, such as source files changing or a base image update. Cache Invalidation ------------------ -When trying to understand why the cache has been invalidated, there are not any -tools that can assist. That said, in a FoundriesFactory each container build -will automatically use the docker build context to create a list of md5sums of all -source files used in the build. Each :doc:`compose-apps` will generate an artifact -named ``-md5sum.txt``, which then can be used to -generate a ``diff`` from build to build to assist in understanding what files may -have changed, and how that effects the caching. Generally, each line in your ``Dockerfile`` -creates it's own image layer, and a corresponding cache layer. If source files change, -or any direct modification to the ``Dockerfile`` occurs, it will invalidate the cache from -that point forward. With the importance of these concepts towards building :doc:`compose-apps`, -it is recommended to read the documentation related to `Dockerfile best practices. `_ +When trying to understand why a cache has been invalidated, there is not any tools that can assist. +That said, in a Factory, each container build automatically uses the Docker build context to create a list of md5sums of all source files used. + +Each :ref:`Compose App ` generates an artifact named ``-md5sum.txt``. +This can then be used to generate a ``diff`` between builds, assisting in understanding what files may have changed, and how the cache may have been effected. + +Generally, each line of your ``Dockerfile`` creates an image layer with a corresponding cache layer. +If source files change, or any direct modification to the ``Dockerfile`` occurs, it will invalidate the cache from that point forward. +Due to the importance of these concepts towards building :ref:`ref-compose-apps`, it is recommended to read the `Dockerfile best practices. `_. diff --git a/source/reference-manual/docker/container-secrets.rst b/source/reference-manual/docker/container-secrets.rst index c67335080..6bd1477c4 100644 --- a/source/reference-manual/docker/container-secrets.rst +++ b/source/reference-manual/docker/container-secrets.rst @@ -3,49 +3,45 @@ Using Secret Credentials When Building Containers ================================================= -There are many cases when building a container that sensitive credentials -may be required. Examples include: +There are many cases when building a container that sensitive credentials may be required. +Examples include: * Downloading a package from a private NPM registry - * Grabbing a file via scp + * Grabbing a file via ``scp``. - Each scenario requires a slightly different approach in the Dockerfile. - However, the general approach to getting these secrets securely into your - container's build context is the same. +Each scenario requires a slightly different approach in the Dockerfile. +However, the general approach to getting these secrets securely into your container's build context is the same. .. note:: - There are several `insecure ways`_ that should be avoid in order to do this - correctly. The proper approach is what's described here based on new - functionality in Docker's BuildKit. + There are several `insecure ways`_ that should be avoided. + The proper approach is described here based on new functionality in Docker's BuildKit. .. _insecure ways: https://pythonspeed.com/articles/docker-build-secrets/ -Quick Background On CI Secrets ------------------------------- +Overview of CI Secrets +----------------------- -The Factory's CI System, JobServ, has a mechanism to configure secrets for -a Project (the factory's LmP build in this case). These secrets are placed -under ``/secrets/`` when a CI Run is executed. For example a Factory might -have secrets: +The FoundriesFactory® CI System, **JobServ**, has a mechanism for configuring secrets for a project—a Factory's LmP build in this case. +These secrets are placed under ``/secrets/`` when a CI Run is executed. +For example, if a Factory has the following secrets: - * secret_1=A Secret Value - * secret_2=Could even\\nbe multi-line + * ``secret_1=A Secret Value`` + * ``secret_2=Could even\\nbe multi-line`` -Each Run performed during a Build will have the files: +Each build run would have the files: - * /secrets/secret_1 - * /secrets/secret_2 + * ``/secrets/secret_1`` + * ``/secrets/secret_2`` -So the magic required here is getting these accessible to the Dockerfile's -build context. +The question is then how to make these accessible to the Dockerfile's build context. Defining Factory Secrets ------------------------ -The `fioctl`_ tool includes support for managing secrets for a factory:: +`Fioctl®`_ includes support for managing secrets for a factory:: # List secrets defined in the factory: fioctl secrets list @@ -57,21 +53,20 @@ The `fioctl`_ tool includes support for managing secrets for a factory:: # Remove secrets defined in the factory: fioctl secrets update secret_1= secret_2= -.. _fioctl: +.. _Fioctl®: https://github.com/foundriesio/fioctl Passing Secrets to Docker's Build Context ----------------------------------------- -First update factory-config.yml file ci-scripts.git to instruct the -container build scripts to pass the factory secrets to docker with:: +Update ``factory-config.yml`` from ci-scripts.git to instruct the build scripts to pass Factory secrets to Docker: - containers: - docker_build_secrets: true +.. code-block:: YAML -Now that CI system knows the factory wants secrets passed into the build -context, its time to update the Dockerfile for the container with something -like:: + containers: + docker_build_secrets: true + +Update the Dockerfile for the container with something like:: # syntax=docker/dockerfile:1.0.0-experimental # NOTE: - the first line must be this "syntax=" to enable this feature. diff --git a/source/reference-manual/docker/containers.rst b/source/reference-manual/docker/containers.rst deleted file mode 100644 index c90f4bcdb..000000000 --- a/source/reference-manual/docker/containers.rst +++ /dev/null @@ -1,149 +0,0 @@ -.. _ref-containers: - -Containers -========== - -Every factory has a containers.git repository. This repository holds the -source for Docker containers to be built as well as :doc:`compose-apps`. -As changes are made to this repository new Targets will be built. The CI -logic for building a Target is based on simple naming rules: - - * Ignore any top-level directory that ends with ``.disabled``. - * A container image will be built for every top-level directory containing - a file named ``Dockerfile``. - * A compose app will be built for every top-level directory containing - a file named ``docker-compose.yml``. - * Both a container image and compose app will be built if a top-level - directory includes both types of files. - - -Advanced Container Usage ------------------------- - -A container directory may contain a ``docker-build.conf`` file that enables -some advanced functionality. This file "sourced" by the CI shell script and -can set a few special variables to influence what's done at build time: - - - * **CI testing** - Adding ``TEST_CMD=""`` directs the CI - builder to run the command inside the container it just built as a means - of verifying its functioning correctly. - - * **SKIP_ARCHS** - By default containers are builds for arm, arm64, and amd64. - If a container won't build for a certain architecture, it can be skipped. - For example ``SKIP_ARCHS=arm64``. - - * **EXTRA_TAGS_$ARCH** - This can work with ``SKIP_ARCHS``. If builds are - skipped for arm64, the arm container could be tagged for it with: - ``EXTRA_TAGS_arm=arm64``. - - * **DOCKER_BUILD_CONTEXT** - Use an alternative directory for the docker - build context. - -Examples -~~~~~~~~ -:: - - # Only build for amd64 and arm - SKIP_ARCHS="arm64" - -:: - - # Use a 32-bit arm container for a 64-bit host: - SKIP_ARCHS="arm64" - EXTRA_TAGS_arm="arm64" - -:: - - # Use container.git as the build context - BUILD_CONTEXT="../" - -Passing Arguments to Build Context ----------------------------------- - -Containers may require Dockerfile `ARG`_ support for including -build time variables. If the file ``.docker_build_args`` exists in a -container directory, the build script will turn the contents -into ``--build-arg`` options passed to the ``docker build`` command. - -Static information can be done by simply defining a file like:: - - # /.docker_build_args - KEY=Value - KEY2="Value with spaces" - -That would produce a build command that included -``--build-arg KEY=Value --build-arg KEY2="Value with spaces"``. - -The need for dynamic arguments usually means the values must -be generated at build time. The way this can be accomplished is by -taking advantage of ``docker-build.conf``. This file is "sourced" -by the build script, so it can be used to generate content dynamically. - -Example -~~~~~~~ -A common case is including Git commit information into the container:: - - /docker-build.conf - # $TAG is set by the build script as the Git short hash of the - # containers.git commit being built. - # - # $x is the path to the container. - cat <$x/.docker_build_args - GIT_SHA=$TAG - GIT_MSG="$(git log --format=%s -1)" - EOF - -:: - - /Dockerfile - ... - # NOTE - These ARG's change *every* build. In order to maximize - # Docker build caching, they should be as close to the end of the - # file as possible so that the steps after these lines don't have - # to get re-run *every* build. - ARG GIT_SHA - ARG GIT_MSG - ENV GIT_SHA=$GIT_SHA - ENV GIT_MSG=$GIT_MSG - -.. _ARG: - https://docs.docker.com/engine/reference/builder/#arg - -Advanced Container Dependencies -------------------------------- - -In rare occasions a Factory may need some custom code to run *before* the -docker build logic is called on each container. This can be done with a file -in the top-level directory of containers.git, ``pre-build.conf``. - -Examples -~~~~~~~~ - -Here are some examples of things that can be done inside -``pre-build.conf``: - -.. code-block:: bash - - # Create a file with build environment for container "shellhttpd": - env > shellhttpd/envvars - -:: - - # Allow containers in factory to use a common base image - - # First: Make our images build in a predictable order. - # This ensures 0base is built first so other containers can inherit it: - export IMAGES=$(find ./ -mindepth 2 -maxdepth 2 -name Dockerfile | cut -d / -f2 | sort) - - # Second: Modify each container to use the locally build arch-specific base image: - _base_img="hub.foundries.io/${FACTORY}/0base:$LATEST-$ARCH" - for x in $IMAGES ; do - echo "Prebuild checking $x for FROM override" - sed -i "s|hub.foundries.io/${FACTORY}/0base|${_base_img}|" $x/Dockerfile - done - -.. note:: - If there is shared code or files between the containers, the recommendation - is to put the common folder in the base image, then have the other - containers inheriting the needed files using :ref:`ug-multi-stage-container`. diff --git a/source/reference-manual/docker/docker-architecture.rst b/source/reference-manual/docker/docker-architecture.rst index 44a431d8b..3dc823f35 100644 --- a/source/reference-manual/docker/docker-architecture.rst +++ b/source/reference-manual/docker/docker-architecture.rst @@ -5,11 +5,11 @@ Architecture Overview ===================== -The LmP/aktualizr-lite is capable of running Docker Compose projects -that are defined in a device's active Target. The Factory needs a way -to distribute the contents of a compose project, so :ref:`ref-compose-apps` -were created. A good way to visualize how things fit together is -by starting with what a typical Target looks like:: +LmP—via aktualizr-lite—runs Docker Compose projects as defined in a device's active Target. +:ref:`ref-compose-apps` were created as a way for a Factory to distribute the contents of a compose project. +A good way to understand how things fit together is by starting with what a typical Target looks like: + +.. code-block:: YAML "intel-corei7-64-lmp-101" : { "hashes" : { @@ -26,17 +26,13 @@ by starting with what a typical Target looks like:: } ... -This Target includes 2 compose apps, ``fiotest`` and ``shellhttpd`` that -will come from a Factory's containers.git repository. The apps have -a "pinned" URL that includes the sha256 checksum of its content. +This Target includes two compose apps, ``fiotest`` and ``shellhttpd``. +These apps are coming from the Factory's containers.git repository. +The apps have a "pinned" URL which includes the sha256 checksum of its content. -The container build process will pin each container defined in the -compose file to an immutable sha256 Docker reference to make sure -the compose definition can't change over time. The app is then -uploaded to the hub.foundries.io Docker registry as a immutable -tarball. +The container build process pins each to an immutable sha256 Docker reference. +This is done to guarantee that the compose definition can not change over time. +The app is then uploaded to the ``hub.foundries.io`` Docker registry as an immutable tarball. -Aktualizr-lite extracts each compose app's tarball into its own -directory, ``/var/sota/compose-apps/``, during the installation -of a Target. Running the compose app is then a simple matter of -aktualizr-lite launching the compose app. +During the installation of a Target, aktualizr-lite extracts each tarball into its own directory, ``/var/sota/compose-apps/``. +Running the compose app is then a matter of aktualizr-lite launching the compose app. diff --git a/source/reference-manual/docker/docker.rst b/source/reference-manual/docker/docker.rst index 8bcdb6ca5..74e7ce5f8 100644 --- a/source/reference-manual/docker/docker.rst +++ b/source/reference-manual/docker/docker.rst @@ -3,14 +3,16 @@ Docker ====== +This section contains reference information and advanced use cases related to Docker, Compose-apps, and related topics. + +.. seealso:: + The :ref:`containers-and-docker` user guide covers general how-to guides. + .. toctree:: :maxdepth: 1 docker-architecture - containers container-secrets - configure-docker-helper private-registries - compose-apps restorable-apps caching diff --git a/source/reference-manual/docker/private-registries.rst b/source/reference-manual/docker/private-registries.rst index 3910e0811..3c4fe786e 100644 --- a/source/reference-manual/docker/private-registries.rst +++ b/source/reference-manual/docker/private-registries.rst @@ -3,40 +3,36 @@ Using Third-Party Private Container Registries ============================================== -Users sometimes need Foundries.io CI services access container images -from a private third-party container registry such as AWS's ECR_. -There are two reasons this would normally happen: +Sometimes there is a need for a Factory's CI services to a private third-party container registry, such as AWS's ECR_. +There are two common reasons this would be needed: - * A container in ``containers.git`` is based on a private container - image. e.g. Its Dockerfile has ``FROM /`` + * A container in ``containers.git`` is based on a private container image. + For example, the Dockerfile has ``FROM /`` - * A Docker Compose App references a private container image. e.g. - Its ``docker-compose.yml`` has a line like ``image: /`` + * A Docker Compose App references a private container image. + For example, a ``docker-compose.yml`` with the line ``image: /`` -In either case, CI will need read access to the private container image -in order to construct a Target. The :ref:`ref-factory-definition` -(``ci-scripts.git``) provides a way to configure this. +The CI needs read access to the private container image in order to construct a Target. +The :ref:`ref-factory-definition` (``ci-scripts.git``) provides a way to configure this. Configuring for CI Azure Container Registry (ACR) ------------------------------------------------- -CI can be configured to use an ACR `service principal`_ with read-only -access to a private ACR instance. First, CI must be configured with -the service principal's ID and password:: +The CI can be configured to use an ACR `service principal`_ with read-only access to a private ACR instance. +First, the CI must be configured with the service principal's ID and password:: $ fioctl secrets update azprincipal=':' -The factory :ref:`configuration ` is then -updated accordingly:: +The Factory :ref:`configuration ` is then updated accordingly: - # factory-config.yml - container_registries: - - type: azure - url: .azurecr.io - azure_principal_secret_name: azprincipal +.. code-block:: YAML + + # factory-config.yml + container_registries: + - type: azure + url: .azurecr.io + azure_principal_secret_name: azprincipal .. _service principal: https://learn.microsoft.com/en-us/azure/container-registry/container-registry-auth-service-principal#authenticate-with-the-service-principal @@ -46,15 +42,14 @@ updated accordingly:: Configuring Devices for ACR ^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Azure does not include a built-in way for devices to `securely access`_ -a container registry like AWS IoT does. The recommended approach for -this scenario is pushing container images from ACR into -`hub.foundries.io` or having a container image built with something similar to:: +Azure does not include a built-in way for devices to `securely access`_ a container registry. +The recommended approach is pushing container images from ACR to ``hub.foundries.io``. +Another approach is having a container image built similar to:: # containers.git /Dockerfile FROM .azurecr.io -Then compose apps can reference this `hub.foundries.io` container image. +Then compose apps can reference the ``hub.foundries.io`` container image. .. _securely access: https://learn.microsoft.com/en-us/answers/questions/734990/iot-device-authentication-with-acr @@ -62,27 +57,26 @@ Then compose apps can reference this `hub.foundries.io` container image. Configuring CI for AWS ECR -------------------------- -CI uses the `aws ecr get-login-password`_ command to authenticate. A -factory can be configured to use this by first providing an AWS -credentials file to CI:: +CI uses the `aws ecr get-login-password`_ command to authenticate. +A Factory can be configured to use this by first providing an AWS credentials file to CI:: $ fioctl secrets update aws_creds="$(cat $HOME/.aws/credentials)" .. note:: - Using a personal credentials file is probably not secure. An AWS - admin should create an IAM account that can only **pull** from - the container registry. + Using a personal credentials file is probably not secure. + An AWS admin should create an IAM account that can only **pull** from the container registry. -Next the factory :ref:`configuration ` -needs to be set. Example configuration:: +Then the :ref:`configuration ` needs to be set. - # factory-config.yml - container_registries: - - type: aws - region: us-east-2 - url: XXXXXXXXXX.dkr.ecr.us-east-2.amazonaws.com - aws_creds_secret_name: aws_creds +.. code-block:: YAML + + # factory-config.yml + container_registries: + - type: aws + region: us-east-2 + url: XXXXXXXXXX.dkr.ecr.us-east-2.amazonaws.com + aws_creds_secret_name: aws_creds .. _ECR: https://aws.amazon.com/ecr/ @@ -93,31 +87,26 @@ needs to be set. Example configuration:: Configuring Devices for AWS ECR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Devices may need access to private ECR container images. While -bootstrapping a project, it is often easiest to simply copy AWS -credentials to each device. However, before going to production, it is -highly recommend to use an approach that can be backed by x509 -certificates such as the ones used for the :ref:`device gateway `. -`AWS IoT`_ includes a mechanism for eliminating hard-coded -credentials_. +Devices may need access to private ECR container images. +While bootstrapping a project, it may be easiest to copy AWS credentials to each device. +However, before going to production, it is highly recommend to use an approach that can be backed by x509 certificates. +This includes the ones used for the :ref:`device gateway `. +`AWS IoT`_ includes a mechanism for eliminating hard-coded credentials_. .. _AWS IoT: https://aws.amazon.com/iot/ .. _credentials: https://aws.amazon.com/blogs/security/how-to-eliminate-the-need-for-hardcoded-aws-credentials-in-devices-by-using-the-aws-iot-credentials-provider/ -Another approach is to follow the procedure as outlined for devices accessing -the :ref:`Azure Container Registry `. This wraps the container -images into `hub.foundries.io` . With this setup, devices rely on the Factory -authentication instead of authenticating to AWS ECR. +Another approach is to follow the procedure outlined for devices accessing the :ref:`Azure Container Registry `. +This wraps the container images into ``hub.foundries.io`` . +With this setup, devices rely on the Factory authentication instead of authenticating to AWS ECR. Configuring for CI Google Artifact Registry (GAR) ------------------------------------------------- -CI can be configured to use a Google Compute Platform(GCP) -`service account`_ with read-only access to a private GAR instance. A -service account can be created that may only do Docker pull operations -with:: +The CI can be configured to use a Google Compute Platform(GCP) `service account`_ with read-only access to a private GAR instance. +A service account can be created that may only do Docker pull operations:: # Create the service account $ NAME= @@ -137,13 +126,11 @@ with:: application_default_credentials.json \ --iam-account=${NAME}@${PROJ_ID}.iam.gserviceaccount.com -The service account key file created above then needs to be configured -for CI with:: +The service account key file created above then needs to be configured for CI:: $ fioctl secrets update gcp_creds==application_default_credentials.json -The factory :ref:`configuration ` is then -updated accordingly:: +The Factory :ref:`configuration ` is then updated accordingly:: # factory-config.yml container_registries: @@ -156,6 +143,5 @@ updated accordingly:: Configuring Devices for GAR ^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Google does not have a way to authenticate it's IoT core devices with -the Artifact Registry. We recommend following the same approach as -outlined for devices accessing the :ref:`Azure Container Registry `. +Google does not have a way to authenticate IoT core devices with the Artifact Registry. +We recommend following the same approach as outlined for devices accessing the :ref:`Azure Container Registry `. diff --git a/source/reference-manual/docker/restorable-apps.rst b/source/reference-manual/docker/restorable-apps.rst index c2fe0cd99..5e5dfac12 100644 --- a/source/reference-manual/docker/restorable-apps.rst +++ b/source/reference-manual/docker/restorable-apps.rst @@ -3,36 +3,47 @@ Restorable Apps =============== -Restorable Apps are Compose Apps that have capability to be restored in a case of any docker store damages without a need to be re-downloaded from Registry. -It's achieved by the distinct way images are pulled and stored on a device. +Restorable Apps are Compose Apps that can be restored in case of any Docker store damages, without needing to be re-downloaded from the Registry. +This is achieved by a distinct way images can pulled and stored on a device. -#. Images are pulled from Registries by `skopeo `_ utility, unlike pulling them by utilizing of the docker's/docker daemon's regular functionality in the case of Compose Apps; -#. Images are stored in two places on a device. In addition to the regular docker daemon's image store (``/var/lib/docker/image/overlay2`` and ``/var/lib/overlay2``), - images are also stored in the folder defined in the ``aktualizr-lite`` config. By default, it's set to ``/var/sota/reset-apps``. - The additional store path can be overridden in the ``*.toml`` configuration file by defining a value of ``[pacman].reset_apps_root`` configuration variable. +#. Images are pulled from registries by the `skopeo `_ utility, + rather than pulling them by utilizing the Docker daemon's regular functionality normally used for Compose Apps; +#. Images are stored in two places on a device. + In addition to the regular Docker daemon's image store (``/var/lib/docker/image/overlay2`` and ``/var/lib/overlay2``), + images are also stored in the folder defined in the ``aktualizr-lite`` config file. + This defaults to ``/var/sota/reset-apps``. + The additional store path can be overridden in the ``*.toml`` config file by defining a value of ``[pacman].reset_apps_root``. -Image layers are stored in the additional store in ``tar.gzip`` format. This helps to reduce the device storage cost of this feature. -Effectively, the difference in the docker store size and the Restorable Apps' additional storage size is equivalent to the compression ratio of ``gzip``. +Image layers are stored in the additional store in ``tar.gzip`` format. +This helps to reduce the device storage cost of this feature. +Effectively, the difference in the Docker store size and the Restorable Apps' additional storage size is equivalent to the compression ratio of ``gzip``. + +Employment of Restorable Apps +----------------------------- Primarily, Restorable Apps are employed to: -------------------------------------------- -#. support "Factory Reset" feature; -#. provide means to restore a docker daemon's store if it gets broken. +#. Support "Factory Reset" feature; +#. Provide means to restore a Docker daemon's store if it gets broken. -Both cases implies that a desired state of Apps (effectively a docker daemon store's state) can be restored even if -a docker daemon's store (``/var/lib/docker``) is deleted or partially broken without a need to re-download images from remote Registries. +Both cases imply that a desired state of Apps can be restored—even if a Docker daemon's store (``/var/lib/docker``) is deleted or broken—without needing to re-download images. Effectively, images are injected from the additional storage to the store in such cases. -Restorable Apps are enabled by default, and a list of Restorable Apps is equal to a list of Apps enabled on a device (a value of ``compose_apps`` parameter if defined or all Target Apps). -A user may extend the default list of Restorable Apps in the following ways: +Restorable Apps are enabled by default. +The list of Restorable Apps is equal to the list of Apps enabled on a device (if defined, the value of ``compose_apps``, otherwise all Target Apps). +You may extend the default list of Restorable Apps in the following ways: + +#. For **manual** registration, by invoking lmp-device-register_ with ``‑‑restorable‑apps ``. +#. For **auto** registration, using ``--restorable-apps `` with lmp-device-auto-register_. +#. By setting ``reset_apps=`` in ``[pacman].reset_apps`` for a device/aklite config. + You may change the given configuration for a specific device, a device group, or all devices (see :ref:`ref-configuring-devices` for details). + +To disable Restorable Apps, specify an empty string ``""`` as the value of ```` in one of the previous options. +For example, ``lmp-device-register --restorable-apps ""``. -#. By invoking of lmp-device-auto-register_ utility with the ``--restorable-apps `` option during manual device registration. -#. If a device auto registration is configured for a factory, then by adding the ``--restorable-apps `` option to the lmp-device-auto-register_ script. - See :ref:`ug-lmp-device-auto-register` for more details. -#. By setting ``reset_apps=`` value in ``[pacman].reset_apps`` of a device/aklite config. - A user may change the given configuration for a specific device, a device group or all factory devices (see :ref:`ref-configuring-devices` for details). +.. seealso:: + :ref:`ug-lmp-device-auto-register` -To disable Restorable Apps, a user should specify an empty string ``""`` as a value of ```` in one of the previous options (e.g. ``lmp-device-register --restorable-apps ""``). +.. _lmp-device-register: https://github.com/foundriesio/lmp-device-register .. _lmp-device-auto-register: https://github.com/foundriesio/meta-lmp/tree/main/meta-lmp-base/recipes-support/lmp-device-auto-register diff --git a/source/tutorials/compose-app/compose-app.rst b/source/tutorials/compose-app/compose-app.rst index b140fa17e..4dae7a11e 100644 --- a/source/tutorials/compose-app/compose-app.rst +++ b/source/tutorials/compose-app/compose-app.rst @@ -24,7 +24,7 @@ Prerequisites - Completed the :ref:`tutorial-gs-with-docker` tutorial. - Completed the :ref:`tutorial-creating-first-target` tutorial. - Completed the :ref:`tutorial-deploying-first-app` tutorial. -- Read the :ref:`ref-compose-apps` reference manual. +- Read the :ref:`ref-compose-apps` guide. Instructions ------------ diff --git a/source/reference-manual/docker/compose-apps.rst b/source/user-guide/containers-and-docker/compose-apps.rst similarity index 54% rename from source/reference-manual/docker/compose-apps.rst rename to source/user-guide/containers-and-docker/compose-apps.rst index 869ac0f9d..02a5643d8 100644 --- a/source/reference-manual/docker/compose-apps.rst +++ b/source/user-guide/containers-and-docker/compose-apps.rst @@ -3,17 +3,16 @@ Compose Apps ============ -Docker-compose Apps or "Compose Apps" are the recommended way for doing -application development in a FoundriesFactory. Docker-compose has emerged -as a great way to develop applications. However, it doesn't specify how to -**distribute** applications. Compose apps fill in this gap for Factory devices. +Docker-compose applications, or "Compose Apps", are how application development is approached with FoundriesFactory®. +However, Docker-compose does not specify how to **distribute** applications. +Compose Apps fill in this gap for Factory devices. Compose Apps in a Factory ------------------------- -Compose apps are automatically managed from containers.git by naming -convention. Any top-level directory containing a docker-compose.yml file -is distributed as compose apps. Here is a simplistic source layout:: +Compose Apps are automatically managed from ``containers.git`` by naming convention. +Any top-level directory containing ``docker-compose.yml`` gets distributed as a compose app. +Here is an example source layout:: # containers.git httpd/ @@ -40,8 +39,7 @@ is distributed as compose apps. Here is a simplistic source layout:: } } -When changes are made to containers.git, the factory will produce a new -Target that includes the updated ``httpd`` compose app:: +When changes are made in ``containers.git``, the Factory will produce a new Target that includes the updated ``httpd`` compose app:: $ fioctl targets show 77 @@ -54,31 +52,25 @@ Target that includes the updated ``httpd`` compose app:: Compose Apps Distribution ------------------------- -Compose apps are distributed by taking advantage of Docker Registry -primitives. The httpd example above becomes a tarball stored -in hub.foundries.io:: +Compose apps are distributed by taking advantage of Docker Registry primitives. +The httpd example above becomes a tarball stored in ``hub.foundries.io``:: # httpd.tgz ./docker-compose.yml ./ngix.conf -The tarball is an exact copy of httpd's directory layout in containers.git -with one important exception: The publishing logic looks at each service -image in the compose file and will pin it to the correct sha256 checksum -at the time of publishing. For this example, ``nginx:alpine`` would -become something like ``nginx:sha256@deadbeef``. This ensures each version of -the compose app is immutable. +The tarball is an exact copy of httpd's directory layout in ``containers.git`` with one important exception: The publishing logic looks at each service image in the compose file and will pin it to the correct sha256 checksum at the time of publishing. +For this example, ``nginx:alpine`` would become something like ``nginx:sha256@deadbeef``. +This ensures each version of the compose app is immutable. -Aktualizr-lite then has logic to grab this content from the docker registry, -validate it cryptographically, and extract it locally so that docker-compose -can consume it. +Aktualizr-lite then has logic to grab this content from the Docker registry, validate it cryptographically, and extract it locally so that docker-compose can consume it. How Does It Fit Together? ------------------------- -Changes to containers produce new TUF Targets that aktualizr-lite can -install. The interesting part of the Target in this case will be:: +Changes to containers produce new TUF Targets that aktualizr-lite can install. +The interesting part of the Target in this case is:: { ... @@ -95,11 +87,11 @@ install. The interesting part of the Target in this case will be:: Examples -------- -Single container application +Single Container Application ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Many users can build their entire application as a single container. In -this scenario containers.git layout might look like:: +Many Factories can build their entire application as a single container. +In this scenario ``containers.git`` layout might look like:: # simple-app/Dockerfile FROM alpine @@ -131,26 +123,23 @@ this scenario containers.git layout might look like:: Dockerfile app.py -Non-comment lines of ``.composeappignores`` will match files -according to Golang's `filepath.Match`_ +Non-comment lines of ``.composeappignores`` will match files according to Golang's `filepath.Match`_ .. _filepath.Match: https://golang.org/pkg/path/filepath/#Match -Each change to containers.git will produce a new compose app with contents:: +Each change to ``containers.git`` will produce a new compose app with contents:: # simple-app.tgz ./docker-compose.yml -In this case ``hub.foundries.io//simple-app:latest`` is pinned to -the exact container built during the change to containers.git. The CI logic -does this automatically for the user. +In this case ``hub.foundries.io//simple-app:latest`` is pinned to the exact container built during the change to ``containers.git``. +The CI logic does this automatically. A Flask Web App ~~~~~~~~~~~~~~~ -This example uses multiple containers to build a typical python3 Flask -application:: +This example uses multiple containers to build a typical Python3 Flask application:: # hello-world/Dockerfile FROM alpine @@ -195,17 +184,15 @@ application:: } } -Changes to containers.git does a couple of interesting things here: +Changes to ``containers.git`` does a couple of interesting things: -#. It will build and publish a version of the hello-world container. For - this example, call it ``hub.foundries.io//hello-world:GIT_SHORT_HASH`` +#. It will build and publish a version of the hello-world container. + For this example, ``hub.foundries.io//hello-world:GIT_SHORT_HASH`` -#. A compose app will be published. The compose app will include the - nginx.conf file and a "pinned" docker-compose.yml. In this case the - containers will be pinned to: +#. A compose app will be published. + The compose app will include ``nginx.conf`` and a "pinned" ``docker-compose.yml``. + In this case the containers will be pinned to: - a. ``nginx:alpine`` - the sha256 checksum of nginx:alpine at the time - this was built. + a. ``nginx:alpine``: the sha256 checksum of ``nginx:alpine`` at the time this was built. - b. ``hub.foundries.io//hello-world`` - the sha256 checksum - of ``GIT_SHORT_HASH`` at the time this was built. + b. ``hub.foundries.io//hello-world``: the sha256 checksum of ``GIT_SHORT_HASH`` at the time this was built. diff --git a/source/reference-manual/docker/configure-docker-helper.rst b/source/user-guide/containers-and-docker/configure-docker-helper.rst similarity index 89% rename from source/reference-manual/docker/configure-docker-helper.rst rename to source/user-guide/containers-and-docker/configure-docker-helper.rst index ba4921cd7..d2ebcbfc0 100644 --- a/source/reference-manual/docker/configure-docker-helper.rst +++ b/source/user-guide/containers-and-docker/configure-docker-helper.rst @@ -3,7 +3,7 @@ Docker Credential Helper ======================== -``fioctl`` has a Docker credential helper, providing easier access to hub.foundries.io. +Fioctl® has a Docker credential helper, providing easier access to ``hub.foundries.io``. This enables the use of Docker commands from a personal computer, such as a laptop. .. note:: @@ -20,7 +20,7 @@ This creates a symlink named ``docker-credential-fio`` in the directory of the d .. important:: Because the Docker client is usually somewhere under ``/usr``, you will want to run ``fioctl configure-docker`` with root permission. -The helper then updates `your` Docker config file, which is located under ``$HOME/.docker``. +The helper then updates *your* Docker config file, which is located under ``$HOME/.docker``. .. note:: The helper configures for the current `user` and not the entire `system`. diff --git a/source/user-guide/container-preloading/container-preloading.rst b/source/user-guide/containers-and-docker/container-preloading.rst similarity index 100% rename from source/user-guide/container-preloading/container-preloading.rst rename to source/user-guide/containers-and-docker/container-preloading.rst diff --git a/source/user-guide/containers-and-docker/containers.rst b/source/user-guide/containers-and-docker/containers.rst new file mode 100644 index 000000000..f5d9b7cf1 --- /dev/null +++ b/source/user-guide/containers-and-docker/containers.rst @@ -0,0 +1,134 @@ +.. _ref-containers: + +Containers +========== + +A Factory's ``containers.git`` repo holds the source for Docker containers, as well as :doc:`compose-apps`. +When changes are made to this repo, new Targets will be built. +The CI logic for building a Target is based on simple naming rules: + + * Ignores top-level directories that ends with ``.disabled``. + * A container image will be built for every top-level directory containing a file named ``Dockerfile``. + * A :ref:`compose app ` will be built for every top-level directory containing a file named ``docker-compose.yml``. + +Both a container image and compose app will be built if both a Dockerfile and Docker-compose file are included. + + +Advanced Container Usage +------------------------ + +A ``docker-build.conf`` file enables advanced functionality. +This is sourced by the CI shell script and can set build time options: + + + * **TEST_CMD**: Adding ``TEST_CMD=""`` directs the CI builder to run the command inside the container to verify that it is functioning correctly. + + * **SKIP_ARCHS**: A container will not be built for the specified architecture, e.g., ``SKIP_ARCHS=arm64``. + + * **EXTRA_TAGS_$ARCH**: This can work with ``SKIP_ARCHS``. + If builds are skipped for arm64, an arm32 container could be *tagged* for it with ``EXTRA_TAGS_arm=arm64``. + + * **DOCKER_BUILD_CONTEXT**: Set an alternative directory for the Docker build context. + +Examples +~~~~~~~~ +:: + + # Only build for amd64 and arm + SKIP_ARCHS="arm64" + +:: + + # Use a 32-bit arm container for a 64-bit host: + SKIP_ARCHS="arm64" + EXTRA_TAGS_arm="arm64" + +:: + + # Use container.git as the build context + BUILD_CONTEXT="../" + +Passing Arguments to Build Context +---------------------------------- + +Containers may require Dockerfile `ARG`_ support for including build time variables. +If the file ``.docker_build_args`` exists in a container director, the contents are used as ``--build-arg`` options. +These are passed to the ``docker build`` command. + +Static information can be done by defining a file like:: + + # /.docker_build_args + KEY=Value + KEY2="Value with spaces" + +This produces a build command that includes ``--build-arg KEY=Value --build-arg KEY2="Value with spaces"``. + +The need for dynamic arguments means some values must be generated at build time. +This can be accomplished is by taking advantage of ``docker-build.conf``; +as it is sourced by the build script, it can be used to generate content dynamically. + +Example +~~~~~~~ + +A common case is including Git commit information in a container:: + + /docker-build.conf + # $TAG is set by the build script as the Git short hash of the + # containers.git commit being built. + # + # $x is the path to the container. + cat <$x/.docker_build_args + GIT_SHA=$TAG + GIT_MSG="$(git log --format=%s -1)" + EOF + +:: + + /Dockerfile + ... + # NOTE - These ARG's change *every* build. In order to maximize + # Docker build caching, they should be as close to the end of the + # file as possible so that the steps after these lines don't have + # to get re-run *every* build. + ARG GIT_SHA + ARG GIT_MSG + ENV GIT_SHA=$GIT_SHA + ENV GIT_MSG=$GIT_MSG + +.. _ARG: + https://docs.docker.com/engine/reference/builder/#arg + +Advanced Container Dependencies +------------------------------- + +If you need custom code to be executed *before* the Docker build, +add ``pre-build.conf`` in the top-level directory of ``containers.git``. + +Examples +~~~~~~~~ + +Here are examples of what can be done in ``pre-build.conf``: + +.. code-block:: bash + + # Create a file with build environment for container "shellhttpd": + env > shellhttpd/envvars + +:: + + # Allow containers in factory to use a common base image + + # First: Make our images build in a predictable order. + # This ensures 0base is built first so other containers can inherit it: + export IMAGES=$(find ./ -mindepth 2 -maxdepth 2 -name Dockerfile | cut -d / -f2 | sort) + + # Second: Modify each container to use the locally build arch-specific base image: + _base_img="hub.foundries.io/${FACTORY}/0base:$LATEST-$ARCH" + for x in $IMAGES ; do + echo "Prebuild checking $x for FROM override" + sed -i "s|hub.foundries.io/${FACTORY}/0base|${_base_img}|" $x/Dockerfile + done + +.. note:: + If there is shared files between containers, the recommendation is to put the common folder in the base image. + The containers can inherit the files using :ref:`ug-multi-stage-container`. diff --git a/source/user-guide/containers-and-docker/index.rst b/source/user-guide/containers-and-docker/index.rst new file mode 100644 index 000000000..2f5f9d1e7 --- /dev/null +++ b/source/user-guide/containers-and-docker/index.rst @@ -0,0 +1,19 @@ +.. _containers-and-docker: + +Containers and Docker +===================== + +This section contains guides related to managing Factory containers and Compose Apps, +with a focus on general usage. + +.. seealso:: + The :ref:`ref-docker` reference manual covers more in-depth information such as architecture, third party registries, and Factory secrets. + +.. toctree:: + :maxdepth: 1 + + containers + configure-docker-helper + compose-apps + container-preloading + multi-stage-container diff --git a/source/user-guide/multi-stage-container/multi-stage-container.rst b/source/user-guide/containers-and-docker/multi-stage-container.rst similarity index 99% rename from source/user-guide/multi-stage-container/multi-stage-container.rst rename to source/user-guide/containers-and-docker/multi-stage-container.rst index 2bf66d0ff..6b43f4237 100644 --- a/source/user-guide/multi-stage-container/multi-stage-container.rst +++ b/source/user-guide/containers-and-docker/multi-stage-container.rst @@ -16,7 +16,7 @@ In the first stage you can install dependencies for development, like a toolchai In the final stage, the image is kept clean with just the necessities for your application. This is done by making use of the final artifacts from the first stage. -This guide presenst a Dockerfile implementing a ``helloworld`` application written in C. +This guide presents a Dockerfile implementing a ``helloworld`` application written in C. In the first example, the Dockerfile is implementing everything in a single stage, leaving all objects and spare software in the image.