Skip to content

Commit

Permalink
Cleanup of docker reference manual
Browse files Browse the repository at this point in the history
Style guidelines applied.

Semantic newlines applied to follow best practices and to assist with
editing.

Moved pages which serve more to guide than inform to a new section of
the user guide, 'Containers and Docker'. Added the appropriate
redirects, as well as updating landing/index pages as needed.

Ran linter, linkcheck, and viewed rendered html.

This commit addresses FFTK-2789
This commit applies to FFTK-988

Signed-off-by: Katrina Prosise <[email protected]>
  • Loading branch information
kprosise committed Nov 9, 2023
1 parent bbf04f4 commit 3ec002f
Show file tree
Hide file tree
Showing 16 changed files with 336 additions and 356 deletions.
5 changes: 5 additions & 0 deletions source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -336,6 +336,11 @@
# Config for sphinx-reredirects, maps source: target, target path is relative to source.
# TODO: troubleshooting sections redirecting to lmp-customization do not appear to be functional
redirects = {
"reference-manual/docker/compose-apps": "../../user-guide/containers-and-docker/compose-apps.html",
"reference-manual/docker/configure-docker-helper": "../../user-guide/containers-and-docker/configure-docker-helper.html",
"reference-manual/docker/containers": "../../user-guide/containers-and-docker/containers.html",
"user-guide/containers-preloading/container-preloading": "../containers-and-docker/container-preloading.html",
"user-guide/multi-stage-container/multi-stage-container": "../containers-and-docker/multi-stage-container",
"reference-manual/factory/team-based-access": "../../user-guide/account-management/team-based-access.html",
"reference-manual/factory/factory-keys": "../../reference-manual/security/factory-keys.html",
"reference-manual/security/secure-boot": "security.html#secure-boot-hardware-root-of-trust",
Expand Down
3 changes: 1 addition & 2 deletions source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ OE/Yocto Project, the Linux microPlatform™ and Docker®.
:name: sec-user-guide

user-guide/account-management/account-management
user-guide/container-preloading/container-preloading
user-guide/containers-and-docker/index
user-guide/custom-ci/custom-ci
user-guide/lmp-customization/lmp-customization
user-guide/lmp-auto-hostname/lmp-auto-hostname
Expand All @@ -47,7 +47,6 @@ OE/Yocto Project, the Linux microPlatform™ and Docker®.
user-guide/mirror-action/mirror-action
user-guide/submodule/submodule
user-guide/custom-sota-client
user-guide/multi-stage-container/multi-stage-container
user-guide/fioctl/index
user-guide/cert-rotation
user-guide/device-gateway-pki/device-gateway-pki
Expand Down
39 changes: 17 additions & 22 deletions source/reference-manual/docker/caching.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,31 +3,26 @@
Caching
=======

Each container built in a FoundriesFactory will publish the build cache layers
to our private registry, hub.foundries.io. Subsequent builds will pull from this
cache, importing it for the current build. This provides efficient incremental
container builds for our FoundriesFactory users. Exporting and importing
these build cache layers uses the built-in features of
`Docker Buildx. <https://docs.docker.com/build/>`_
Each container built in a Factory publishes the build cache layers to our private registry, ``hub.foundries.io``.
Subsequent builds pull/import from this cache.
This provides efficient, incremental container builds.
This exporting and importing of build cache layers uses built-in features of `Docker Buildx <https://docs.docker.com/build/>`_.

While the cache is very helpful, there are a few things to note:
While the cache is useful, there are a few things to note:

* The build cache for each container image is branch specific
* It is architecture specific
* It can be invalidated for a few reasons, such as source files changing
or a base image update.
* The build cache for each container image is *branch specific*
* It is *architecture specific*
* It can be invalidated for a few reasons, such as source files changing or a base image update.

Cache Invalidation
------------------

When trying to understand why the cache has been invalidated, there are not any
tools that can assist. That said, in a FoundriesFactory each container build
will automatically use the docker build context to create a list of md5sums of all
source files used in the build. Each :doc:`compose-apps` will generate an artifact
named ``<compose-app-name>-md5sum.txt``, which then can be used to
generate a ``diff`` from build to build to assist in understanding what files may
have changed, and how that effects the caching. Generally, each line in your ``Dockerfile``
creates it's own image layer, and a corresponding cache layer. If source files change,
or any direct modification to the ``Dockerfile`` occurs, it will invalidate the cache from
that point forward. With the importance of these concepts towards building :doc:`compose-apps`,
it is recommended to read the documentation related to `Dockerfile best practices. <https://docs.docker.com/develop/develop-images/dockerfile_best-practices/>`_
When trying to understand why a cache has been invalidated, there is not any tools that can assist.
That said, in a Factory, each container build automatically uses the Docker build context to create a list of md5sums of all source files used.

Each :ref:`Compose App <ref-compose-apps>` generates an artifact named ``<compose-app-name>-md5sum.txt``.
This can then be used to generate a ``diff`` between builds, assisting in understanding what files may have changed, and how the cache may have been effected.

Generally, each line of your ``Dockerfile`` creates an image layer with a corresponding cache layer.
If source files change, or any direct modification to the ``Dockerfile`` occurs, it will invalidate the cache from that point forward.
Due to the importance of these concepts towards building :ref:`ref-compose-apps`, it is recommended to read the `Dockerfile best practices. <https://docs.docker.com/develop/develop-images/dockerfile_best-practices/>`_.
57 changes: 26 additions & 31 deletions source/reference-manual/docker/container-secrets.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,49 +3,45 @@
Using Secret Credentials When Building Containers
=================================================

There are many cases when building a container that sensitive credentials
may be required. Examples include:
There are many cases when building a container that sensitive credentials may be required.
Examples include:

* Downloading a package from a private NPM registry
* Grabbing a file via scp
* Grabbing a file via ``scp``.

Each scenario requires a slightly different approach in the Dockerfile.
However, the general approach to getting these secrets securely into your
container's build context is the same.
Each scenario requires a slightly different approach in the Dockerfile.
However, the general approach to getting these secrets securely into your container's build context is the same.

.. note::

There are several `insecure ways`_ that should be avoid in order to do this
correctly. The proper approach is what's described here based on new
functionality in Docker's BuildKit.
There are several `insecure ways`_ that should be avoided.
The proper approach is described here based on new functionality in Docker's BuildKit.

.. _insecure ways:
https://pythonspeed.com/articles/docker-build-secrets/


Quick Background On CI Secrets
------------------------------
Overview of CI Secrets
-----------------------

The Factory's CI System, JobServ, has a mechanism to configure secrets for
a Project (the factory's LmP build in this case). These secrets are placed
under ``/secrets/`` when a CI Run is executed. For example a Factory might
have secrets:
The FoundriesFactory® CI System, **JobServ**, has a mechanism for configuring secrets for a project—a Factory's LmP build in this case.
These secrets are placed under ``/secrets/`` when a CI Run is executed.
For example, if a Factory has the following secrets:

* secret_1=A Secret Value
* secret_2=Could even\\nbe multi-line
* ``secret_1=A Secret Value``
* ``secret_2=Could even\\nbe multi-line``

Each Run performed during a Build will have the files:
Each build run would have the files:

* /secrets/secret_1
* /secrets/secret_2
* ``/secrets/secret_1``
* ``/secrets/secret_2``

So the magic required here is getting these accessible to the Dockerfile's
build context.
The question is then how to make these accessible to the Dockerfile's build context.

Defining Factory Secrets
------------------------

The `fioctl`_ tool includes support for managing secrets for a factory::
`Fioctl®`_ includes support for managing secrets for a factory::

# List secrets defined in the factory:
fioctl secrets list
Expand All @@ -57,21 +53,20 @@ The `fioctl`_ tool includes support for managing secrets for a factory::
# Remove secrets defined in the factory:
fioctl secrets update secret_1= secret_2=

.. _fioctl:
.. _Fioctl®:
https://github.com/foundriesio/fioctl

Passing Secrets to Docker's Build Context
-----------------------------------------

First update factory-config.yml file ci-scripts.git to instruct the
container build scripts to pass the factory secrets to docker with::
Update ``factory-config.yml`` from ci-scripts.git to instruct the build scripts to pass Factory secrets to Docker:

containers:
docker_build_secrets: true
.. code-block:: YAML
Now that CI system knows the factory wants secrets passed into the build
context, its time to update the Dockerfile for the container with something
like::
containers:
docker_build_secrets: true
Update the Dockerfile for the container with something like::

# syntax=docker/dockerfile:1.0.0-experimental
# NOTE: - the first line must be this "syntax=" to enable this feature.
Expand Down
149 changes: 0 additions & 149 deletions source/reference-manual/docker/containers.rst

This file was deleted.

30 changes: 13 additions & 17 deletions source/reference-manual/docker/docker-architecture.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@
Architecture Overview
=====================

The LmP/aktualizr-lite is capable of running Docker Compose projects
that are defined in a device's active Target. The Factory needs a way
to distribute the contents of a compose project, so :ref:`ref-compose-apps`
were created. A good way to visualize how things fit together is
by starting with what a typical Target looks like::
LmP—via aktualizr-lite—runs Docker Compose projects as defined in a device's active Target.
:ref:`ref-compose-apps` were created as a way for a Factory to distribute the contents of a compose project.
A good way to understand how things fit together is by starting with what a typical Target looks like:

.. code-block:: YAML
"intel-corei7-64-lmp-101" : {
"hashes" : {
Expand All @@ -26,17 +26,13 @@ by starting with what a typical Target looks like::
}
...
This Target includes 2 compose apps, ``fiotest`` and ``shellhttpd`` that
will come from a Factory's containers.git repository. The apps have
a "pinned" URL that includes the sha256 checksum of its content.
This Target includes two compose apps, ``fiotest`` and ``shellhttpd``.
These apps are coming from the Factory's containers.git repository.
The apps have a "pinned" URL which includes the sha256 checksum of its content.

The container build process will pin each container defined in the
compose file to an immutable sha256 Docker reference to make sure
the compose definition can't change over time. The app is then
uploaded to the hub.foundries.io Docker registry as a immutable
tarball.
The container build process pins each to an immutable sha256 Docker reference.
This is done to guarantee that the compose definition can not change over time.
The app is then uploaded to the ``hub.foundries.io`` Docker registry as an immutable tarball.

Aktualizr-lite extracts each compose app's tarball into its own
directory, ``/var/sota/compose-apps/<app>``, during the installation
of a Target. Running the compose app is then a simple matter of
aktualizr-lite launching the compose app.
During the installation of a Target, aktualizr-lite extracts each tarball into its own directory, ``/var/sota/compose-apps/<app>``.
Running the compose app is then a matter of aktualizr-lite launching the compose app.
8 changes: 5 additions & 3 deletions source/reference-manual/docker/docker.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,16 @@
Docker
======

This section contains reference information and advanced use cases related to Docker, Compose-apps, and related topics.

.. seealso::
The :ref:`containers-and-docker` user guide covers general how-to guides.

.. toctree::
:maxdepth: 1

docker-architecture
containers
container-secrets
configure-docker-helper
private-registries
compose-apps
restorable-apps
caching
Loading

0 comments on commit 3ec002f

Please sign in to comment.