Using Docker Inside the Container

Run 64-bit Docker images within your systemd-nspawn container!

Introduction

Docker is a popular set of products that use OS-level virtualization to deliver software in packages called containers. By default however, the security features of systemd-nspawn's own (!) containers prevent Docker being usable inside them.

Accordingly, in this brief tutorial I'll run through the changes required to make this possible.

Prerequisites

You should be running an up-to-date instance of raspbian-nspawn-64, and have checked that you can e.g. start a 64-bit shell successfully.

You don't need to have installed Docker in the 64-bit container yet (as it will fail to initialize correctly if you try). We'll cover installation later in this tutorial.

Required Modifications to the Host System

Container Setup

Working at a (32-bit) terminal in the host (Raspbian) system, as the pi (or other regular user) user, let's begin to make the necessary modifications, by editing debian-buster-64 container's control file so as to:

• retain all capabilities;
• prevent the add_key and keyctl syscalls from being blocked by seccomp_bpf; and
• make the host's control group (cgroup) namespace visible within the guest.

To do so, issue:

pi@raspberrypi:~ $ sudo nano -w /etc/systemd/nspawn/debian-buster-64.nspawn

Then modify the [Exec] section, removing any current Capability=... lines (if present), and replacing with:

# required for Docker
Capability=all
SystemCallFilter=add_key keyctl

Leave the rest of the section as-is, including any PrivateUsers=no stanzas etc.

Then, move on to the [Files] section, and append the following:

# required for Docker
Bind=/sys/fs/cgroup

Leave the rest of the section (and file) as is. Save, and exit nano.

These modifications will not take effect until the container has been restarted.

Next, create an additional drop-in snippet for the nspawn debian-buster-64 container service, thus:

pi@raspberrypi:~ $ sudo systemctl edit systemd-nspawn@debian-buster-64

and in the new file that opens, enter:

[Service]
# required for Docker
Environment=SYSTEMD_NSPAWN_USE_CGNS=0

If this file already has some content, you can simply apply the above text to the end.

Save, and exit the editor.

Tip: you can can use systemd-delta to see which unit files on your RPi have been overridden or extended (via drop-ins), and systemctl cat <unit> (e.g., systemctl cat systemd-nspawn@debian-buster-64) to view the content of a unit file and all associated drop-in snippets.

Kernel Modules

To function correctly, the Docker daemon requires that a number of kernel modules are loaded, to function correctly. However, for good reasons (!), systemd-nspawn containers can't modprobe.

So, we need to set up these modules to be autoloaded by the host. Issue:

pi@raspberrypi:~ $ sudo nano -w /etc/modules

and append to that file:

overlay
bridge
br_netfilter

Leave the rest of the file as-is. Save, and exit nano.

Kernel Run-Time Configuration

Lastly, we also need to ensure (at least) IPv4 forwarding is turned on in the kernel. Again, we need to arrange this outside the container. To do so persistently, issue:

pi@raspberrypi:~ $ sudo nano -w /etc/sysctl.conf

and uncomment (i.e., remove the leading # from) the following line, so it reads:

net.ipv4.ip_forward=1

Leave the rest of the file as-is. Save, and exit nano.

Finally, ensure you have no unsaved work, and then reboot your RPi to have all the changes taken up.

Once your system comes back up, you are in a position to install 64-bit Docker!

Installing Docker in the 64-bit Debian Buster Container

To begin, open a 64-bit shell on your RPi (either using System Tools→Terminal (64-bit), or by running ds64-shell from a regular, 32-bit Raspbian terminal).

Then, following these instructions (copied below for convenience), issue:

Although believed correct at the time of writing, the below steps are given in somewhat telegraphic style, and the required procedure may also change somewhat over time. Always treat the official Docker instructions (just linked) as your primary reference, if in doubt.

pi@debian-buster-64:~ $ sudo apt-get -y remove docker docker-engine docker.io containerd runc

This will remove any prior copy of Docker from the systemd-nspawn container, as it is better to start fresh. It's fine if the above reports that only some, or none, of the cited packages is/are already installed.

Next, issue:

pi@debian-buster-64:~ $ sudo apt-get update
pi@debian-buster-64:~ $ sudo apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg2 \
    software-properties-common
pi@debian-buster-64:~ $ curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -

to install the prerequisites to allow apt to fetch a repository over HTTPS, and the official Docker repo GPG key.

Once complete, verify the key; issue:

pi@debian-buster-64:~ $ sudo apt-key fingerprint 0EBFCD88

pub   4096R/0EBFCD88 2017-02-22
      Key fingerprint = 9DC8 5822 9FC7 DD38 854A  E2D8 8D81 803C 0EBF CD88
uid                  Docker Release (CE deb) <[email protected]>
sub   4096R/F273FCD8 2017-02-22

The exact details printed may differ slightly from the above, but the key fingerprint should be as shown.

Assuming the fingerprint matches, install the official Docker repo (and update the local metadata to reflect this); issue:

pi@debian-buster-64:~ $ sudo add-apt-repository \
   "deb [arch=arm64] https://download.docker.com/linux/debian \
   $(lsb_release -cs) \
   stable"
pi@debian-buster-64:~ $ sudo apt-get update

Now you can actually install Docker! Issue:

pi@debian-buster-64:~ $ sudo apt-get install -y docker-ce docker-ce-cli containerd.io

Testing

Still working within the 64-bit shell, you can now test out Docker, by running its 'hello world' program. Issue:

pi@debian-buster-64:~ $ sudo docker run hello-world

and with luck, you should see terminal output similar to that shown below:

Obviously, there may be small differences in the output you see depending on exact version, locale etc.

If so, then congratulations! You now have Docker running inside your 64-bit Buster systemd-nspawn container.

You can find some useful aarch64 Docker container images here.

For example, download and start an ubuntu (CLI) image:

pi@debian-buster-64:~ $ sudo docker run -it ubuntu

Have fun ^-^

Hints and Tips

Although it is possible to install and run GUI-based applications using Docker, setting up the 'wiring' to display these cleanly in the guest desktop is slightly more involved, and out of scope for these brief notes.

The approach covered in this article does, however, work well for installing e.g. server nodes and the like, as the host network interfaces are (by default) available within the 64-bit container, and the container is started automatically at system boot. For further details on starting your Docker containers themselves automatically, please see e.g. these notes.

Tip: to allow the pi user account to access the docker command without sudo, execute the following inside the container:

pi@debian-buster-64:~ $ sudo usermod -aG docker pi

Acknowledgements

This tutorial was inspired by this article on the ArchLinux wiki.