Slow container boot due to double Unit startup

[Tenzer]

Hi

I am currently evaluating using Unit for a Python WSGI application, and I've found the Docker containers with Unit to be pretty slow to start up.

Part of this is down to the Python application being sizable and the number of processes we are running, but it's only made worse by the docker-entrypoint.sh script, which loads in the configuration for the application from /docker-entrypoint.sh/ into Unit, which then starts the application with the number of processes we want, then shuts down Unit for it to then have to go through the work again.

Is there any good reason for Unit to be restarted?
Could we instead place our configuration in /var/lib/unit/ to avoid this extra work?

[ac000]

Hi

I am currently evaluating using Unit for a Python WSGI application, and I've found the Docker containers with Unit to be pretty slow to start up.

Part of this is down to the Python application being sizable and the number of processes we are running, but it's only made worse by the docker-entrypoint.sh script, which loads in the configuration for the application from /docker-entrypoint.sh/ into Unit, which then starts the application with the number of processes we want, then shuts down Unit for it to then have to go through the work again.

Hmm, that does sound nuts!

Is there any good reason for Unit to be restarted?

Good question, I'm not really familiar with this docker stuff.

Could we instead place our configuration in /var/lib/unit/ to avoid this extra work?

You could try this completely untested patch, it should leave Unit running after its initial startup/config.

diff --git a/pkg/docker/docker-entrypoint.sh b/pkg/docker/docker-entrypoint.sh
index 4646409f..3c3e95fd 100755
--- a/pkg/docker/docker-entrypoint.sh
+++ b/pkg/docker/docker-entrypoint.sh
@@ -76,26 +76,10 @@ if [ "$1" = "unitd" ] || [ "$1" = "unitd-debug" ]; then
             curl_put /usr/share/unit/welcome/welcome.json "config"
         fi
 
-        echo "$0: Stopping Unit daemon after initial configuration..."
-        kill -TERM $(/bin/cat /var/run/unit.pid)
-
-        for i in $(/usr/bin/seq $WAITLOOPS); do
-            if [ -S /var/run/control.unit.sock ]; then
-                echo "$0: Waiting for control socket to be removed..."
-                /bin/sleep $SLEEPSEC
-            else
-                break
-            fi
-        done
-        if [ -S /var/run/control.unit.sock ]; then
-            kill -KILL $(/bin/cat /var/run/unit.pid)
-            rm -f /var/run/control.unit.sock
-        fi
-
         echo
-        echo "$0: Unit initial configuration complete; ready for start up..."
+        echo "$0: Unit initial configuration complete"
         echo
+
+       exit 0
     fi
 fi
-
-exec "$@"

[Tenzer]

Thanks for the tip. I ended up putting it into a separate script so it's easier to document and work with:

#!/bin/bash

# Run Unit so it can initialise the configuration, this makes subsequent startups faster.
# https://github.com/nginx/unit/discussions/1114

CONFIG=$1
CONTROL_SOCKET=/tmp/control.unit.sock
PID_FILE=/tmp/unit.pid

# Start Unit
unitd --control unix:${CONTROL_SOCKET} --pid $PID_FILE

# Wait for the control socket to be created
for _ in $(seq 5); do
    if [ ! -S $CONTROL_SOCKET ]; then
        sleep 1
    else
        break
    fi
done

# Post configuration to Unit, blocks until loaded
curl --silent --request PUT --data "@$CONFIG" --unix-socket $CONTROL_SOCKET http://localhost/config

# Stop Unit again
kill "$(cat $PID_FILE)"

Looks to work a treat!

[tippexs]

Will this script be copied into the build container? You could use Dockers multi-stage build to build the state and COPY --builder into the application container. I think this should find its way in our documentation including scripts like this in an examples directory here on github.

[Tenzer]

Yes, I copy it into the container together with our application code and then run it with a RUN statement.

For now, I've put it in-line as part of the container build, as it helps ensure we are using the same version of Unit for generating the configuration as we do at run time.

[tippexs]

Thanks for making this clear! Let me draft a quick PHP (I am a PHP guy sorry) Multi-Stage with the script and see how that goes. Will share that in a minute

[tippexs]

This is my Multi-Stage Dockerfile

FROM unit:php as builder
# Copy Application to the build container
COPY ./src /var/www/html
# Copy the new `init` script into docker-entrypoint.d
COPY ./scripts/init.sh /tmp/init.sh
# COPY Unit Configuration
COPY config.json /tmp/config.json

RUN ls -lah /var/lib/unit/
RUN /tmp/init.sh /tmp/config.json

FROM unit:php
COPY ./src /var/www/html

COPY --from=builder /var/lib/unit/ /var/lib/unit/

# Adding CMD Line just for documentation. This is part of the official Image anyway.
CMD ["unitd", "--no-daemon", "--control", "unix:/var/run/control.unit.sock"]

The init.sh is your script. Thanks for that! The configuration is really simple:

{
  "listeners": {
    "*:80": {
      "pass": "routes"
    }
  },
  "routes": [
    {
      "action": {
        "share": "/var/www/html/webroot$uri",
        "fallback": {
          "pass": "applications/testapp/index"
        }
      }
    }
  ],

  "applications": {
    "testapp": {
      "type": "php",
      "processes": 40,
      "environment": {
        "DEBUG": "true"
      },
      "targets": {
        "index": {
          "root": "/var/www/html/",
          "script": "index.php"
        }
      }
    }
  }
}

The COPY --from=builder will work with NJS and Certs as well which is another performance improvement. I actually like the idea of doing it this way. A LOT cleaner than blindly copy the state dir and faster than the current solutions. For development environments I am reconfiguring Unit via an exposed TCP Control Endpoint from IntelliJ so that not where I need the performance but for Production this is a solid solution and it makes sense to move this into the build step.

Another thing I would like to add to this example are healthchecks for docker compose and a way to manipulate the environment after deployed the application to production. This is a very common use case for me in my environments. I am having a healthcheck.php that will only return 201 if the environment was set by my CI Pipeline. Simple but super effective to not mark the container as healthy before everything is ready to serve traffic.

[ac000]

Hi @Tenzer

One immediate issue I can see with that, is docker-entrypoint.sh then will stay as PID 1 inside the container, instead of handing that over to unitd via the last exec "$@" line.

Hmm, is it normal to not run an init(1) process in a container?

[Tenzer]

In my experience people generally don't use init since containers generally only run one program each. That said, there are tools such as tini and dumb-init that can help in situations where you end up with zombie processes that aren't reaped and such, which bash won't handle, but if you instead run the program (ie. unitd) directly as PID 1, that tends to not be a problem.

[ac000]

Thanks for clarifying... The last time I used containers was around ~7 years ago using LXC with full OS's inside.

[tippexs]

Hi @Tenzer Thanks for reporting this. As you can see this is an "challenge" we are currently actively investigating and we are about to enhance. What @lcrilly already mentioned - you are good to use Units state directory for now and see if this will speed up your container startup (it should) OR to make the state part of the docker build process and populate this into the state.

[bunny-therapist]

I understand that dynamically configuring unit via a control socket is some fundamental design decision, but am I wrong to think that it does complicate matters when running in containers? I am trying to replace nginx + python-app server runner with unit for our deployments in kubernetes, and we have restrictions such as a read-only filesystem (except /tmp, which is mounted on startup and thus cannot be populated during build), can't run the main pod process as root, and must be able to configure things like client max body size for a deployment (i.e., it cannot be built into the image).

With nginx, we could just create a config file from environment variables in the entrypoint, then launch nginx with that config file. To do the same with unit, unit must be started as a daemon, then configured via control socket, then killed, then started again.

I understand this may not be what you intended and may even be architecturally difficult, but I feel like it would be much simpler if one could have the option of simply starting unit from the command line with a config file - without a control socket, e.g., unitd --config config.json. (If you wanted to, you could even make it so that if you provide a config file, you are not allowed provide a control socket, possibly there may even be some resources that could be saved since you would not need to manage any dynamic configuration. You could also make it so that a static config is only possible in no-daemon mode, and that you have to provide other directories and can't use /var/lib/unit).