[RFC] Orchestrator of choice

[aldy505]

We need just right orchestrator for TokoBapak.

It should:

• Be able to orchestrate Docker containers
• Do health checks on those Docker containers
• Accepts and load balance HTTP and gRPC connection
• Be lightweight, must not consume high processing power or RAM

This decision will of course be coupled with the API gateway of choice that exists on a separate thread.

cc @ii64 @zasdaym

[aldy505]

The way the deployment process works is (I guess, free for comments) by spawning a watchtower on the VM that we're using (probably will use Contabo VPS S -- that's 4 core CPU + 8 GB RAM). The container registry will be Github container registry -- it's free with any container size as long as it's a public repository -- which I think can easily be tracked because it's both manageable via each repository that creates the container image and via the whole organization packages (sample as seen here).

I am actually thinking of two ways: we provide both Hashicorp Nomad (or maybe k8s) and plain old Docker Compose. It's just a single node deployment, I think it's kinda overkill and wasteful if we're going for Nomad (or again, k8s) as the first priority.

[zasdaym]

Nomad is lightweight and can be used for a single node. We automatically got a cool dashboard and secret management also.

In short, the options are:

1. Docker Compose + https://containrrr.dev/watchtower/. Secrets will be managed by directly assigning environment variables or mounting files to containers. Pipelines don't need to contain any deploy job, as deployment is in a "pull" style.
2. Nomad with Docker driver. Secrets will be managed via Nomad Variables. Deployment side, each app will need a .nomad file, that will be used in the pipeline with command like: nomad run deploy.nomad.

[aldy505]

If we use Nomad, we would still need watchtower too right?

[zasdaym]

No, the deployment will be in a "push" style. Except if Watchtower supports custom "action" when a new version of an image is found.

[aldy505]

Watchtower does support that though. See https://containrrr.dev/watchtower/arguments/#without_updating_containers

[zasdaym]

Nice, so the deployment model is decided to be "pull". We wait for the other members of @TokoBapak/infrastructure about Docker Compose vs Nomad.

[aldy505]

Actually, I'd like to support both method. Provide Nomad for deployment on the running VM, and provide Compose for running on their local instances if they don't like much more stuff going on.

Are both of them the perfect method, or is there any other alternative that we can consider?

[ii64]

Nomad or Docker Compose will be the same at the end if we are using Watchtower to deploy container image, it interact directly to the Docker Engine socket and I don't think this way going to be recorded by Nomad server, cmiiw

[aldy505]

@ii64 Zasda said earlier that if we're using Nomad, we won't be needing Watchtower. Is that right?

I haven't use Nomad personally though, but I've used Watchtower. So, I don't really know about container update behaviour on Nomad.

[zasdaym]

@ii64 @aldy505 Watchtower will not interact directly with the Docker Engine socket, we'll use https://containrrr.dev/watchtower/arguments/#without_updating_containers so Watchtower will run nomad job -force-reschedule <job_id> when a new image is found, so every deployment will always be done via Nomad.

[zasdaym]

Another upside of using Nomad is we can provide cool Web UI for our team members. 😄