capi-helm-fluxcd-config

This repository contains configuration for deploying and operating Kubernetes clusters on OpenStack using a GitOps workflow.

The specific tools used to accomplish this are Cluster API for the Kubernetes provisioning, Flux CD for automation and sealed secrets for managing secrets.

Cluster API is a set of Kubernetes operators with associated custom resources that allow Kubernetes clusters to be provisioned and operated by creating instances of those resources in a "management" Kubernetes cluster.

The clusters created using configurations derived from this repository are "self-managed". This means that the Cluster API controllers and the resources defining the cluster are managed on the cluster itself.

Bootstrapping a cluster

The use of self-managed clusters creates a bootstrapping problem, i.e. how does the cluster get created if it manages itself? The solution to this is to use a one-off bootstrapping process that performs the following steps:

1. Create an ephemeral Kubernetes cluster using kind
2. Use the ephemeral cluster to create the Cluster API cluster
   • Install Flux on the ephemeral cluster
   • Use Flux to install the Cluster API controllers on the ephemeral cluster
   • Use Flux to create the Cluster API cluster using the ephemeral cluster
   • Wait for the Cluster API cluster to become ready
3. Get the kubeconfig file for the Cluster API cluster
4. Install Flux and the Cluster API controllers on the Cluster API cluster
5. Move the Cluster API resources from the ephemeral cluster to the Cluster API cluster
   • Suspend reconciliation of the Cluster API resources on the ephemeral cluster
   • Copy the Cluster API resources to the Cluster API cluster
   • Resume reconciliation of the Cluster API resources on the Cluster API cluster

At this point, the cluster is self-managed. The remaining steps set the cluster up to be managed using Flux going forward:

6. Use Flux to install the sealed secrets controller
7. Seal the cluster credentials using kubeseal
8. Commit the sealed secret to git and push it
9. Create a Flux source pointing at the git repository
   • If you are using git over SSH, this will generate an SSH keypair and prompt you to add the public key to the git repository as a deploy key
10. Create a Flux kustomization pointing at the cluster configuration

At this point, the cluster is self-managed using Flux and the ephemeral cluster is deleted.

This repository includes a Python script that performs these steps (see below).

Usage

First, create a copy of the repository (like a fork but detached).

# Clone the repository
git clone https://github.com/stackhpc/capi-helm-fluxcd-config.git my-cluster-config
cd my-cluster-config

# Rename the origin remote to upstream so that we can pull changes in future
git remote rename origin upstream

# Add the new origin remote and push the initial commit
git remote add origin <url>
git push -u origin main

Create a cluster configuration under the ./clusters directory, following the example.

As part of this process, you must create an application credential for the target project in OpenStack. The credentials.yaml file in the cluster configuration must then be updated with the following information:

• The OpenStack auth URL and region, which you can get from your cloud admin
• The ID and secret of the application credential
• The ID of the target project in OpenStack

During the bootstrap process, this file will be encrypted using kubeseal and pushed to the git repository. Be careful to NEVER commit the unencrypted version.

Once you are happy with your cluster configuration, you can bootstrap the cluster.

The following tools are required to execute the bootstrap script:

• Python 3
• git
• Docker
• kind
• kustomize
• kubeseal
• flux CLI

Next, create a Python environment with the required dependencies installed, e.g.:

python3 -m venv ./.venv
source .venv/bin/activate
pip install -U pip
pip install -r requirements.txt

Then run the bootstrap command for the cluster that you created:

./bin/manage bootstrap my-cluster

Once the bootstrap is complete, you will have a cluster that can be managed by making changes to the git repository.

Accessing the cluster

The bootstrap process produces a kubeconfig file that can be used to access the cluster, which is written into the directory containing the cluster configuration.

WARNING

The kubeconfig file is NOT committed to git, and if lost is difficult to recover.

It should be stored somewhere safe where it can be shared with team members who need it.