CAPA Flatcar

[alex-dabija]

User Story

-As a cluster admin, I want to use Flatcar as the operating system on all Cluster API cluster nodes in order to have a container oriented operating system with all the hardening and settings of the current Giant Swarm clusters.

Towards epic.

Background

Cluster API for AWS (CAPA) clusters use the default Ubuntu image.

[calvix]

testing PRs:
giantswarm/cluster-aws#49
giantswarm/cluster-api-provider-aws-app#106
giantswarm/cluster-api-app#54

[calvix]

first bump on the road - kubernetes-sigs/cluster-api#6403
Seems like base64 encoding is not properly supported for files in kubeadmconfig

[calvix]

so everything tested, flatcar is running on all nodes including bastion, ssh works as well, I will keep the branch for cluster-aws until we will have a new version fo CAPI and CAPA controllers which support this flatcar release, we need to wait until the new releases ar mode aupstream

[calvix]

did another quick PoC and found a circular dependency that will need to be solved before we can start working on Flatcar

with Flatcar you need to specify IAM roles that will be used to access the ignition files stored in S3:

  s3Bucket:
    controlPlaneIAMInstanceProfile: control-plane-{{ include "resource.default.name" $ }}
    name: {{ include "aws-region" . }}-capa-{{ include "resource.default.name" $ }}
    nodesIAMInstanceProfiles:
    {{- range $name, $value := .Values.nodePools | default .Values.defaultMachinePools }}
    - nodes-{{ $name }}-{{ include "resource.default.name" $ }}
    {{- end }}

during the infrastructure initialization phase capa-controller is creating S3 buckets and defines policies on which roles can access the S3 bucket but that will fail as the IAM role does not exist yet due to this circular dependency:

capa-iam-operator needs for some reason irsa-cloudfront-secret to be available before creating the IAM roles for bastion and control plane ( it can create machine pool role), irsa-operator cannot create the secret until the service account secret key is generated - service account key is not generated because the infrastructure is not yet in provisioned state due capa-controller not being able to prepare the S3 buckets

So it goes like this

->>> capa-controller won't finish provision of cluster infrastructure -> waiting for IAM Role of CP
capa-iam-operator wont create CP IAM role -> waiting for IRSA secret
irsa-operator wont create irsa secret -> waiting for SA secret
capi-controller wont create SA secret -> waiting for provisioned infrastructure ->>>

I think we should break the dependency of capa-iam-operator on IRSA secret if that is possible and create basic roles, the IRSA role can be created later when the necessery info is available but it should not block the other roles

[AndiDog]

If our dependency is mostly on the dynamic CloudFront domain, we could use a fixed alias domain instead. I think that's a feature of irsa-operator already. Then we don't have to wait on the secret to read the domain, but make an assumption.

See also #2387.

[alex-dabija]

Blocked by #2420.

[calvix]

Before merging we need to update IAM role for all existing MCs to include new S3 bucket permissions

[calvix]

Done, PR merged, release created and the permissions were updated