Running buildah in a EKS Fargate pod

[koweblomke]

I am trying to setup a CI/CD pipeline running on AWS EKS Fargate.
Currently we are using docker to build container images with the pods running on EC2 nodes in privileged mode.
I was hoping I could use buildah (or podman) to build container images on Fargate.

But I run into a problem:

Using https://www.redhat.com/sysadmin/podman-inside-kubernetes as reference I see we have run the pods either in privileged mode or we have to add a daemonset for the fuse-overlayfs or we have to add security capabilities.

neither of these options seem to be possible on AWS Fargate, we can't run pods in privileged mode. We are alse not able to run a daemonset with this fuse driver. And adding security capabilities also isn't allowed.

Does this mean there is no way we can use buildah on AWS Fargate?

[rhatdan]

How were you able to build container images with Docker in this environment before?

[koweblomke]

I wasn’t, this is my first try on Fargate.

At the moment we use EKS on self managed EC2 nodes. Here we run pods in privileged mode with docker in docker.

I wanted to try building images in non privileged mode without docker. This would also make it possible to build images in AWS Fargate. But I ran into these issues described above.

There is a blog which describes it is possible with kaniko: https://aws.amazon.com/blogs/containers/building-container-images-on-amazon-ecs-on-aws-fargate/
But I wanted to use buildah because of its flexibility.

[rhatdan]

Building container images requires one of three setups.
1 CAP_SYS_ADMIN which allows rootfull Podman/Buildah/Docker
2 CAP_SETUID & CAP_SETGID which allows rootless Podman & Buildah to build container images.
3 Kubernetes to launch a container with multiple UIDs inside of a user namespace. Then Podman and Buildah would be able to build with no Host Caps (Kubernetes is working on how to do this).

If your environment will not give you one of these then you will not able to build within it.

[koweblomke]

I guess this means we cannot build containers on AWS Fargate with Podman, Buildah and Docker.
So we need an EC2 instance for building Container Images until maybe option 3 is available.

Thanks for your reaction and clarification on this matter.