refactor: cross compile for arm64 image build #222

KevFan · 2023-10-24T08:29:02Z

Building arm64 image via QEMU takes over 4 hours on github actions which makes the development / release process less than ideal.

Building this image via cross compile decreases this to ~20mins

Changes are based on:

On Mac M1, tested this with:

amd64 - docker pull quay.io/kuadrant/limitador:quicker-workflow --platform=linux/amd64

arm64 - docker pull quay.io/kuadrant/limitador:quicker-workflow

KevFan · 2023-10-24T08:33:26Z

limitador-server/Cargo.toml

@@ -37,6 +37,8 @@ const_format = "0.2.31"
 lazy_static = "1.4.0"
 clap = "4.3"
 sysinfo = "0.29.7"
+openssl = { version = "0.10.57", features = ["vendored"] }


Required updating openssl - https://github.com/Kuadrant/limitador/actions/runs/6573992478/job/17858112986

Otherwise will hit this error - https://stackoverflow.com/questions/68871193/pkg-config-error-during-rust-cross-compilation

Dockerfile

KevFan · 2023-10-24T08:39:53Z

.github/workflows/build-image.yaml

+          file: ${{ matrix.dockerfile }}
+          platforms: |
+            ${{ matrix.platform }}
+          provenance: false


Cross compiling can produce unknown manifests

GitHub Action produces unknown architecture and OS docker/build-push-action#820 (comment)

https://github.com/Kuadrant/limitador/actions/runs/6611529337/job/17955607182

Under inspect images

Or https://quay.io/repository/kuadrant/limitador?tab=tags&tag=33d19929330db5302ed3917266fcbf5ec0271fa8

Child Manifests

Setting provenance to false looks to fix this

KevFan · 2023-10-24T10:48:04Z

Dockerfile.aarch64

Failed to use ubi as the build / run stage due issues mentioned by @eguzki in #214 #215

So the main discussion here is are we fine with using community / non-productised images for this dockerfile / build for arm64 ? 🤔

Personally I dont think it should block this due to:

Downstream requirements should generally not leak into the upstream

Productised builds should be built on a different infra, which should instead use a "productised" Dockerfile which must use productised images as a requirement.

The main Dockerfile which use the ubi image can serve as the "productised" dockerfile which both the amd64 & arm64 image can be built from anyway. Not sure is this "productised" dockerfile usually hosted / provided elsewhere anyway in which case the main dockerfile should follow a similar structure to this dockerfile instead as it's simpler overall

The quicker build time (20mins) outweighs the downstream requirement to use productised images here since it delays the development flow / community release flow significantly (4 hours to build arm64 image)

Disadvantages to this is that:

Need to maintain 2 dockerfiles

The dockerfiles uses different build / runtime stages to each other currently

Potentially get rate limited by dockerhub occassionally since we use community images

Interested in peoples thoughts on this espeically @alexsnaps @didierofrivia @eguzki @gsaslis

in case it helps, the community Dockerfile is different from the product one - but it makes perfect sense to keep the differences to a minimum.

UBI images are publicly available and should be preferred whenever possible. Check out Reasons to Use UBI at https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image.

Personally, I don't see this as leaking requirements from the downstream into the upstream, but as recognizing that behind open source there's a business. We should not kill the business, neither punish the upstream. Instead, we need to find a balance that lets both to thrive as efficient as possible. Generally, UBI is a good compromise.

In contrast, our experience in the past of keeping multiple Dockerfiles and especially with different base images between upstream and downstream has been total chaos. A lot worse than waiting 4 hours for a build to finish. Believe me!

If building for arm64 on UBI base is as inefficient as x12 slower, pressing Red Hat to fix that should be our top priority.

Meanwhile, maybe building for arm64 could be triggered automatically only when merging to main/release-* branches and manually on-demand?

Thanks @guicassolato, good points ! 👍 I guess the only thing is Red Hat support of rust seems pretty minimal ( https://catalog.redhat.com/software/containers/rhel8/rust-toolset/5b9c6aa85a134643ef2ecc63?architecture=amd64&image=607765c8d70cc51ae2772196 / https://catalog.redhat.com/software/containers/search?gs&q=rust) and the main issue I believe to building on ubi using cross compile is it's missing a libc6-dev-arm64-cross equivalent that is available in Debian (#215). I can reach out to the ubi team on their thoughts to supporting this, but I'm guessing this may take a while.

I guess my perspective is that for the upstream, I'm not sure do we need to / should confine ourselves to using ubi8 at the expense of the developer experience / release flow for limitador itself. Especially if / when the productised build comes, it will be built elsewhere with another dockerfile that needs to be maintained regardless.

Many other Red Hat projects looks to use the community images in their upstream (golang equivalents seems a lot simpler):

https://github.com/3scale/3scale-operator/blob/master/Dockerfile#L2

https://github.com/3scale-ops/marin3r/blob/main/Dockerfile#L2

A compromise is that we can have the runtime stage to be ubi but the build stage to use a community image which looks to have worked for me after some tinkering (docker pull quay.io/kuadrant/limitador:ubi9-test) 🤔 :

82e978e

https://github.com/Kuadrant/limitador/actions/runs/6639578417

https://quay.io/repository/kuadrant/limitador?tab=tags&tag=ubi9-test

Yeah, I agree maintaining multiple Dockerfiles may not be worth it in the end (for example, with the current approach, the arm64 build might break on runtime due to glibc versioning mismatch that isn't caught until someone runs this dockerfile / pulls the arm64 image and runs it 🤔). I'll defer to @alexsnaps @didierofrivia @eguzki for the best course of action as they are the main recent contributiors for whether these changes are worth the potential future headache 🤔

To add one more point of view:

In general, keeping the same parent image between community + product build (and trying to minimize other differences) actually helps us deliver more value to the community - because we need to spend less time porting to the product build those community changes that we want to ship as part of the product. That way, we can spend more of our time working on the community project ;)

My 5 cents:

We use rust based images in community. It makes life easier (as of today) for the upstream project. And the upstream is what gets the daily work. Downstream is automated and who cares if it takes 20min or 2H? And maybe downstream does not even require cross compiling. At least for ARM. Maybe requires images for RH products in IBM (powerpc and z mainframes).

I reckon the benefits of being aligned (regarding the base image) with downstream and the use of UBI images. But only when the cost is not higher than the benefit. IMO the cost is higher than the benefits as of today.

Let me support my opinion with one illustrating example:

Rust based dockerfile

FROM rust:1.72 as limitador-build RUN apt update && apt upgrade -y RUN apt install -y protobuf-compiler clang WORKDIR /usr/src/limitador ARG GITHUB_SHA ENV GITHUB_SHA=${GITHUB_SHA:-unknown} ENV RUSTFLAGS="-C target-feature=-crt-static" COPY . . RUN cargo build --release``

Ubi based dockerfile

FROM registry.access.redhat.com/ubi8/ubi:8.7 as limitador-build ENV CARGO_NET_GIT_FETCH_WITH_CLI=true ARG RUSTC_VERSION=1.72.0 # the powertools repo is required for protobuf-c and protobuf-devel RUN dnf -y --setopt=install_weak_deps=False --setopt=tsflags=nodocs install \ http://mirror.centos.org/centos/8-stream/BaseOS/`arch`/os/Packages/centos-gpg-keys-8-6.el8.noarch.rpm \ http://mirror.centos.org/centos/8-stream/BaseOS/`arch`/os/Packages/centos-stream-repos-8-6.el8.noarch.rpm \ && dnf -y --setopt=install_weak_deps=False --setopt=tsflags=nodocs install epel-release \ && dnf config-manager --set-enabled powertools RUN PKGS="gcc-c++ gcc-toolset-12-binutils-gold openssl-devel protobuf-c protobuf-devel git clang kernel-headers" \ && dnf install --nodocs --assumeyes $PKGS \ && rpm --verify --nogroup --nouser $PKGS \ && yum -y clean all RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --no-modify-path --profile minimal --default-toolchain ${RUSTC_VERSION} -c rustfmt -y WORKDIR /usr/src/limitador ARG GITHUB_SHA ENV GITHUB_SHA=${GITHUB_SHA:-unknown} ENV RUSTFLAGS="-C target-feature=-crt-static" COPY . . RUN source $HOME/.cargo/env \ && cargo build --release

UBI based image requires the usage of centos repos in the limitador project (protobuf-c and protobuf-devel packages)

In the effort to use the UBI as base image, for ARM images, this PR is building on rust based images and then in the next stage fitting the binary into an ubi image. This is, IMO, an additional cost of building in UBI for the limitador project. Some extra complexity we will need to maintain and that will be the source of future issues for sure.

.github/workflows/build-image.yaml

eguzki · 2023-12-12T10:44:09Z

LGTM

I would merge this and open discussion about the community base image.

pehala · 2023-12-11T13:59:10Z

Dockerfile

@@ -2,19 +2,19 @@
 # Build Stage
 # ------------------------------------------------------------------------------

-FROM registry.access.redhat.com/ubi8/ubi:8.7 as limitador-build
+FROM registry.access.redhat.com/ubi9/ubi:9.2 as limitador-build


Could you also use the bullseye image as you do in aarch64? In that way, both of the builds would use the same images and should be much closer than are right now

I can, as long we are "happy" that this Dockefile will no longer be a reflection of what the productised Dockerfile will be 🤔

.github/workflows/build-image.yaml

eguzki · 2023-12-12T10:52:49Z

Dockerfile.aarch64

+# Build Stage cross compiling
+# ------------------------------------------------------------------------------
+
+# Use bullseye as build image instead of Bookworm as ubi9 does not not have GLIBCXX_3.4.30


Building arm64 image via QEMU takes over 4 hours on github actions which makes the development / release process less than ideal. Building this image via cross compile decreases this to ~20mins

KevFan · 2024-01-08T15:23:12Z

Rebased with main

didierofrivia · 2024-01-08T16:34:23Z

.github/workflows/build-image.yaml

-          dockerfiles: |
-            ./Dockerfile
+          context: .
+          labels: ${{ steps.meta.outputs.labels }}


Which labels are set by default from the output of the meta step?

This is the JSON output for this build step (taken from the workflow logs):

{ "tags": [ "quay.io/kuadrant/limitador:quicker-workflow" ], "labels": { "org.opencontainers.image.created": "2024-01-08T15:23:11.687Z", "org.opencontainers.image.description": "Rate limiter", "org.opencontainers.image.licenses": "Apache-2.0", "org.opencontainers.image.revision": "a39035119c7899901dfc785e[59](https://github.com/Kuadrant/limitador/actions/runs/7449493828/job/20266248072?pr=222#step:4:68)380d52b9b11986", "org.opencontainers.image.source": "https://github.com/Kuadrant/limitador", "org.opencontainers.image.title": "limitador", "org.opencontainers.image.url": "https://github.com/Kuadrant/limitador", "org.opencontainers.image.version": "quicker-workflow" }, "annotations": [ "manifest:org.opencontainers.image.created=2024-01-08T15:23:11.[68](https://github.com/Kuadrant/limitador/actions/runs/7449493828/job/20266248072?pr=222#step:4:77)7Z", "manifest:org.opencontainers.image.description=Rate limiter", "manifest:org.opencontainers.image.licenses=Apache-2.0", "manifest:org.opencontainers.image.revision=a39035119c7899901dfc785e59380d52b9b11986", "manifest:org.opencontainers.image.source=https://github.com/Kuadrant/limitador", "manifest:org.opencontainers.image.title=limitador", "manifest:org.opencontainers.image.url=https://github.com/Kuadrant/limitador", "manifest:org.opencontainers.image.version=quicker-workflow" ] }

KevFan marked this pull request as draft October 24, 2023 08:29

KevFan commented Oct 24, 2023

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

KevFan commented Oct 24, 2023

View reviewed changes

KevFan force-pushed the quicker-workflow branch from c2ab004 to 6ef75c2 Compare October 24, 2023 08:52

KevFan commented Oct 24, 2023

View reviewed changes

KevFan commented Nov 3, 2023

View reviewed changes

.github/workflows/build-image.yaml Outdated Show resolved Hide resolved

KevFan force-pushed the quicker-workflow branch from 973e864 to 5d0bd09 Compare November 6, 2023 12:49

KevFan force-pushed the quicker-workflow branch from 5d0bd09 to b2ad142 Compare December 7, 2023 13:13

KevFan added the kind/enhancement New feature or request label Dec 7, 2023

KevFan self-assigned this Dec 7, 2023

KevFan added size/medium kind/investigation dependencies Pull requests that update a dependency file labels Dec 7, 2023

KevFan marked this pull request as ready for review December 7, 2023 13:14

KevFan requested review from alexsnaps, eguzki and didierofrivia December 7, 2023 14:07

alexsnaps added this to the Limitador Server v1.4.0 milestone Dec 8, 2023

eguzki reviewed Dec 12, 2023

View reviewed changes

.github/workflows/build-image.yaml Outdated Show resolved Hide resolved

.github/workflows/build-image.yaml Outdated Show resolved Hide resolved

pehala reviewed Dec 12, 2023

View reviewed changes

eguzki reviewed Dec 12, 2023

View reviewed changes

KevFan force-pushed the quicker-workflow branch from b2ad142 to db68416 Compare January 5, 2024 16:21

KevFan added 3 commits January 8, 2024 15:22

refactor: cross compile for arm64 image build

c775a73

Building arm64 image via QEMU takes over 4 hours on github actions which makes the development / release process less than ideal. Building this image via cross compile decreases this to ~20mins

refactor: use ubi9 for Dockerfile & run stage in Dockerfile.aarch64

4f38269

refactor: algin dockerfile build stages to use rust:1.72-bullseye

a390351

KevFan force-pushed the quicker-workflow branch from db68416 to a390351 Compare January 8, 2024 15:22

didierofrivia reviewed Jan 8, 2024

View reviewed changes

refactor: SHA image tag only for main branch

c1b029c

alexsnaps approved these changes Jan 17, 2024

View reviewed changes

KevFan merged commit 4ebe4ac into main Jan 17, 2024
20 checks passed

eguzki deleted the quicker-workflow branch January 18, 2024 13:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: cross compile for arm64 image build #222

refactor: cross compile for arm64 image build #222

KevFan commented Oct 24, 2023 •

edited

Loading

KevFan Oct 24, 2023

KevFan Oct 24, 2023

KevFan Oct 24, 2023

gsaslis Oct 24, 2023

guicassolato Oct 25, 2023

KevFan Oct 25, 2023

gsaslis Oct 26, 2023

eguzki Dec 12, 2023

eguzki Dec 12, 2023

eguzki Dec 12, 2023

eguzki commented Dec 12, 2023

pehala Dec 11, 2023

KevFan Jan 5, 2024

eguzki Dec 12, 2023

KevFan commented Jan 8, 2024

didierofrivia Jan 8, 2024

KevFan Jan 9, 2024

refactor: cross compile for arm64 image build #222

refactor: cross compile for arm64 image build #222

Conversation

KevFan commented Oct 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eguzki commented Dec 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KevFan commented Jan 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KevFan commented Oct 24, 2023 •

edited

Loading