ci(docker): rework workflow (3rd time) #289

hero-intelligent · 2023-09-21T16:30:05Z

Background

Thanks for your patience. Maybe this will help. Only the tested will be squashed into one commit and pushed here. I'll try not to close pr so frequently. Feel free to review my working status in my repo.

Checklist

The Pull Request has been fully tested
There's an entry in the CHANGELOGS
There is a user-facing docs PR against https://github.com/daeuniverse/daed

Full Changelogs

ci(docker): docker build once and push to multiple repos
ci(docker): docker build and push with tags
ci(docker): come back arm64 platform support (Experimental)
ci(docker): publish dockerhub image
ci(docker): action version upgrade
ci(docker): multiple action trigger
ci(docker): build with tag on dashboard
ci(docker): i386, armv7 armv8 platform support (Experimental)

Issue Reference

#280
#267
#286

Test Result

Action page
Dockerhub page
Download log archive
View raw logs

It should work, but we need cooperation.

hero-intelligent · 2023-09-22T11:59:59Z

Achievements:

Compared to current stage, my version enables i386, armv7, armv8 and arm64(maybe a duplicate of armv8), besides of amd64. Image will be built once and then push to ghcr, quay and dockerhub, with tag of either version or date, as well as latest.

Once upgraded, most of use cases will be satisfied.

Ready for review. @kunish @mzz2017 @yqlbu

Limitations and explanation:

I've tried my best. As for paralleled image build process using matrix, I've kind of gave up.

The whole build process will run about one hour, which may cause a problem of hitting the limit of free action access of 2000 minutes per month per account.

As for the parallel one, a single run takes only about half an hour, but now it is entirely unusable.
Images now can only push to one registry. If you want to push to another registry, you have to run multiple times, which takes even longer than the chronological one.
As for supporting armv5 and v6, it can only be achieved by parallel build process, for images must be built and pushed at the same time, otherwise the latter push will overwrite the former one. Besides, I need to write another Dockerfiles to achieve these architecture.

Mips64le (solely debian based)
armv5 (solely debian based)
armv6 (solely alpine based)
Others (self build from scratch)

Goals not achieved:

Platform support for more platforms
Parallel build process

Later I'll update the documents. Thanks for your patience again. I also learned a lot from this experience.

hero-intelligent · 2023-10-19T15:16:35Z

Excuse me? Has anyone pay any attention to this pr yet?

piyoki · 2023-10-19T15:34:48Z

@hero-intelligent When it comes to build candidates, I think we ONLY need to take the following architectures into consideration:

amd64
arm64

publish.Dockerfile

piyoki · 2023-10-19T15:49:48Z

From your sample workflow run, everything meets our expectations except the list of build candidates (platforms). Ideally, we should ONLY consider offering this containerization deployment alternative to amd64 and arm64 users and ignoring all other unnecessary platforms given the consideration of less common adoption of other platforms such as ppc64le.

hero-intelligent · 2023-10-20T03:42:27Z

In my opinion, at least i386 and armv7 should be supported, as there are still be some people willing to run this program on these machines, but they may not able to build on their own.
For ARG DAED_VERSION=self-build, the self-build needn't be deleted, as in CI it will be overwritten to TAG, while coming into effect when self built without passing into arguments.
For these lengthy stage name, I did it on purpose to avoid non-necessary edit. If you don't like it, I'll accept your advice.

full change log: ci(docker): come back arm64 platform support (Experimental) ci(docker): publish dockerhub image ci(docker): action version upgrade ci(docker): multiple action trigger -------------------- * add dockerhub * fix workflow error * fix workflow error * dockerhub * no quay * nothing important * dockerhub username * nothing important * test architecture * test repo * new platforms * nth important * nth important * ci(build): publish docker image * ci: upgrade docker build workflow * ci: error fix * nth important * ci: standardize Dockerfile * ci: docker build roll back * correct mistake recursive * test push * ci(docker): rework docker publish workflow, push to dockerhub Signed-off-by: hero-intelligent <[email protected]> --------- Signed-off-by: hero-intelligent <[email protected]>

* add dockerhub * fix workflow error * fix workflow error * dockerhub * no quay * nothing important * dockerhub username * nothing important * test architecture * test repo * new platforms * nth important * nth important * ci(build): publish docker image * ci: upgrade docker build workflow * ci: error fix * nth important * ci: standardize Dockerfile * ci: docker build roll back * correct mistake recursive * test push * ci(docker): build with tag on dashboard * ci(docker):upgrade and rework docker ci * ci(docker): comment * ci: use DOCKERHUB_TOKEN --------- Signed-off-by: hero-intelligent <[email protected]>

* add dockerhub * fix workflow error * fix workflow error * dockerhub * no quay * nothing important * dockerhub username * nothing important * test architecture * test repo * new platforms * nth important * nth important * ci(build): publish docker image * ci: upgrade docker build workflow * ci: error fix * nth important * ci: standardize Dockerfile * ci: docker build roll back * correct mistake recursive * test push * ci(docker): build with tag on dashboard * ci(docker):upgrade and rework docker ci * ci(docker): comment * ci: use DOCKERHUB_TOKEN * ci(docker): more platform, matrix(disabled), build test * ci(docker): needs build-web * ci(docker):armv5 won't build Armv5 can build on debian but cannot build on alpine Armv6 can build on alpine but cannot build on debian Signed-off-by: hero-intelligent <[email protected]> * not pullute Dockerfile Signed-off-by: hero-intelligent <[email protected]> * Delete not-tested .github/workflows/test-publish-docker-image.yml Signed-off-by: hero-intelligent <[email protected]> * Delete deprecated .github/workflows/docker-ci.yml Signed-off-by: hero-intelligent <[email protected]> * ci(docker): publish-docker-image.yml ci(docker): build web outside the Dockerfile ci(docker): docker enabled armv7 and armv8 Signed-off-by: hero-intelligent <[email protected]> --------- Signed-off-by: hero-intelligent <[email protected]>

Signed-off-by: hero-intelligent <[email protected]>

~~self-build~~ Co-authored-by: kev <[email protected]> Signed-off-by: hero-intelligent <[email protected]>

kunish · 2023-10-20T08:25:51Z

.github/workflows/publish-docker-image.yml

+    types: [prereleased,released]
+
+env:
+  REGISTRY_IMAGE: daeuniverse/daed


This variable is no longer necessary

kunish · 2023-10-20T08:30:11Z

.github/workflows/publish-docker-image.yml

+        # with:
+        #   submodules: 'recursive'


We did it on purpose, as those source code files under submodules are also used by linux distro packages, such as archlinux and gentoo. For example, the packaging process of gentoo is done inside a isolated environment, which means we can't do git clone as we desire.

Oh, does build-web need sub-modules as well? I thought this process just renders htmls.

Yeah, they are requested by package maintainers

hero-intelligent · 2023-10-20T08:58:43Z

now the image in this ghcr.io is still not usable. please build again as soon as possible, if this pr can be merged.

kunish · 2023-10-20T09:10:06Z

Awesome, let's ship it!

hero-intelligent · 2023-10-20T09:26:42Z

This build process is time consuming, which takes about one hour. You can disable on push after this run time if you don't need to build after every single update.

kunish · 2023-10-20T09:39:12Z

This build process is time consuming, which takes about one hour. You can disable on push after this run time if you don't need to build after every single update.

OK

piyoki · 2023-10-20T10:05:26Z

@hero-intelligent I will update the workflow definition accordingly. Stay tuned.

hero-intelligent requested review from a team as code owners September 21, 2023 16:30

dae-prow bot assigned hero-intelligent Sep 21, 2023

dae-prow bot added ci not-yet-tested labels Sep 21, 2023

hero-intelligent requested a review from a team as a code owner September 22, 2023 14:53

piyoki suggested changes Oct 19, 2023

View reviewed changes

publish.Dockerfile Outdated Show resolved Hide resolved

publish.Dockerfile Outdated Show resolved Hide resolved

publish.Dockerfile Outdated Show resolved Hide resolved

publish.Dockerfile Outdated Show resolved Hide resolved

publish.Dockerfile Outdated Show resolved Hide resolved

piyoki removed the not-yet-tested label Oct 19, 2023

hero-intelligent and others added 7 commits October 20, 2023 16:20

ci: docker image push with tag

a7dd193

doc(docker): Update getting-started.md

24d91f7

Signed-off-by: hero-intelligent <[email protected]>

ci(docker): add test tag

f70cc90

Signed-off-by: hero-intelligent <[email protected]>

rename stage in publish.Dockerfile

da9d8ad

hero-intelligent requested a review from piyoki October 20, 2023 08:20

Update publish.Dockerfile

82cb0c6

~~self-build~~ Co-authored-by: kev <[email protected]> Signed-off-by: hero-intelligent <[email protected]>

kunish requested changes Oct 20, 2023

View reviewed changes

docker workflow

5cbc43f

hero-intelligent requested a review from kunish October 20, 2023 08:54

kunish approved these changes Oct 20, 2023

View reviewed changes

kunish merged commit 79dd55a into daeuniverse:main Oct 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci(docker): rework workflow (3rd time) #289

ci(docker): rework workflow (3rd time) #289

hero-intelligent commented Sep 21, 2023 •

edited

Loading

hero-intelligent commented Sep 22, 2023 •

edited

Loading

hero-intelligent commented Oct 19, 2023

piyoki commented Oct 19, 2023

piyoki commented Oct 19, 2023

hero-intelligent commented Oct 20, 2023

kunish Oct 20, 2023

kunish Oct 20, 2023

hero-intelligent Oct 20, 2023

kunish Oct 20, 2023

hero-intelligent commented Oct 20, 2023

kunish commented Oct 20, 2023

hero-intelligent commented Oct 20, 2023

kunish commented Oct 20, 2023

piyoki commented Oct 20, 2023

ci(docker): rework workflow (3rd time) #289

ci(docker): rework workflow (3rd time) #289

Conversation

hero-intelligent commented Sep 21, 2023 • edited Loading

Background

Checklist

Full Changelogs

Issue Reference

Test Result

hero-intelligent commented Sep 22, 2023 • edited Loading

Achievements:

Limitations and explanation:

Goals not achieved:

hero-intelligent commented Oct 19, 2023

piyoki commented Oct 19, 2023

piyoki commented Oct 19, 2023

hero-intelligent commented Oct 20, 2023

kunish Oct 20, 2023

Choose a reason for hiding this comment

kunish Oct 20, 2023

Choose a reason for hiding this comment

hero-intelligent Oct 20, 2023

Choose a reason for hiding this comment

kunish Oct 20, 2023

Choose a reason for hiding this comment

hero-intelligent commented Oct 20, 2023

kunish commented Oct 20, 2023

hero-intelligent commented Oct 20, 2023

kunish commented Oct 20, 2023

piyoki commented Oct 20, 2023

hero-intelligent commented Sep 21, 2023 •

edited

Loading

hero-intelligent commented Sep 22, 2023 •

edited

Loading