.github: auto-merge PRs of Kernel and ca-certificates

[dongsupark]

This PR is a proof-of-concept of auto-merging PRs created by GitHub Actions.

It is to address maintenance issues with PRs in coreos-overlay, mostly Kernel and ca-certificates. Github Actions automatically create 5 new PRs for new Kernel versions, 1 for 5.10 and 4 for 5.15. We do not merge those immediately until the moment of building Releases. When new PRs are created again in-between, we manually close old Kernel PRs and consider recent versions as the next merge candidates, without merging the PRs. That is already error-prone. On top of that, in case of Stable, we pin to a specific version like 5.15.92 for the next Stable release. In most cases, it is not clear which should be the next Stable Kernel, so we have to manually comment "this is the next Kernel version" in an auto-generated PR. I would say, the whole workflow is already a far from how bots are supposed to work, as it is not automatic at all.

Instead we auto-merge all PRs of Kernel and ca-certificates, except for the corner case of Stable Kernel.

Repository configuration

First of all, we need to configure in the settings of coreos-overlay repo.

Go to Settings >> General.

• Enable Allow auto-merge

Go to Settings >> Code and automation >> Branches.

• On Branch protection rules, do Add rule.

  • In the Branch name pattern, type in main.
    • Go to Protect matching branches, and then
      • Enable Require a pull request before merging.
      • Enable Dismiss stale pull request approvals when new commits are pushed.
      • Enable Require status checks to pass before merging, and then
        • Require branches to be up to date before merging
        • In the search field of Search for status checks in the last week for this repository, type in ci-check.
          • Select ci-check, a no-op check.
          • The name should match with name of the no-op workflow to be created under .github/workflow in coreos-overlay.
    • In the bottom of the page, click on Create.
  • Do the same thing again for maintenance branches.
    • In this case, the branch name pattern should be flatcar-3[0-9][0-9][0-9], which should match all branches.

Generate a PAT (Personal Access Token) that should be used only for auto-merging PRs. Expiration date should be fairly long, e.g. 1 year.

Go to Settings >> Secrets and variables >> Actions.

• Select New repository secrets to create a secret.
  • Name has be AUTOMERGE_TOKEN, and paste the token generated in there.

Workflow

Each auto-merge workflow basically makes use of peter-evans/enable-pull-request-automerge. Although other options are out there, we use this solution mainly for simplicity.

We should use a new token AUTOMERGE_TOKEN, generated for auto-merge workflow in coreos-overlay, because that is the standard approach recommended by GitHub.

We have to introduce a no-op workflow ci-check, simply runs only /bin/true. because at least one status check is required by auto-merge functionality of GitHub Actions. We need to backport the new workflow file to all maintenance branches, LTS, Stable, Beta, Alpha. Otherwise required status checks of each PR could not finish at all, so auto-merge could not run.

Kernel

For ordinary channels, e.g. LTS, Beta, Alpha, main, it is pretty straightforwards. Simply enable auto-merge right away, then the PRs would be merged immediately.

For Stable, however, we cannot do so, because of a certain delay of Kernel versions is supposed to take place. For example, 5.15.92 should be the next Stable Kernel, while the next Beta Kernel would be 5.15.98. So the idea is to detect if the Kernel version of the new PR is in the range of [current stable, next stable]. If yes, create a PR and turn on auto-merge. If not, do not create a PR.

TODO: we should probably split Kernel into separate workflows, not to make all channels run at once in parallel. Ideally we should also keep the PRs open for a few hours, using labels like do-not-merge. However, it is not trivial for workflows of the current code base to run in multiple steps like that. At the moment I do not want to make the code complicated.

Detection of the next Stable Kernel version could unexpectedly fail in the middle of the release process.

ca-certificates

It works just like Kernel, but this part does not have a special case for Stable. Simply auto-merge.

[pothos]

Can we use the build secret token from github-actions-overlay.txt.gpg (add the permissions) instead of creating a new one to manually manage? (This one expires 2023-03-17 which is tomorrow, by the way! I think we should create calendar events for expiry.)

Does Require a pull request before merging conflict with the release tagging?

[pothos]

We need a way to stop the auto merging because it could lead to breaking all builds all the time. For that I think it would be good enough to have a delay of ~30 minutes before the auto merge, and then a check whether a 'on-hold' label is present on the PR.

[t-lo]

We could require the PRs to pass a GH actions CI build / test; automation for scripts should be ready in a few days, and we could add automation for coreos-overlay right after that.

[dongsupark]

We need a way to stop the auto merging because it could lead to breaking all builds all the time. For that I think it would be good enough to have a delay of ~30 minutes before the auto merge, and then a check whether a 'on-hold' label is present on the PR.

Exactly, that's what I would like to achieve. For example, create 2 different workflows, merge-autolabel-kernel, which reacts on PR events to actually merge the PR, and autolabel-kernel which periodically gives a label 'automerge` to every open PR, daily ~4pm. In theory that sounds like a good idea. In practice, however it was not so trivial for the latter workflow to get actually a PR number, after create-pull-request all finished. That's where I gave up. Let me do other experiments about that.

[pothos]

same here, we need a way to stop this from happening if we can't update for whatever reason

[dongsupark]

Can we use the build secret token from github-actions-overlay.txt.gpg (add the permissions) instead of creating a new one to manually manage? (This one expires 2023-03-17 which is tomorrow, by the way! I think we should create calendar events for expiry.)

As discussed in the chat, the token in that file is actually FLATCAR_PORTAGE_STABLE_ACCESS_TOKEN, which is not used any more since we moved Rust to coreos-overlay. Let's remove that, and add a new token.

Does Require a pull request before merging conflict with the release tagging?

Short answer: no.

Long: First, as long as we keep using the peter-evans/enable-pull-request-automerge, we do not need Require a pull request before merging. This was my misunderstanding. I removed the requirement from the description.

Even if we enable Require a pull request before merging, it is still possible for Flatcar buildbot to directly push to branches, because another option Do not allow bypassing the above settings is not enabled by default.