Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the maintenance procedure #187

Merged
merged 9 commits into from
Oct 28, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ This project adheres to [Semantic Versioning](http://semver.org/).
We migrated the image repositories of meows to `ghcr.io`.
From meows v0.14.0, please use the following images.

- https://github.com/cybozu-go/meows/pkgs/container/meows-controller
- https://github.com/cybozu-go/meows/pkgs/container/meows-runner
- <https://github.com/cybozu-go/meows/pkgs/container/meows-controller>
- <https://github.com/cybozu-go/meows/pkgs/container/meows-runner>

The images on Quay.io ([meows-controller](https://quay.io/repository/cybozu/meows-controller), [meows-runner](https://quay.io/repository/cybozu/meows-runner)) will not be updated in the future.

Expand All @@ -41,7 +41,6 @@ The images on Quay.io ([meows-controller](https://quay.io/repository/cybozu/meow
- Support Kubernetes 1.27 ([#178](https://github.com/cybozu-go/meows/pull/1781))
- Build with go 1.21 ([#178](https://github.com/cybozu-go/meows/pull/178))


## [0.12.0] - 2023-07-05

### Changed
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ You can run jobs in your GitHub Actions workflows on your Kubernetes cluster wit
## Docker images

Docker images are available on [ghcr.io](https://github.com/orgs/cybozu-go/packages?repo_name=meows)

- [Controller](https://github.com/cybozu-go/meows/pkgs/container/meows-controller)
- [Runner](https://github.com/cybozu-go/meows/pkgs/container/meows-runner)

Expand Down
53 changes: 26 additions & 27 deletions RELEASE.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,12 @@
Release procedure
=================
# Release procedure

This document describes how to release a new version of meows.

Versioning
----------
## Versioning

Follow [semantic versioning 2.0.0][semver] to choose the new version number.

Prepare change log entries
--------------------------
## Prepare change log entries

Add notable changes since the last release to [CHANGELOG.md](CHANGELOG.md).
It should look like:
Expand All @@ -19,65 +16,67 @@ It should look like:
## [Unreleased]

### Added

- Implement ... (#35)

### Changed

- Fix a bug in ... (#33)

### Removed

- Deprecated `-option` is removed ... (#39)

(snip)
```

Bump version
------------
## Bump version

1. Determine a new version number. Then set `VERSION` variable.

```console
```bash
# Set VERSION and confirm it. It should not have "v" prefix.
$ VERSION=x.y.z
$ echo $VERSION
VERSION=x.y.z
echo $VERSION
```

2. Make a branch to release

```console
$ git neco dev "bump-$VERSION"
```bash
git switch -c "bump-$VERSION"
```

3. Edit `CHANGELOG.md` for the new version ([example][]).
4. Bump image version.

```console
$ sed -i -E "s/(.*newTag: ).*/\1${VERSION}/" config/controller/kustomization.yaml config/agent/kustomization.yaml
$ sed -i -E "s/(.*Version = ).*/\1\"${VERSION}\"/" constants.go
```bash
sed -i -E "s/(.*newTag: ).*/\1${VERSION}/" config/controller/kustomization.yaml config/agent/kustomization.yaml
sed -i -E "s/(.*Version = ).*/\1\"${VERSION}\"/" constants.go
```

5. Commit the change and push it.

```console
$ git commit -a -m "Bump version to $VERSION"
$ git neco review
```bash
git commit -a -m "Bump version to $VERSION"
git push origin "bump-$VERSION"
```

6. Merge this branch.
7. Add a git tag to the main HEAD, then push it.

```console
```bash
# Set VERSION again.
$ VERSION=x.y.z
$ echo $VERSION
VERSION=x.y.z
echo $VERSION

$ git checkout main
$ git pull
$ git tag -a -m "Release v$VERSION" "v$VERSION"
git checkout main
git pull
git tag -a -m "Release v$VERSION" "v$VERSION"

# Make sure the release tag exists.
$ git tag -ln | grep $VERSION
git tag -ln | grep $VERSION

$ git push origin "v$VERSION"
git push origin "v$VERSION"
```

GitHub actions will build and push artifacts such as container images and
Expand Down
16 changes: 5 additions & 11 deletions docs/commands.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,10 @@
CLI options
===========
# CLI options

`controller`
-----------
## `controller`

The CLI allows you to use the following options:

```bash
```console
$ controller -h
Kubernetes controller for GitHub Actions self-hosted runner

Expand Down Expand Up @@ -41,9 +39,7 @@ Flags:
--zap-stacktrace-level level Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').
```


`slack-agent`
-------------
## `slack-agent`

The Slack agent is a server program.
This notifies CI results and accepts requests for extending Pods' lifecycles
Expand All @@ -68,9 +64,7 @@ Flags:
-v, --verbose Verbose.
```


`meows`
------
## `meows`

This is a tool command to do some operations.
It enables to send requests to the slack-agent, or to control the GitHub runners.
Expand Down
10 changes: 5 additions & 5 deletions docs/design.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ Runner has the `status` and `busy` state as written [here](https://docs.github.c
If the `--ephemeral` option is given to `config.sh` does not repeat the
long polling again, and never gets `online` after the assigned job is done.
This behavior is useful for ensuring to make a clean environment for each job.
ref: https://docs.github.com/en/actions/hosting-your-own-runners/autoscaling-with-self-hosted-runners#using-ephemeral-runners-for-autoscaling
ref: <https://docs.github.com/en/actions/hosting-your-own-runners/autoscaling-with-self-hosted-runners#using-ephemeral-runners-for-autoscaling>

#### A job is scheduled only on a `online` runner

Expand Down Expand Up @@ -189,7 +189,7 @@ meows sets the namespaced name of a `RunnerPool` as a custom label.
command when the job is failed. The `if: failure()` syntax allows users
to run the step only when one of previous steps exit with non-zero code.
1. Publish the timestamp of when to delete this pod in the `/deletion_time` endpoint.
If the job is succeeded or canceled, the `Pod` publishes the current time for
If the job is succeeded or canceled, the `Pod` publishes the current time for
delete itself. If the job is failed, the `Pod` publishes the future time for
delete itself, for example 20 min later.
1. The Slack agent notifies the result of the job on a Slack channel.
Expand All @@ -208,15 +208,15 @@ A Runner `Pod` has the following state as a GitHub Actions job runner.
for example, booting a couple of VMs needed in a job before the job is assigned.
- `running`: `Pod` is running. Registered in GitHub Actions.
- `debugging`: The job has finished with failure and Users can enter `Pod` to debug.
- `stale`: The environment in the `Pod` is dirty. If a runner restarts before completing a job,
- `stale`: The environment in the `Pod` is dirty. If a runner restarts before completing a job,
the environment in the `Pod` may be dirty. This state means waiting for the Pod
to be removed to prevent Job execution with that stale Pod.

In addition, it has the following states as the exit state of the execution result of `Runner.Listener`.

- `retryable_error`: If execution fails due to a factor other than a job, restart `Runner.Listener`.
- `updating`: When a new `Runner.Listener` is released, it updates itself and restarts` Runner.Listener`.
- `undefined`: When the exit code of `Runner.Listener` is undefined. It restarts` Runner.Listener`.
- `updating`: When a new `Runner.Listener` is released, it updates itself and restarts `Runner.Listener`.
- `undefined`: When the exit code of `Runner.Listener` is undefined. It restarts `Runner.Listener`.

The above states are exposed from `/metrics` endpoint as Prometheus metrics. See [metrics.md](metrics.md).

Expand Down
47 changes: 23 additions & 24 deletions docs/development.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
Development guide
=================
# Development guide

Testing
-------
## Testing

There are 2 kinds of test included in this repository.

Expand All @@ -18,7 +16,7 @@ difficult to test in some parts and some parts of the code are not tested intent
What kindtest covers is:

- Runner `Pod`s are registered to GitHub Actions on a test repository.
- At present, the test repository is a fixed one (`github.com/neco-test/meows-ci`).
- At present, the test repository is a fixed one (`github.com/neco-test/meows-ci`).
- GitHub Actions workflows run on the `Pod`s.
- Runner `Pod`s send messages to Slack agent.
- Slack agent sends messages to Slack.
Expand All @@ -41,7 +39,8 @@ In order to run the kindtest, you need to prepare as follows.
You can run the kindtest as following.

1. Create secret files for kindtest.
```bash

```console
$ vi .secret.private-key.pem
# Save your GitHub App private key in this file.

Expand All @@ -56,40 +55,41 @@ You can run the kindtest as following.
```

2. Install tools.

```bash
$ make setup
make setup
```

3. Run kindtest.

```bash
# Start kind cluster.
$ make -C kindtest start
make -C kindtest start

# Run test on kind.
$ make -C kindtest test
make -C kindtest test

# Stop kind cluster.
$ make -C kindtest stop
make -C kindtest stop
```


### Run slack agent manually

Then, run a server with the following commands:

```bash
# Run server process
$ export SLACK_CHANNEL=#<your Slack Channel>
$ export SLACK_APP_TOKEN=<your Slack App Token>
$ export SLACK_BOT_TOKEN=<your Slack Bot Token>
$ go run ./cmd/slack-agent -d
export SLACK_CHANNEL=#<your Slack Channel>
export SLACK_APP_TOKEN=<your Slack App Token>
export SLACK_BOT_TOKEN=<your Slack Bot Token>
go run ./cmd/slack-agent -d
```

You can test both the failure and success messages by actually sending them:

```bash
# client
$ cat <<EOF > /tmp/github.env
cat <<EOF > /tmp/github.env
{
"actor": "user",
"git_ref": "branch-name",
Expand All @@ -103,27 +103,26 @@ $ cat <<EOF > /tmp/github.env
EOF

# success
$ go run ./cmd/meows slackagent send pod success -f /tmp/github.env
go run ./cmd/meows slackagent send pod success -f /tmp/github.env

# failure
$ go run ./cmd/meows slackagent send pod failure --extend -f /tmp/github.env
go run ./cmd/meows slackagent send pod failure --extend -f /tmp/github.env
```

Then, click the button on the Slack message, and check if a receiving log appears
on the terminal.

How to run meows for development
----------------------------------------------------
## How to run meows for development

If you need to run the controller on your local environment, this is the easiest way to do that.
You can reuse the token for the test repository, which is prepared for CI.
But please be careful that your local environment steals the job that is expected to run on a node created in CI and might cause a failure on CI.

```bash
# Create secret files for kindtest.
$ vi .secret.private-key.pem
$ vi .secret.env.sh
vi .secret.private-key.pem
vi .secret.env.sh

$ make -C kindtest start
$ make -C kindtest bootstrap
make -C kindtest start
make -C kindtest bootstrap
```
49 changes: 49 additions & 0 deletions docs/maintenance.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Maintenance

## How to update supported Kubernetes

Meows supports the three latest Kubernetes versions.
If a new Kubernetes version is released, please update the followings:

### 1. Update supported kubernetes and dependencies versions

- Kubernetes versions: You can check the versions at <https://hub.docker.com/r/kindest/node/tags>.
- `k8s-version` in [.github/workflows/main.yaml](/.github/workflows/main.yaml)
- "Supported software" in [README.md](/README.md)
- Tools versions:
- Update `CONTROLLER_GEN_VERSION` in [Makefile](/Makefile) to the latest version from <https://github.com/kubernetes-sigs/controller-tools/releases>.
- Update `RUNNER_VERSION` in [Dockerfile](/Dockerfile) to the latest version from <https://github.com/actions/runner/releases>.
- In [kindtest/Makefile](/kindtest/Makefile):
- Update `KINDTEST_K8S_VERSION` to the latest supported version of kubernetes.
- Update `KUSTOMIZE_VERSION` to the latest version from <https://github.com/kubernetes-sigs/kustomize/releases>.
- Update `KIND_VERSION` to the latest version from <https://github.com/kubernetes-sigs/kind/releases>.
- Update `CERT_MANAGER_VERSION` to the latest version from <https://github.com/cert-manager/cert-manager/releases>.
- After saving the changes above, update `ENVTEST_K8S_VERSION` in [Makefile](/Makefile) to the latest patch version among the latest supported kubernetes minor versions listed by running `make setup && tmp/bin/setup-envtest list` at the root of this repository. If the latest minor supported version is `1.30.Z`, find `1.30.Z+` from the output but not `1.31.Z`.
- Other dependencies versions:
- Update `ghcr.io/cybozu/golang` image in [Dockerfile](/Dockerfile) to the latest version from <https://github.com/cybozu/neco-containers/pkgs/container/golang>.
- `go.mod` and `go.sum`:
- Run `go get -u ./...`.

If Kubernetes or controller-runtime API has changed, please update the relevant source code accordingly.

### 2. Update meows by running `make`

You can update meows by running the following `make` commands:

```sh
make setup
make manifests
make build
```

### 3. Fix test code if tests fail

After pushing the change, if the CI fails, fix the tests and push the changes again.

_e.g._, <https://github.com/cybozu-go/meows/pull/185>

### 4. Release the new version

After merging the changes above, follow the procedures written in [Release.md](/RELEASE.md) and release the new version.

_e.g._, <https://github.com/cybozu-go/meows/pull/186>
3 changes: 1 addition & 2 deletions docs/metrics.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
Metrics
===========
# Metrics

## Controller

Expand Down
Loading
Loading