AWS Runner

This runner provisions instances in various clouds for both mac, windows and Linux. It also sets up the lite-engine and installs git. The installation of Docker on the instances allows the running of the build in Hybrid mode: where Drone Plugins can run or build steps in container along with build steps on the instance operating system. Pools of hot swappable EC2 instances are created on startup of the runner to improve build spin up time.

Installation

For more information about installing this runner look at the installation documentation.

Configuration

For more information about configuring this runner look at the configuration documentation.

Creating a build pipelines

For more information about creating a build pipeline look at the pipeline documentation.

Design

This runner was initially designed in the following proposal.

Release procedure

MAKE SURE THE BUILD STATUS IS GREEN BEFORE YOU RELEASE A NEW VERSION

Build and test the mac binary version

1. Build the mac binary version of the runner.

   CGO_ENABLED=1 go build -o drone-runner-aws-darwin-amd64

2. Run the exec command to test the runner. The pool name must be macpool.

   drone-runner-aws-darwin-amd64 exec test_files/compiler/drone_mac.yml --pool pool.yml --debug --trace --repo-http='https://github.com/tphoney/bash_plugin' --repo-branch='main' --commit-target='main' --commit-after='7e5f437589cdf071769158ce219b2f443ca13074'

Run the changelog generator

docker run -it --rm -v "$(pwd)":/usr/local/src/your-app githubchangeloggenerator/github-changelog-generator -u drone-runners -p drone-runner-aws -t <secret github token>

You can generate a token by logging into your GitHub account and going to Settings -> Personal access tokens.

Next we tag the PR's with the fixes or enhancements labels. If the PR does not fufil the requirements, do not add a label.

Run the changelog generator again with the future version according to semver.

docker run -it --rm -v "$(pwd)":/usr/local/src/your-app githubchangeloggenerator/github-changelog-generator -u drone-runners -p drone-runner-aws -t <secret token> --future-release v1.0.0

Create your pull request for the release. Get it merged then tag the release.

Testing against a custom lite engine

• build the lite-engine
• host the lite-engine binary python3 -m http.server
• run ngrok to expose the webserver ngrok http 8000
• add the ngrok url to the env file DRONE_LITE_ENGINE_PATH=https://c6bf-80-7-0-64.ngrok.io

Testing the delegate command

• Run the delegate command, wait for the pool creation to complete.
• setup an instance:

curl -d '{"id": "unique-stage-id","correlation_id":"abc1","pool_id":"ubuntu", "setup_request": {"network": {"id":"drone"}, "platform": { "os":"ubuntu" }}}' -H "Content-Type: application/json" -X POST  http://127.0.0.1:3000/setup

• run a step on the instance:

curl -d '{"stage_runtime_id":"unique-stage-id","pool_id":"ubuntu","instance_id":"<INSTANCE ID>","correlation_id":"xyz2", "start_step_request":{"id":"step4", "image": "alpine:3.11", "working_dir":"/tmp", "run":{"commands":["sleep 30"], "entrypoint":["sh", "-c"]}}}' -H "Content-Type: application/json" -X POST  http://127.0.0.1:3000/step

or, directly by IP address returned by the setup API call:

curl -d '{"ip_address":"<IP OF INSTANCE>","pool_id":"ubuntu","correlation_id":"xyz2", "start_step_request":{"id":"step4", "image": "alpine:3.11", "working_dir":"/tmp", "run":{"commands":["sleep 30"], "entrypoint":["sh", "-c"]}}}' -H "Content-Type: application/json" -X POST  http://127.0.0.1:3000/step

• destroy an instance:

curl -d '{"stage_runtime_id":"unique-stage-id","instance_id":"<INSTANCE ID>","pool_id":"ubuntu","correlation_id":"uvw3"}' -H "Content-Type: application/json" -X POST  http://127.0.0.1:3000/destroy

Testing the runner in delegate-less mode

The AWS runner can also connect to the Harness platform where it functions as both a task receiver and executor. In the delegate mode, the task receiving is done by the java delegate process. In the delegate-less mode, the task receiving is done by the same runner process.

The delegate-less mode does not start an API server, instead it polls the manager periodically to receive any tasks generated by the Harness platform. Hence, it is best to test this mode directly with Harness.

Steps:

• Create a .env file with variables needed to connect with the Harness platform.

DLITE_ACCOUNT_ID=<account-id>
DLITE_ACCOUNT_SECRET=<account-secret>
DLITE_MANAGER_ENDPOINT=<manager-endpoint>
DLITE_NAME=<name-of-runner>

The values of the above variables can be found by logging into a Harness account and adding a delegate. The spec for the delegate would contain the account ID, the secret (DELEGATE_TOKEN), and the endpoint (MANAGER_HOST_AND_PORT)

• Run the runner in delegate-less mode go run main.go dlite --pool=pool.yml

If the logs say the runner has been registered successfully, you should be able to see the runner in the delegates screen on Harness.

• Create a pipeline and execute it. Since this is in beta at the moment, a UI does not exist on Harness for it. To be able to leverage this runner, remove the infrastructure part in the pipeline and add a field runsOn: <pool-name> at the same level as execution: (directly under spec)

• You should see logs in the runner corresponding to the created tasks.