Skip to content

Latest commit

 

History

History
79 lines (61 loc) · 4.31 KB

RKE-EC2-ASGs.adoc

File metadata and controls

79 lines (61 loc) · 4.31 KB
Raw

General, semi-random, notes

  • Older doc that focuses on Ubuntu and Rancher self-CA certs: https://rancher.com/docs/rancher/v2.5/en/cluster-admin/cluster-autoscaler/amazon/

  • Note that in the script below, --ca-checksum is ommitted when Rancher server is using a public TLS cert

  • The checksum is not available from the API server. You have to pull it down and get the SHA256 checksum manually.

  • EC2 Master nodes need three (non-obvious) things to work correctly:

    1. The proper Role/Profile (see the URL above)

      • Note that the URL above splits what are actually policies into two Roles/Profiles. The two policies actually go into a single Role/Profile for the Master nodes

    2. The proper tags, especially kubernetes.io/cluster/<clusterID>: owned

    3. The user-data script (though can be entered after node creation)

  • ASG-Worker nodes need the same things, except:

    • The Role/Profile is different (far fewer permissions)

    • More tags, which are specified in the ASG

      • Since tags contain the cluster name and clusterID, each ASG is cluster specific

    • The user-data script is required, which is specified in the Launch Configuration

      • Since the user-data script is cluster specific, each Launch Configuration is cluster specific

  • The URL above sets Masters as control-plane and etcd, which means if the ASG-Workers are scaled to zero, the cluster will go into a failed state, due to the lack of coreDNS

    • I prefer to add worker to the Masters, and use taints/tolerations to keep workloads off of them

Updated user-data script for SLES, works for Masters and ASG-Workers:

#!/bin/bash -x

## User-data script for Master and ASG-Worker nodes

#### User modifiable variables
RKE_VERSION="v2.6.2"
RKE_CLUSTER_TOKEN="jmmpttxk8lmnfjbgqq64lvspbvn6bm4jn4mbhfh868fn69zlm85sjb"
RANCHER_SERVER_URL="https://rancher-demo.susealliances.com"

## Set the RANCHER_NON_PUBLIC_CA_CHECKSUM variable to an empty value (RANCHER_NON_PUBLIC_CA_CHECKSUM="") if Rancher server is using a TLS certificate issued by a publicly registered CA, i.e. Let's Encrypt
RANCHER_NON_PUBLIC_CA_CHECKSUM=""

## If using a TLS certificate issued by ANY private CA:
## 1. Pull down the private CA certificate from Rancher server (https://<Rancher URL>/v3/settings/cacerts) and get the SHA256 hash of it
## 1.a. Don't forget to replace \n with new lines, if needed: echo "-----BEGIN CERTIFICATE-----\nMII...vuqg==\n-----END CERTIFICATE-----\n" | sed 's/\\n/\n/g' | sha256sum | awk '{print$1}'
## 2. Uncomment the RANCHER_NON_PUBLIC_CA_CHECKSUM variable and populate the checksum
#RANCHER_NON_PUBLIC_CA_CHECKSUM="--ca-checksum <RANCHER_CA_CHECKCSUM>"

## Set the appropriate role for node
K8S_ROLES="--etcd --controlplane --worker"
#K8S_ROLES="--worker"

cat <<EOF > /etc/sysctl.d/90-kubelet.conf
vm.overcommit_memory = 1
vm.panic_on_oom = 0
kernel.panic = 10
kernel.panic_on_oops = 1
kernel.keys.root_maxkeys = 1000000
kernel.keys.root_maxbytes = 25000000
EOF
sysctl -p /etc/sysctl.d/90-kubelet.conf

systemctl enable docker
systemctl start docker
sleep 10

TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
PRIVATE_IP=$(curl -H "X-aws-ec2-metadata-token: ${TOKEN}" -s http://169.254.169.254/latest/meta-data/local-ipv4)
PUBLIC_IP=$(curl -H "X-aws-ec2-metadata-token: ${TOKEN}" -s http://169.254.169.254/latest/meta-data/public-ipv4)

sudo docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run --name rancher-agent rancher/rancher-agent:${RKE_VERSION} --server ${RANCHER_SERVER_URL} --token ${RKE_CLUSTER_TOKEN} ${RANCHER_NON_PUBLIC_CA_CHECKSUM} --address ${PUBLIC_IP} --internal-address ${PRIVATE_IP} ${K8S_ROLES}

To get the hash of a private CA cert through the API access:

Note
 if the "Value:" at https://<Rancher URL>/v3/settings/cacerts is empty, Rancher is using a public cert.

  • If needed, create a bearer token through the GUI to access the API server

  • Get the ca-cert:

curl -s -H "Authorization: Bearer $APITOKEN" https://127.0.0.1/v3/settings/cacerts --insecure | jq -r .value