support etcd cli via kine endpoint

[poblin-orange]

support etcd cli via kine endpoint

In order to support k3s backend change (eg: from mysql to postgres), we need to provide an etcd cli and cli configuration targetting the kine endpoint.

see:

• https://docs.k3s.io/advanced#using-etcdctl
• https://etcd.io/docs/v3.5/dev-guide/interacting_v3/ etcdctl: a command line tool for interacting with the etcd server
• https://gist.github.com/superseb/0c06164eef5a097c66e810fe91a9d408
• https://rancher-users.slack.com/archives/CGGQEHPPW/p1684616312714139

Support embedded etcd

In order to support k3s backend change (eg: from mysql to embedded etcd), I need to leverage an etcd cli and cli configuration targetting the k3s embedded etcd cluster.

ls -al /var/vcap/store/k3s-server/server/tls/etcd/
total 48
drwx------ 2 root root 4096 Oct  4 19:58 .
drwx------ 4 root root 4096 Oct  4 19:58 ..
-rw-r--r-- 1 root root 1140 Oct  4 19:58 client.crt
-rw------- 1 root root  227 Oct  4 19:58 client.key
-rw------- 1 root root  566 Oct  4 19:53 peer-ca.crt
-rw------- 1 root root  227 Oct  4 19:53 peer-ca.key
-rw-r--r-- 1 root root 1352 Oct  4 19:58 peer-server-client.crt
-rw------- 1 root root  227 Oct  4 19:58 peer-server-client.key
-rw------- 1 root root  570 Oct  4 19:53 server-ca.crt
-rw------- 1 root root  227 Oct  4 19:53 server-ca.key
-rw-r--r-- 1 root root 1364 Oct  4 19:58 server-client.crt
-rw------- 1 root root  227 Oct  4 19:58 server-client.key

Workaround for missing etcdctl cli on the bosh release

Inspired from https://gist.github.com/superseb/0c06164eef5a097c66e810fe91a9d408 k3s etcd commands

# Lookup current etcd version
grep 'starting etcd server' /var/vcap/sys/log/k3s-server/*.log
#> /var/vcap/sys/log/k3s-server/k3s-server.stderr.log:{"level":"info","ts":"2024-08-01T10:43:47.664738Z","caller":"etcdserver/server.go:858","msg":"starting etcd server","local-member-id":"356f4aa799dfa8d0","local-server-version":"3.5.13","cluster-id":"c6e4720454a4c070","cluster-version":"3.5"}

#From local-server-version
ETCD_VER=v3.5.13

ETCD_VER=${ETCD_VER:-v3.5.13}
# Download etcdctl

https_proxy=http://system-internet-http-proxy.internal.paas:3128
HTTPS_PROXY=http://system-internet-http-proxy.internal.paas:3128
HTTP_PROXY=http://system-internet-http-proxy.internal.paas:3128
http_proxy=http://system-internet-http-proxy.internal.paas:3128
https_proxy=http://system-internet-http-proxy.internal.paas:3128                                                                                                                                               export https_proxy                                                                                
export http_proxy                                                                                 


GITHUB_URL=https://github.com/etcd-io/etcd/releases/download
DOWNLOAD_URL=${GITHUB_URL}

rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
rm -rf /tmp/etcd-download-test && mkdir -p /tmp/etcd-download-test

curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /usr/local/bin --strip-components=1
rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz

etcd --version
etcdctl version

unset https_proxy
unset http_proxy
unset HTTP_PROXY
unset HTTPS_PROXY

# Use etcdctl with bosh specific paths 
# Display cluster status
ETCDCTL_ENDPOINTS='https://127.0.0.1:2379/' ETCDCTL_CACERT='/var/vcap/store/k3s-server/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/vcap/store/k3s-server/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/vcap/store/k3s-server/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint status --cluster --write-out=table

export ETCDCTL_ENDPOINTS='https://127.0.0.1:2379/'
export ETCDCTL_CACERT='/var/vcap/store/k3s-server/server/tls/etcd/server-ca.crt' 
export ETCDCTL_CERT='/var/vcap/store/k3s-server/server/tls/etcd/server-client.crt' 
export ETCDCTL_KEY='/var/vcap/store/k3s-server/server/tls/etcd/server-client.key' 
export ETCDCTL_API=3 

etcdctl --dial-timeout=20s --command-timeout=50s get /registry --prefix --keys-only  | grep -v ^$ | head
#> /registry/acme.cert-manager.io/challenges/10-cert-manager-wildcards-gen/ferretdb-wildcard-public-tls-1-2792303281-2725165740
#> ...

Display nb of keys, from https://etcd.io/blog/2023/how_to_debug_large_db_size_issue/
etcdctl --dial-timeout=20s --command-timeout=50s get /registry --prefix --keys-only | grep -v ^$ | awk -F '/' '{ h[$3]++ } END {for (k in h) print h[k], k}' | sort -nr