Recovery problem

[clickersmudge]

Hi,
I created a test cluster

cluster_autoscaler_max_replicas = 1
cluster_autoscaler_min_replicas = 0
cluster_name = "dev"
control_plane_type = "cpx11"
control_plane_vm_count = 1
datacenter = "nbg1"
initial_machinedeployment_replicas = 5
ip_range = "10.0.0.0/16"
lb_type = "lb11"
network_zone = "eu-central"
os = "ubuntu"
ssh_public_key_file = "~/.ssh/dev.pub"
worker_type = "cx21"

After that i immediately wanted to test the recovery scenario. (https://docs.kubermatic.com/kubeone/v1.7/guides/manual-cluster-recovery/)

export AWS_ACCESS_KEY_ID="<AWS_ACCESS_KEY_ID>"
export AWS_SECRET_ACCESS_KEY="<AWS_SECRET_ACCESS_KEY>"
export RESTIC_PASSWORD="<RESTIC_PASSWORD>"
export RESTIC_REPOSITORY="<RESTIC_REPOSITORY>" # s3:<ENDPOINT>/<BUCKET>
restic cache --cleanup
restic snapshots
restic restore <ID> --target .
terraform taint 'hcloud_server.control_plane[0]'
terraform taint 'hcloud_server.control_plane[1]'
terraform taint 'hcloud_server.control_plane[2]'
terraform apply
terraform output -json > tf.json
kubeone apply --credentials credentials.yaml --manifest kubeone.yaml --tfjson tf.json --no-init
rsync --archive --verbose ./backup <USER>@<HOST>:~/ # Control plane 1
ssh <USER>@<HOST>
  sudo rsync --archive --verbose $HOME/backup/pki/etcd /etc/kubernetes/pki/
  sudo rsync --archive --verbose $HOME/backup/pki/kubernetes/ /etc/kubernetes/pki
  sudo chown --recursive root:root /etc/kubernetes
  sudo ctr image pull registry.k8s.io/etcd:3.5.9-0
  sudo ctr run --rm \
    --mount type=bind,src=$HOME/backup,dst=/backup,options=rbind:ro \
    --mount type=bind,src=/var/lib,dst=/var/lib,options=rbind:rw \
    --env ETCDCTL_API=3 \
    registry.k8s.io/etcd:3.5.9-0 \
    sh etcdctl \
    snapshot restore \
    --data-dir=/var/lib/etcd \
    --name=<CONTROL_PLANE_NAME> \
    --initial-advertise-peer-urls=https://<PRIVATE_IP>:2380 \
    --initial-cluster=<CONTROL_PLANE_NAME>=https://<PRIVATE_IP>:2380 \
    /backup/etcd-snapshot.db
kubeone apply --credentials credentials.yaml --manifest kubeone.yaml --tfjson tf.json

At first glance, everything works properly.
Unfortunately, the apply command doesn't quite work.

WARN[23:35:49 CET] Retrying task...                             
INFO[23:35:49 CET] Waiting for machine-controller to come up... 

In the second terminal, I check the pods.

I checked the command

kubeone apply --credentials credentials.yaml --manifest kubeone.yaml --tfjson tf.json --force-upgrade

I can't diagnose the problem. Please help.

[kron4eg]

terraform taint 'hcloud_server.control_plane[0]'
terraform taint 'hcloud_server.control_plane[1]'
terraform taint 'hcloud_server.control_plane[2]'
terraform apply

That one destroyed your cluster basically. With 3 control plane nodes you can loose only 1, not all 3 at once.

[clickersmudge]

@kron4eg If I had one control plane and one MachineDeployment then backups cannot be restored?

[clickersmudge]

@kron4eg Hi,
Problem resolved.
Can you add to the documentation.
https://docs.kubermatic.com/kubeone/v1.7/guides/manual-cluster-recovery/
At the very end.
In a separate terminal when there is a problem with "Waiting for machine-controller to come up... "

export KUBECONFIG=$PWD/<KUBECONFIG>
kubectl get node
kubectl label node <NAME> node-role.kubernetes.io/control-plane=

[kron4eg]

What version of kubeone / kubernetes you've been running?

[clickersmudge]

1.7.2

[kron4eg]

@clickersmudge and which kubernetes version? Can you please post the kubeone config you had.

[clickersmudge]

@kron4eg Hi, Sorry for the late reply. KubeOne 1.7.2, Kubernetes v1.27.9. I no longer have KubeConfig for this environment.

[kron4eg]

OK, I've created the bug issue: #3017. Further progress will be there.