Pods in kube-system are constantly restarting

[Minhp18]

We're now having a HA cluster with 3 master nodes and 6 worker nodes. cloud-controller-manager, kube-controller-manager, kube-scheduler, rke2-snapshot-controller, nfs-subdir-external-provisioner-79b9b6c6d9-rcbtp pods are constantly restarting with all the same error log:

1 leaderelection.go:332] error retrieving resource lock kube-system/kube-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

1 leaderelection.go:332] error retrieving resource lock kube-system/rke2-cloud-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-cloud-controller-manager?timeout=5s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

[brandond]

Things can't talk to the apiserver. Check the apiserver and etcd pod logs? Kube-proxy too perhaps.