[Bug]: Kafka/ZooKeeper cannot schedule due to pod antiaffinity

[faurik]

Bug Description

Sometimes (not every time) Kafka or ZooKeeper pods cannot schedule on the cluster.

This happens because of combination of two reasons:

1. Kafka/ZooKeeper definition has podAntiAffinity rules to avoid scheduling on the same node
2. PersistentVolumes are local (LVM/ZFS/etc) and remote mounts are forbidden
3. PVs are not provisioned by controller.

Elaboration:
When StatefulSet is created and kube-controller-manager is going to provision pods/PVCs, it firstly gets scheduling information (kube-scheduler is responsible for this), and result of scheduling depends on requests, affinity rules, etc. When kube-controller-manager gets that info, it is provisioning PVCs first, and it puts annotation volume.kubernetes.io/selected-node with value of selected node name into each created PVC. After that, CSI controller uses that info to provision PVC on the desired node. After, kube-controller creates pods, which are scheduled across the cluster to the provisioned PVCs. It leads to one important point: during provisioning of PVCs kube-controller-manager respects all the given affinity rules.

On the other hand, for some reason strimzi-kafka-operator takes care of creating PVCs instead of kube-controller manager. Therefore, there is no annotation with node name required by CSI controller and PVCs are provisioned randomly, which can lead to several PVs created on the same physical node. Which, in turn, lead to broken scheduling in case of podAntiAffinity.

Steps to reproduce

1. Ensure test environment has CSI driver which provisions volumes on the k8s nodes (for instance, we are using linstor-csi)
2. Create Kafka resource with several replicas of Kafka/ZooKeeper and podAntiAffinity rules
3. Check PVCs annotations (it has no annotation volume.kubernetes.io/selected-node)
4. Check distribution of Persistent Volumes across the cluster.

As I said above, issue is not reproduced each time.

Expected behavior

PVCs don't break scheduling process.

Strimzi version

0.20.0

Kubernetes version

OKD 4.7

Installation method

Helm Chart

Infrastructure

Bare-metal

Configuration files and logs

No response

Additional context

We have quite old version of strimzi-kafka-operator. but I fluently checked the source, and seems like this logic (PVC is created by operator before STS) is still there.

[faurik]

I am not expecting of fix for our very old installation, I guess we will fix it on our job for upgrading OKD is in progress for a while, but probably community would like to fix this in the new versions :)

[scholzj]

This does not seem like a bug. First of all, you are using a very old Strimzi and Kubernetes version. The latest versions work very differently. They for example don't use any StatefulSets. Also, when the PVC is bound is controlled by your Storage class and its volumeBindingMode -> if you set it to WaitForFirstConsumer it will be binding the PVC only when the Pod is created.

[scholzj]

Triaged on the community call on 2.5.2024: Should be converted to discussion as it seems more like a question.