Cruise Control is not recognizing the need to rebalance

[foet-cw]

Hi Strimzi,
I'm trying to use Cruise Control, and am only having partial success.
I've setup a Kafka cluster using Strimzi, have created automated producers and consumers, putting a significant amount of messages/data into the cluster.
I've been able to use the examples here: https://github.com/strimzi/strimzi-kafka-operator/tree/main/examples/cruise-control for a few different operations.
Specifically, I've been able to remove brokers, and add brokers without issue.

• The broker pods have lots of resources available (cpu/mem/disk) and we're running 14 brokers in this test cluster currently in KRaft mode.

Errors I fixed

I'm using k edit k to modify my running config, and solved 2 errors:

• Initially I was getting this error:

"Error processing POST request '/rebalance' due to: 'com.linkedin.kafka.cruisecontrol.exception.OptimizationFailureException: [NetworkInboundCapacityGoal] Insufficient capacity for networkInbound (Utilization 3658436.50, Allowed Capacity 112000.00, Threshold: 0.80). Add at least 444 brokers with the same networkInbound capacity (10000.00) as broker-0. Add at least 444 brokers with the same networkInbound capacity (10000.00) as broker-0.'.".
So I overrode and raised the NetworkInboundCapacity goal, and that fixed that error.

• Next I ran into this error in my proposals:

Provision Recommendation: [RackAwareGoal] Remove at least 2 rack with brokers.
Provision Status: OVER_PROVISIONED

So I used: overprovisioned.min.extra.racks: 4 which cleared that up.

The issue

However, when I try to balance a cluster, all I'm getting back is proposals that indicate everything is already 100% balanced.
Like this (santized):

k describe kafkarebalance rebalance-test-6 -n kafka-test
Name:         rebalance-test-6
Namespace:    kafka-test
Labels:       strimzi.io/cluster=kafka-test-cluster
Annotations:  <none>
API Version:  kafka.strimzi.io/v1beta2
Kind:         KafkaRebalance
Metadata:
  Creation Timestamp:  2024-10-28T20:58:21Z
  Generation:          1
  Resource Version:    696152389
  UID:                 <foo>
Spec:
  Goals:
    DiskCapacityGoal
    RackAwareGoal
    ReplicaCapacityGoal
  Skip Hard Goal Check:  true
Status:
  Conditions:
    Last Transition Time:  2024-10-28T20:58:21.632562428Z
    Status:                True
    Type:                  ProposalReady
  Observed Generation:     1
  Optimization Result:
    After Before Load Config Map:  rebalance-test-6
    Data To Move MB:               0
    Excluded Brokers For Leadership:
    Excluded Brokers For Replica Move:
    Excluded Topics:
    Intra Broker Data To Move MB:         0
    Monitored Partitions Percentage:      100
    Num Intra Broker Replica Movements:   0
    Num Leader Movements:                 0
    Num Replica Movements:                0
    On Demand Balancedness Score After:   100
    On Demand Balancedness Score Before:  100
    Provision Recommendation:             
    Provision Status:                     RIGHT_SIZED
    Recent Windows:                       1
  Session Id:                             <foo>
Events:                                   <none>

I should also note that when I removed 3 brokers in the cluster, the disk did NOT get balanced among the remaining brokers, instead it ended up with several using much more disk than others.

Imbalanced Resources

I currently have 3 brokers with around 100MB of disk used (the ones that were removed/re-added), while I have 2 brokers over 1TB of disk used, and several more in the 500-700GB range.
Knowing that disk alone may not be enough for it to rebalance, the CPU usage is also pretty skewed to certain brokers. The container_cpu_usage_seconds_total is as high as 2-3 per certain brokers, while near zero for some (like the ones I evicted and re-added. Memory usage is also in a similar state, the 3 nodes only having used about 6-7GB of mem, while other nodes in the cluster are using >100GB of mem per broker.
I'm sharing this to try to paint the picture that the broker pods are significantly imbalanced and I believe KCC should be doing something when I generate a proposal.

Here's what I've tried:

• the equivalent of kafka-rebalance-full.yaml where I don't specify a spec so it just uses the generated proposal that happens every 15 min.
• the equivalent of kafka-rebalance-with-goals.yaml, where I started with the 7 goals in the example, and kept paring it back till it was just the 3 required hard goals. No difference in the generated proposal between the various iterations.
• I've tried changing some of the default analyzer thresholds (such as disk.balance.threshold) to a more sensitive setting like 1.05, but was denied the ability to change that via k edit k because it said it was an invalid change.
• After I removed 3 brokers, then re-added them, they have not gotten any partitions. With some brokers hosting hundreds of partitions (measuring in the TB range), and the 3 I added back in hosting zero partitions/data, it feels like there's a setting or default that is preventing the cluster from balancing.
• I tried the skipHardGoalCheck: true to no effect (though I did see that status change to true in the proposal)

I'm trying to figure out 2 things:

• Besides the 2 fixes I made above, all other settings are out of the box for KCC, and I'm thinking I need to set and/or override something in order to get it to balance.
• When doing k edit k what is the syntax for adjusting things like analyzer thresholds or goals? I haven't been able to find an example in the documentation, apologies if I've missed it.
  Thank you for your time and help!

EDIT - Sorry about the messed up formatting in the proposal, I tried a few different ways and the formatting keeps getting destroyed.

[scholzj]

For a start you should probably:

• Make sure the things you share are properly formatted and readable and ideally share them in the YAML format (with -o yaml) to make them reproducible
• Share more details for all the step. E.g. what exactly you changed in the different steps, how did your KafkaRebalance resources used used looked like etc.

[foet-cw]

OK, fixed the formatting. The code block button only did ` instead of ``` and I didn't notice that. I haven't posted on GH before, so please pardon my naivety. Thank you for the reply!

As far as the things I've tried, each attempt (after fixing the errors I mentioned) was more or less as follows.
I tried as simple as:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: rebalance-test-6
  labels:
    strimzi.io/cluster: kafka-test-cluster
spec: {}

Which follows the model here: https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/cruise-control/kafka-rebalance-full.yaml

Then added goals one by one, until I ended up at this:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: rebalance-test-6
  labels:
    strimzi.io/cluster: kafka-test-cluster
spec:
  goals:
    - CpuCapacityGoal
    - NetworkInboundCapacityGoal
    - DiskCapacityGoal
    - RackAwareGoal
    - MinTopicLeadersPerBrokerGoal
    - NetworkOutboundCapacityGoal
    - ReplicaCapacityGoal

Which follows the example here: https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/cruise-control/kafka-rebalance-with-goals.yaml

The output each time is like what I shared above, with everything indicating no changes to make (excerpt):

    Data To Move MB:               0
    ...
    Intra Broker Data To Move MB:         0
    Monitored Partitions Percentage:      100
    Num Intra Broker Replica Movements:   0
    Num Leader Movements:                 0
    Num Replica Movements:                0
    On Demand Balancedness Score After:   100
    On Demand Balancedness Score Before:  100

I did try one with just the hard goals set as well, but no change:

spec:
  goals:
    - DiskCapacityGoal
    - RackAwareGoal
    - ReplicaCapacityGoal

I've tried generating a number of different proposals (over a dozen). As I mentioned, evicting 3 brokers did drain the topics off them correctly, however it didn't balance the disk or resource utilization among the remaining brokers, so now the cluster (to my eye) is fairly imbalanced.

[scholzj]

@kyguy or @ppatierno might be able to provide more details.

But in general, the capacity goals are checking that things are within the capacity. They do not necessarily care that one disk is empty while other has some data as long as they are withing the capacity limits. If you want to distribute things equally, you should use the distribution goals - e.g. DiskUsageDistributionGoal. That will try to distribute things equally.

---

"Error processing POST request '/rebalance' due to: 'com.linkedin.kafka.cruisecontrol.exception.OptimizationFailureException: [NetworkInboundCapacityGoal] Insufficient capacity for networkInbound (Utilization 3658436.50, Allowed Capacity 112000.00, Threshold: 0.80). Add at least 444 brokers with the same networkInbound capacity (10000.00) as broker-0. Add at least 444 brokers with the same networkInbound capacity (10000.00) as broker-0.'.".
So I overrode and raised the NetworkInboundCapacity goal, and that fixed that error.

I guess for this you can also configure the expected network capacity. If you configured resources for your Kafka cluster, Strimzi might know how much CPU the pods have etc. Or it might know how big the disks are. But the networking capacity is something we can't get on our own. So you can not use the goal, or you can configure it to what you expect your Pods to be able to handle as the default value does not seem to correspond to your cluster.

---

Provision Recommendation: [RackAwareGoal] Remove at least 2 rack with brokers.
Provision Status: OVER_PROVISIONED

Glad that you managed to work around it. But I guess this is a possible misconfiguration of your Kafka cluster / its topics? How many racks do you have? How many replicas do your topics have?

[foet-cw]

Hi Scholzj,
Thanks for the response, I'll check out the distribution goals. I didn't realize the point you made about it won't balance if things are within limits. That definitely seems to be what's happening. Thank you, I really appreciate the tip and the perspective!

Re: the network capacity error - Yeah, that's what I did to fix that one. The out of the box defaults were comparatively low throughput per broker, and I just needed to up that. Working on finding the right number, but with our testing we're pushing some pretty sizeable volume which is what drove the "444 more brokers needed" message.

Re: the rack over provisioning - Yeah, this is a side effect of our current test env. We have RF=3 set at the cluster level for all Topics, and we have 7 racks in our test cluster. The default setting of overprovisioned.min.extra.racks: 2 wasn't quite enough, and I incrementally raised it till that alert cleared. I wasn't clear if that would have prevented the cluster from balancing so I figured I'd rule that out.

I'll experiment with the distribution goals, and I'm optimistic that will resolve my issue. I'll report back.

Thank you for the time in responding, and the detailed suggestions!

[kyguy]

I guess for this you can also configure the expected network capacity. If you configured resources for your Kafka cluster, Strimzi might know how much CPU the pods have etc. Or it might know how big the disks are. But the networking capacity is something we can't get on our own. So you can not use the goal, or you can configure it to what you expect your Pods to be able to handle as the default value does not seem to correspond to your cluster.

Agree with Jakub, the first place to look when having balancing issues like this surrounding compute resources is the capacity configurations in the Kafka custom resource. The disk capacity per broker should be automatically detected and set without need of user intervention. However, the CPU and network throughput capacity settings need special attention by the user. It sounds like the issues surrounding network throughput capacity have been worked out. For sorting out issues surround CPU capacity check out the docs here: [1]

Disk and CPU capacity limits are automatically generated by Strimzi, so you do not need to set them. In order to guarantee accurate rebalance proposals when using CPU goals, you can set CPU requests equal to CPU limits in Kafka.spec.kafka.resources. That way, all CPU resources are reserved upfront and are always available. This configuration allows Cruise Control to properly evaluate the CPU utilization when preparing the rebalance proposals based on CPU goals. In cases where you cannot set CPU requests equal to CPU limits in Kafka.spec.kafka.resources, you can set the CPU capacity manually for the same accuracy.

If there are proposal generation issues surrounding resources after setting the resource capacities in the Kafka custom resource feel free to post the configurations here and we can take a closer look!

[1] https://strimzi.io/docs/operators/in-development/full/configuring.html#property-cruise-control-broker-capacity-reference

[foet-cw]

Thanks Kyle (Fellow Bostonian wave!),
Yeah, when reading the documentation the other day I saw that cpu and disk should be auto-detected (which is nice). I'll take a look at the CPU side and make sure we're setup correctly. Our test cluster is pretty well spec'd so I have some room to play with there.

Jakub, just wanted to followup here, your suggestion was the solution.
In all my permutations/tests over the past few days, I'd played with lots of default goals, but hadn't touched the distribution goals. 🤦

Not only is the cluster balancing resources evenly across pods right now, but it's moving quickly, load looks good, and it's much more balanced even at the halfway point now. I have a followup item to play with the other distribution settings and find the right balance for me, as well as Kyle's suggestions about network and CPU. Big thanks for calling that out!

I also wanted to say a special thanks to Jakub for the info, and the patience with me. As they say on radio "longtime listener, first time caller", I hadn't ever posted in GitHub discussions before, and the formatting didn't work quite as I'd expected. I appreciate your calm response, nudging me towards the right format, as well as not just dismissing my wall of info when it wasn't formatted. I come with significant Kafka background, but am relatively new to using Strimzi so it's a lot of cool new things to learn. Thank you very much for all the help, getting past this hurdle has made my day! 😄 I appreciate the help, and hope you have a nice week!