Strimzi can't roll Cruise Control

[teimyBr]

Describe the bug
We running Strimzi Operator and a Kafka Cluster. Sometimes the Strimzi Operator can't roll Cruise Control.

We have Certificates which have 10 days Lifetime. Thats the reason why Strimzi roll the Kafka and all other Componetes.

Cruise Control Error:

Readiness probe failed: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 9090: Connection refused

Strimzi Error:

2022-05-29 14:49:08 ERROR Util:149 - Reconciliation #13163(timer) Kafka(master-backbone/kafka): Exceeded timeout of 300000ms while waiting for Deployment resource kafka-cruise-control in namespace master-backbone to be ready
2022-05-29 14:49:08 ERROR AbstractOperator:247 - Reconciliation #13163(timer) Kafka(master-backbone/kafka): createOrUpdate failed
io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for Deployment resource kafka-cruise-control in namespace master-backbone to be ready
	at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:150) ~[io.strimzi.operator-common-0.28.0.jar:0.28.0]
	at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:69) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) [io.netty.netty-transport-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2022-05-29 14:49:08 WARN  AbstractOperator:532 - Reconciliation #13163(timer) Kafka(master-backbone/kafka): Failed to reconcile
io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for Deployment resource kafka-cruise-control in namespace master-backbone to be ready
	at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:150) ~[io.strimzi.operator-common-0.28.0.jar:0.28.0]
	at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:69) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) [io.netty.netty-transport-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at java.lang.Thread.run(Thread.java:829) [?:?]

Expected behavior
Strimzi can roll Cruise Control

Environment (please complete the following information):

• Strimzi version: 0.28.0
• Installation method: Helm chart
• Kubernetes cluster: 1.20
• Infrastructure: On premise

[scholzj]

Can you please use proper formating to make it readable?

[scholzj]

Thanks. The log snippet from the Cluster Operator does not say much. It just notes that the Cruise Control deployment / pod did not become ready. That said, the renewal happens over multiple rolling updates. So I think you need to share the full Cluster OPerator log to understand what was happening when.

I'm not sure where exactly did you got the Cruise Control log. But I think you will need to share the full log from the Cruise Control pod to understand why is it not getting ready.

[teimyBr]

INFO ClusterOperator:123 - Triggering periodic reconciliation for namespace *0

INFO AbstractOperator:226 - Reconciliation #13148(timer) Kafka(master-backbone/kafka): Kafka kafka will be checked for creation or modification

INFO AbstractOperator:373 - Reconciliation #13148(timer) Kafka(master-backbone/kafka): Reconciliation is in progress

ERROR Util:149 - Reconciliation #13148(timer) Kafka(master-backbone/kafka): Exceeded timeout of 300000ms while waiting for Deployment resource kafka-cruise-control in namespace master-backbone to be ready

Reconciliation #13148(timer) Kafka(master-backbone/kafka): createOrUpdate failed

and then the long error messages.

[scholzj]

Yeah, but this does not seem to be any reconciliation which is renewing the certificates. This looks only like some follow-up reconciliation from when things were already broken. What is needed is the log from the renewal (ideally at a DEBUG level, but even without it it might be helpful)

[rajarshp]

@teimyBr @scholzj any solution on this issue or how did you fix it ? in one of the cluster when I am trying install kafka using RedHat Streams operator it is giving me the same error.

[teimyBr]

was fixed by a newer version, in the newer version we dont face the issue any more

@teimyBr @scholzj any solution on this issue or how did you fix it ? in one of the cluster when I am trying install kafka using RedHat Streams operator it is giving me the same error.