Longhorn read-only filesystem & Strimzi not able to recover alone

[w3blogfr]

Hello,

We are using Longhorn & Strimzi operator in our cluster.

Sometime, due to an IO issue, longhorn volume become read-only, this is a safety mechanism of Longhorn.
Most of the time, this issue doesn't have a huge consequence. Longhorn have a feature named "Automatically Delete Workload Pod when The Volume Is Detached Unexpectedly", the pod is deleted, and recreate automatically thanks to a top deployment/statefullset layer. Longhorn is not configured to automatically delete pod standalone created by an custom operator. (see longhorn/longhorn#8353)

In previous strimzi version, it was not a problem, Strimzi was creating a statefullset layer to manager pods, it's not the case anymore, Strimzi operator now manage pods directly (probably for a good reason).

Modify liveness or readiness will only make the pod restart, and restart doesn't trigger a detachment/attachment from a kubernetes point of view.

Do you have any idea about how to workaround that problem?
I supposed we need to add a check inside operator to check if the volume is read-only or not, but I have no idea about how to do that.
I'm also open to any kind of other ideas.

Thanks

[scholzj]

I'm not sure we want/can add custom workarounds for various storage implementations out there. We do not have the resources to maintain that, but also don't have any resources to test things like that to make sure they still work etc. Maybe Longhorn can add support for some annotation or something on the Pod that you can use to tell it that it should delete even the od managed by a custom controller? You can set such annotation through the Kafka CR for example to control it.

[w3blogfr]

I know strimzi will recreate the pod if it has been deleted, but is it ok to have the pod killed by something else than strimzi-operator by design?

Regards.

[scholzj]

I guess it depends on how you look at it:

• Depending on the state of the other pods, deleting the pod might cause availability issues. However, that was the same with StatefulSets in our case, so this would not differ from how it worked before.
• If the storage is read-only, it will likely very quickly get out-of-sync anyway and impact the availability. So not sure the above really matters in this situation.

So, ideally, you should not delete the Pods on your own like this. But in a scenario like this, I do not think it really matters.