From d18c2ea8001a779edff79c17fa94840ad155c9c1 Mon Sep 17 00:00:00 2001 From: Michael Klishin Date: Tue, 14 May 2024 12:04:03 -0400 Subject: [PATCH 1/4] RabbitMQ: explain how to properly avoid a deployment deadlock Explain the problem and two widely used solutions instead of recommending force booting nodes. Signed-off-by: Michael Klishin --- bitnami/rabbitmq/README.md | 48 ++++++++++++++++---------------------- 1 file changed, 20 insertions(+), 28 deletions(-) diff --git a/bitnami/rabbitmq/README.md b/bitnami/rabbitmq/README.md index 0806a622a2e4bc..b6afdf294efd63 100644 --- a/bitnami/rabbitmq/README.md +++ b/bitnami/rabbitmq/README.md @@ -263,41 +263,33 @@ extraConfiguration: | log.console.formatter = json ``` -### Recover the cluster from complete shutdown +### Avoid Deadlocked Deployemnts After a Cluster-Wide Restart -> IMPORTANT: Some of these procedures can lead to data loss. Always make a backup beforehand. +RabbitMQ nodes assume their peers come back online within five minutes (by default). With the `OrderedReady` pod management policy is used +with a readiness probe that implicitly requires a fully booted node, the deployment can deadlock: -The RabbitMQ cluster is able to support multiple node failures but, in a situation in which all the nodes are brought down at the same time, the cluster might not be able to self-recover. + * Kubernetes will expect the first node to pass a readiness probe + * The readiness probe may require a fully booted node + * The node will fully boot after it detects that its peers have come online + * Kubernetes will not start any more pods until the first one boots -This happens if the pod management policy of the statefulset is not `Parallel` and the last pod to be running wasn't the first pod of the statefulset. If that happens, update the pod management policy to recover a healthy state: +Using [RabbitMQ Cluster Operator](https://www.rabbitmq.com/kubernetes/operator/operator-overview) is the easies solution. -```console -$ kubectl delete statefulset STATEFULSET_NAME --cascade=false -helm upgrade RELEASE_NAME oci://REGISTRY_NAME/REPOSITORY_NAME/rabbitmq \ - --set podManagementPolicy=Parallel \ - --set replicaCount=NUMBER_OF_REPLICAS \ - --set auth.password=PASSWORD \ - --set auth.erlangCookie=ERLANG_COOKIE -``` - -> Note: You need to substitute the placeholders `REGISTRY_NAME` and `REPOSITORY_NAME` with a reference to your Helm chart registry and repository. For example, in the case of Bitnami, you need to use `REGISTRY_NAME=registry-1.docker.io` and `REPOSITORY_NAME=bitnamicharts`. +Alternatively, the following combination of deployment settings avoids the problem: -For a faster resyncronization of the nodes, you can temporarily disable the readiness probe by setting `readinessProbe.enabled=false`. Bear in mind that the pods will be exposed before they are actually ready to process requests. + * Use `podManagementPolicy: "Parallel"` to boot multiple cluster nodes in parallel + * Use `rabbitmq-diagnostics ping` for readiness probe -If the steps above don't bring the cluster to a healthy state, it could be possible that none of the RabbitMQ nodes think they were the last node to be up during the shutdown. In those cases, you can force the boot of the nodes by specifying the `clustering.forceBoot=true` parameter (which will execute [`rabbitmqctl force_boot`](https://www.rabbitmq.com/rabbitmqctl.8.html#force_boot) in each pod): +Note that forcing nodes to boot is **not a solution** and doing so **can be dangerous**. Forced booting is a last resort mechanism +in RabbitMQ that helps make remaining clusters nodes to recover and rejoin each other after a permanent loss of some of their former +peers. In other words, forced booting a node is en emergency event recovery procedure. -```console -helm upgrade RELEASE_NAME oci://REGISTRY_NAME/REPOSITORY_NAME/rabbitmq \ - --set podManagementPolicy=Parallel \ - --set clustering.forceBoot=true \ - --set replicaCount=NUMBER_OF_REPLICAS \ - --set auth.password=PASSWORD \ - --set auth.erlangCookie=ERLANG_COOKIE -``` - -> Note: You need to substitute the placeholders `REGISTRY_NAME` and `REPOSITORY_NAME` with a reference to your Helm chart registry and repository. For example, in the case of Bitnami, you need to use `REGISTRY_NAME=registry-1.docker.io` and `REPOSITORY_NAME=bitnamicharts`. +To learn more, see -More information: [Clustering Guide: Restarting](https://www.rabbitmq.com/clustering.html#restarting). + * [RabbitMQ Clustering guide: Node Restarts](https://www.rabbitmq.com/docs/clustering#restarting) + * [RabbitMQ Clustering guide: Restarts and Readiness Probes](https://www.rabbitmq.com/docs/clustering#restarting-readiness-probes) + * [Recommendations](https://www.rabbitmq.com/docs/cluster-formation#peer-discovery-k8s) for [Operator](https://www.rabbitmq.com/kubernetes/operator/operator-overview)-less (DIY) deployments to Kubernetes + * [DIY RabbitMQ deployments on Kubernetes](https://www.rabbitmq.com/blog/2020/08/10/deploying-rabbitmq-to-kubernetes-whats-involved): What's Involved? ### Known issues @@ -876,4 +868,4 @@ Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and -limitations under the License. \ No newline at end of file +limitations under the License. From 08d8d7a1eddf050421bc47eb5555e8c4267ae229 Mon Sep 17 00:00:00 2001 From: Michael Klishin Date: Tue, 14 May 2024 12:32:09 -0400 Subject: [PATCH 2/4] Markdown linter changes, use an H2 header --- bitnami/rabbitmq/README.md | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/bitnami/rabbitmq/README.md b/bitnami/rabbitmq/README.md index b6afdf294efd63..448511420b0608 100644 --- a/bitnami/rabbitmq/README.md +++ b/bitnami/rabbitmq/README.md @@ -263,22 +263,22 @@ extraConfiguration: | log.console.formatter = json ``` -### Avoid Deadlocked Deployemnts After a Cluster-Wide Restart +## How to Avoid Deadlocked Deployemnts After a Cluster-Wide Restart RabbitMQ nodes assume their peers come back online within five minutes (by default). With the `OrderedReady` pod management policy is used with a readiness probe that implicitly requires a fully booted node, the deployment can deadlock: - * Kubernetes will expect the first node to pass a readiness probe - * The readiness probe may require a fully booted node - * The node will fully boot after it detects that its peers have come online - * Kubernetes will not start any more pods until the first one boots + - Kubernetes will expect the first node to pass a readiness probe + - The readiness probe may require a fully booted node + - The node will fully boot after it detects that its peers have come online + - Kubernetes will not start any more pods until the first one boots Using [RabbitMQ Cluster Operator](https://www.rabbitmq.com/kubernetes/operator/operator-overview) is the easies solution. Alternatively, the following combination of deployment settings avoids the problem: - * Use `podManagementPolicy: "Parallel"` to boot multiple cluster nodes in parallel - * Use `rabbitmq-diagnostics ping` for readiness probe + - Use `podManagementPolicy: "Parallel"` to boot multiple cluster nodes in parallel + - Use `rabbitmq-diagnostics ping` for readiness probe Note that forcing nodes to boot is **not a solution** and doing so **can be dangerous**. Forced booting is a last resort mechanism in RabbitMQ that helps make remaining clusters nodes to recover and rejoin each other after a permanent loss of some of their former @@ -286,15 +286,17 @@ peers. In other words, forced booting a node is en emergency event recovery proc To learn more, see - * [RabbitMQ Clustering guide: Node Restarts](https://www.rabbitmq.com/docs/clustering#restarting) - * [RabbitMQ Clustering guide: Restarts and Readiness Probes](https://www.rabbitmq.com/docs/clustering#restarting-readiness-probes) - * [Recommendations](https://www.rabbitmq.com/docs/cluster-formation#peer-discovery-k8s) for [Operator](https://www.rabbitmq.com/kubernetes/operator/operator-overview)-less (DIY) deployments to Kubernetes - * [DIY RabbitMQ deployments on Kubernetes](https://www.rabbitmq.com/blog/2020/08/10/deploying-rabbitmq-to-kubernetes-whats-involved): What's Involved? + - [RabbitMQ Clustering guide: Node Restarts](https://www.rabbitmq.com/docs/clustering#restarting) + - [RabbitMQ Clustering guide: Restarts and Readiness Probes](https://www.rabbitmq.com/docs/clustering#restarting-readiness-probes) + - [Recommendations](https://www.rabbitmq.com/docs/cluster-formation#peer-discovery-k8s) for [Operator](https://www.rabbitmq.com/kubernetes/operator/operator-overview)-less (DIY) deployments to Kubernetes + - [DIY RabbitMQ deployments on Kubernetes](https://www.rabbitmq.com/blog/2020/08/10/deploying-rabbitmq-to-kubernetes-whats-involved): What's Involved? -### Known issues + +## Known issues - Changing the password through RabbitMQ's UI can make the pod fail due to the default liveness probes. If you do so, remember to make the chart aware of the new password. Updating the default secret with the password you set through RabbitMQ's UI will automatically recreate the pods. If you are using your own secret, you may have to manually recreate the pods. + ## Persistence The [Bitnami RabbitMQ](https://github.com/bitnami/containers/tree/main/bitnami/rabbitmq) image stores the RabbitMQ data and configurations at the `/opt/bitnami/rabbitmq/var/lib/rabbitmq/` path of the container. From b65ccf036669544c3dede48dfb6126b561584115 Mon Sep 17 00:00:00 2001 From: Michael Klishin Date: Tue, 14 May 2024 12:33:25 -0400 Subject: [PATCH 3/4] Try to appease the Markdown linter --- bitnami/rabbitmq/README.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/bitnami/rabbitmq/README.md b/bitnami/rabbitmq/README.md index 448511420b0608..3b0c4a090cf745 100644 --- a/bitnami/rabbitmq/README.md +++ b/bitnami/rabbitmq/README.md @@ -286,17 +286,15 @@ peers. In other words, forced booting a node is en emergency event recovery proc To learn more, see - - [RabbitMQ Clustering guide: Node Restarts](https://www.rabbitmq.com/docs/clustering#restarting) - - [RabbitMQ Clustering guide: Restarts and Readiness Probes](https://www.rabbitmq.com/docs/clustering#restarting-readiness-probes) - - [Recommendations](https://www.rabbitmq.com/docs/cluster-formation#peer-discovery-k8s) for [Operator](https://www.rabbitmq.com/kubernetes/operator/operator-overview)-less (DIY) deployments to Kubernetes - - [DIY RabbitMQ deployments on Kubernetes](https://www.rabbitmq.com/blog/2020/08/10/deploying-rabbitmq-to-kubernetes-whats-involved): What's Involved? - +- [RabbitMQ Clustering guide: Node Restarts](https://www.rabbitmq.com/docs/clustering#restarting) +- [RabbitMQ Clustering guide: Restarts and Readiness Probes](https://www.rabbitmq.com/docs/clustering#restarting-readiness-probes) +- [Recommendations](https://www.rabbitmq.com/docs/cluster-formation#peer-discovery-k8s) for [Operator](https://www.rabbitmq.com/kubernetes/operator/operator-overview)-less (DIY) deployments to Kubernetes +- [DIY RabbitMQ deployments on Kubernetes](https://www.rabbitmq.com/blog/2020/08/10/deploying-rabbitmq-to-kubernetes-whats-involved): What's Involved? ## Known issues - Changing the password through RabbitMQ's UI can make the pod fail due to the default liveness probes. If you do so, remember to make the chart aware of the new password. Updating the default secret with the password you set through RabbitMQ's UI will automatically recreate the pods. If you are using your own secret, you may have to manually recreate the pods. - ## Persistence The [Bitnami RabbitMQ](https://github.com/bitnami/containers/tree/main/bitnami/rabbitmq) image stores the RabbitMQ data and configurations at the `/opt/bitnami/rabbitmq/var/lib/rabbitmq/` path of the container. From b0ab86dcbf9bab565e3a55f2a04a34b65115b02f Mon Sep 17 00:00:00 2001 From: Michael Klishin Date: Tue, 14 May 2024 12:35:34 -0400 Subject: [PATCH 4/4] Appease the Markdown linter --- bitnami/rabbitmq/README.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/bitnami/rabbitmq/README.md b/bitnami/rabbitmq/README.md index 3b0c4a090cf745..4d39f2fe5c2739 100644 --- a/bitnami/rabbitmq/README.md +++ b/bitnami/rabbitmq/README.md @@ -268,17 +268,17 @@ extraConfiguration: | RabbitMQ nodes assume their peers come back online within five minutes (by default). With the `OrderedReady` pod management policy is used with a readiness probe that implicitly requires a fully booted node, the deployment can deadlock: - - Kubernetes will expect the first node to pass a readiness probe - - The readiness probe may require a fully booted node - - The node will fully boot after it detects that its peers have come online - - Kubernetes will not start any more pods until the first one boots +- Kubernetes will expect the first node to pass a readiness probe +- The readiness probe may require a fully booted node +- The node will fully boot after it detects that its peers have come online +- Kubernetes will not start any more pods until the first one boots -Using [RabbitMQ Cluster Operator](https://www.rabbitmq.com/kubernetes/operator/operator-overview) is the easies solution. +Using [RabbitMQ Cluster Operator](https://www.rabbitmq.com/kubernetes/operator/operator-overview) is the easies solution. Alternatively, the following combination of deployment settings avoids the problem: - - Use `podManagementPolicy: "Parallel"` to boot multiple cluster nodes in parallel - - Use `rabbitmq-diagnostics ping` for readiness probe +- Use `podManagementPolicy: "Parallel"` to boot multiple cluster nodes in parallel +- Use `rabbitmq-diagnostics ping` for readiness probe Note that forcing nodes to boot is **not a solution** and doing so **can be dangerous**. Forced booting is a last resort mechanism in RabbitMQ that helps make remaining clusters nodes to recover and rejoin each other after a permanent loss of some of their former