fix error when delete mgh then recreate it

[ldpliu]

Summary

we should delete kafka pvc when delete mgh, if not, when we recreate mgh, the kafka pod can not start

[root@ip-172-31-15-97 multicluster-global-hub]# oc get po
NAME                                                READY   STATUS             RESTARTS      AGE
kafka-kafka-0                                       0/1     CrashLoopBackOff   2 (9s ago)    41s
kafka-kafka-1                                       0/1     CrashLoopBackOff   2 (11s ago)   41s
kafka-kafka-2                                       0/1     CrashLoopBackOff   2 (9s ago)    40s
multicluster-global-hub-grafana-cc8b57d74-v6fh2     2/2     Running            0             89s
multicluster-global-hub-grafana-cc8b57d74-vvc9c     2/2     Running            0             89s
multicluster-global-hub-manager-698594c9-6tpcp      1/1     Running            2 (90s ago)   92s
multicluster-global-hub-manager-698594c9-jbvjv      1/1     Running            2 (86s ago)   92s
multicluster-global-hub-operator-776678446c-7gksz   1/1     Running            0             6m8s
multicluster-global-hub-postgresql-0                2/2     Running            0             92s
strimzi-cluster-operator-v0.43.0-76f57fb5b7-kgs4d   1/1     Running            0             80s

(org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig) [main] Exception in thread "main" java.lang.RuntimeException: Invalid cluster.id in: /var/lib/kafka/data-0/kafka-log0/meta.properties. Expected PPAVF8rBTuq25YVTs2LT_Q, but read NCsZMgUBSwuPQDozJaC8zg at org.apache.kafka.metadata.properties.MetaPropertiesEnsemble.verify(MetaPropertiesEnsemble.java:509) at kafka.tools.StorageTool$.formatCommand(StorageTool.scala:531) at kafka.tools.StorageTool$.runFormatCommand(StorageTool.scala:140) at kafka.tools.StorageTool$.execute(StorageTool.scala:80) at kafka.tools.StorageTool$.main(StorageTool.scala:53) at kafka.tools.StorageTool.main(StorageTool.scala)

Related issue(s)

https://issues.redhat.com/browse/ACM-16507
Fixes #

Tests

• Unit/function tests have been added and incorporated into make unit-tests.
• Integration tests have been added and incorporated into make integration-test.
• E2E tests have been added and incorporated into make e2e-test-all.
• List other manual tests you have done.

[clyang82]

/hold
as we discussed offline, please investigate the real root case instead of deleting the pv

[ldpliu]

It looks must delete data or cluster id in data when recreate kafka cluster.
Please see:
https://github.com/orgs/strimzi/discussions/10082
https://issues.apache.org/jira/browse/KAFKA-7335

[clyang82]

LGTM

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: clyang82, ldpliu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [clyang82]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[clyang82]

/unhold
I am OK to delete the pv due to:

1. we do not backup the kafka pv
2. there is no solution from strimzi community. the only workaround is set the clusterID into .status. that workaround is not fit for us.

[ldpliu]

/retest

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
81.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud