Operator CrashLoopBackOff due to Liveness probe error

[Javatar81]

Describe the bug

In the events we see the liveness probe fail:

Liveness probe error: Get "http://10.128.2.97:8081/healthz": dial tcp 10.128.2.97:8081: connect: connection refused body:

This is the operator logs:

1
I1012 15:04:24.582908       1 request.go:682] Waited for 1.032624601s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/submariner.io/v1?timeout=32s
2
2023-10-12T15:04:33Z	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": "127.0.0.1:8080"}
3
I1012 15:04:34.915972       1 request.go:682] Waited for 1.03513177s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/cluster.open-cluster-management.io/v1alpha1?timeout=32s
4
2023-10-12T15:04:44Z	INFO	controllers.Helm	Watching resource	{"group": "charts.janus-idp.io", "version": "v1alpha1", "kind": "Backstage"}
5
2023-10-12T15:04:44Z	INFO	setup	configured watch	{"gvk": "charts.janus-idp.io/v1alpha1, Kind=Backstage", "chartPath": "helm-backstage/charts/backstage", "maxConcurrentReconciles": 10, "reconcilePeriod": "1m0s"}
6
2023-10-12T15:04:44Z	INFO	setup	starting manager
7
2023-10-12T15:04:44Z	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"}
8
2023-10-12T15:04:44Z	INFO	Starting server	{"kind": "health probe", "addr": "[::]:8081"}
9
I1012 15:04:44.171868       1 leaderelection.go:248] attempting to acquire leader lease openshift-operators/rhdh-operator...
10
I1012 15:05:01.818264       1 leaderelection.go:258] successfully acquired lease openshift-operators/rhdh-operator
11
2023-10-12T15:05:01Z	DEBUG	events	rhdh-operator-7b9b9d8cb8-9m428_4eaec52b-e57a-43c4-a4fb-f59e78b20421 became leader	{"type": "Normal", "object": {"kind":"Lease","namespace":"openshift-operators","name":"rhdh-operator","uid":"fe6d8060-900d-41fc-a23b-0988b039bf51","apiVersion":"coordination.k8s.io/v1","resourceVersion":"4475768571"}, "reason": "LeaderElection"}
12
2023-10-12T15:05:01Z	INFO	Starting EventSource	{"controller": "backstage-controller", "source": "kind source: *unstructured.Unstructured"}
13
2023-10-12T15:05:01Z	INFO	Starting EventSource	{"controller": "backstage-controller", "source": "kind source: *v1.Secret"}
14
2023-10-12T15:05:01Z	INFO	Starting Controller	{"controller": "backstage-controller"}

Expected Behavior

Pod starts and runs without liveness checks failing.

What are the steps to reproduce this bug?

1. Create Subscription for operator
2. Wait for operator pod running
3. CrashLoopBackOff after several restarts due to liveness probe

Versions of software used and environment

• OCP 4.13.14
• rhdhoperator.v1.0.0