Splunk Operator: Can we avoid multiple restarts of Splunk when the pod restarts?

[gjanders]

Please select the type of request

Bug

Tell us more

Describe the request

When an existing Splunk indexer pod starts, the container starts as expected and the Splunk prints the "My GUID" message as expected.

However, the ansible playbook in Splunk always triggers a 2nd restart, in some cases I've seen 3 restarts. I find this consistently in my environment including the test environment.

Note I've also confirmed the same behaviour occurs on Splunk search heads, so I assume it's a generic issue with the ansible re-setting a setting and then deciding a restart is "required". I've done a difference of the configuration during these restarts and I usually find that encrypted pass4SymmKeys/other values were re-encrypted to a new value but have the same decrypted value (i.e. it was a pointless change)

Expected behavior

When a pod restarts I expect Splunk to startup, and not to restart when no changes are required. The ansible playbook from Splunk triggers the restart for no obvious reason.

Splunk setup on K8S

Splunk indexer cluster, 4 nodes, multi-site (2 nodes per site) in development.

Reproduction/Testing steps

Build indexer cluster, once all nodes are online restart 1 pod.
Observe the pod, and check how many times "My GUID" appears in the splunkd log.

K8s environment

K8s 1.28, same issue was on K8s 1.24.

Proposed changes(optional)

Update the splunk ansible playbooks embedded inside the container to reduce restarts to only when a "real" change has occurred and not on every time the pod comes online (when the pod has previously existed and has persistent storage)

Additional context(optional)

I have at least one support case which is likely related to this issue as the cluster/bucket level issues start only after the indexer is restarted shortly after coming online.

[vivekr-splunk]

Hello @gjanders , thank you for raising this issue. we are actively working with ansible team to address this issue. we will update you with our findings.

[gjanders]

Thanks for the update