Splunk Operator: permit disabling splunkd TLS

[paheath]

Please select the type of request

Bug

Tell us more

Describe the request
It appears that the operator does not function properly when TLS on the Splunk API (8089) is disabled. The operator does not appear to throw obvious errors, but the cluster manager fails to install and come ready. The indexer cluster is stuck waiting, and never appears to instantiate its stateful set. I suspect this is because the operator can not perform the API calls required to complete the cluster manager installation/configuration. The operator should function in an environment with TLS on splunkd disabled.

Expected behavior
The operator should identify when it's running in an environment with TLS disabled. It should modify the protocol it uses to make Splunk API calls accordingly. It should also pass the URL of the cluster manager and license manager to connect the cluster with the correct protocol.

Splunk setup on K8S
cluster manager, indexer cluster, search head, standalone heavy forwarder, license manager, monitoring console. Using the App Framework and Smartstore.

Reproduction/Testing steps
Set splunk.ssl.enable to false in the defaults.yml file on every Splunk CR per the using default settings documentation. Watch for successful cluster manager and indexer cluster pod creation.

K8s environment
On-prem cluster running Istio service mesh.

Additional context(optional)
The intent here is to disable TLS at the application level so Istio can perform in-cluster mTLS encryption that is invisible to Splunk.

[k4r1]

I also came across this, exactly the same behaviour with no stateful set for the indexer cluster.

I found that I could at least get the startupProbe / readinessProbe for the cluster manager to pass by adding an env var to the CR:

  extraEnv:
  - name: SPLUNKD_SSL_ENABLE
    value: "false"

However, the operator logs still complain about not being able to reach the various components, I assume it's using https with no way to change it - will probably need a code change.

[vivekr-splunk]

Hello @paheath we will test this scenario and get back to you .

[paheath]

Thank you! I've done a fair amount of testing for this, and have gotten Splunk to function normally with strict pod-to-pod mTLS enabled by deploying docker-splunk directly without the operator. It's possible that the only thing required to get this to work is:

• remove the istio annotations applied by the operator that force istio to ignore all Splunk ports
• create a flag that forces the operator to disable TLS for all splunk api calls made by the operator
• create a flag that forces the operator to deploy the workload pods with SPLUNKD_SSL_ENABLE = false

One other quirk I noticed, the cli command (and I assume the equivalent API call) that connects indexer peers to the cluster manager does not accept a manager URI with plain http. The command usage is documented here, but I don't see a mention of this behavior: https://docs.splunk.com/Documentation/Splunk/latest/Indexer/ConfigurepeerswithCLI. I was able to work around this in my istio mTLS testing by configuring the indexer -> manager connection directly with config files applied through the default.yaml mechanism supported by splunk-ansible.

Let me know if I can provide any more information that might be helpful for your testing.