Secret Management: Support rotation of the admin password

[gjanders]

Please select the type of request

Enhancement

Tell us more

Describe the request
The goal is the ability to change the password of the admin user.
Currently a secret such as:
splunk-test-cluster-secret

Will have a "password", this is the admin password to the cluster manager and every indexer in that namespace

What I would like is the ability to provide a new password, or even have the operator rotate the password itself (a new password that I can specify would be preferable), let's assume I can add a:
newpassword: mynewpassword

Into the secret, and once found by the operator it should be able to use the old password to update each cluster manager & indexer to use the new password specified, then patch the required secret so future cluster bundle apply commands or any related commands use the new password.

Expected behavior
As per description, I want the ability to rotate the splunk instances admin passwords. I would prefer to be able to specify a password to be used for rotation as this will comply with the organisations security requirements
However, even having the ability to reset the password to one chosen by the SOK would be an improvement

Splunk setup on K8S
In my example I'm mentioning a indexercluster and clustermanager CR, this would also equally apply to standalone and searchheadcluster CR's.

Reproduction/Testing steps
N/A

K8s environment
K8s cluster

Proposed changes(optional)
As per description, provide a method where I can provide a new password and this password should be updated on all required Splunk instances involved.

Additional context(optional)
Feel free to clarify on github or community slack

[yaroslav-nakonechnikov]

have you tried to update password and delete pod?
it should take last secret and start splunk with new password.

[gjanders]

As in it will force the new password into the pod? That might work, but it would be a terrible way to update the secret

What I would do is just splunk edit user on the command and update the secret. Since the secret is mounted and not an environment variable I'd expect it would likely just work.

I could potentially create a CronJob in K8s level that does this on a regular basis and that might work well for a standalone.

However, I have indexer clusters and search head clusters as the main parts of the environment. Furthermore, I cannot just "stop" the cluster manager/indexers each time I want to rotate a password.

Changing the password on the CM + each indexer would likely work, but there would be some edge cases to take into account:

• What if the indexers REST API is down? As per the Splunk docs I could run the splunk cmd splunkd rest --noauth POST command if I can access the indexer in question
• What if the password change fails? The above process would again likely work
• What if an indexer manages to startup but has an older copy of the password ? (As it was rotated while the pod was down). In this scenario we'd have to force reset the password if the admin user already exists

I believe these challenges can be solved, but I'd like to have the Splunk operator do the work here

[yaroslav-nakonechnikov]

yes, i feel your pain.

and i bet there will be no solution till container is based on installing full splunk package in container by ansible inside.

[akondur]

Hey @gjanders , the Splunk secret tokens apply across all Splunk deployments across the K8s namespace. Please refer the documentation to edit the password token in the Splunk Global namespace scoped secret which will apply to all the deployments in the namespace. The operator should take care of the password change.

Ideally you could have a K8s cron job to edit this secret. Note: You will have to plan for all the pods being rebooted in this scenario.

Are you specifically looking for changing CM/Indexers vs rest of Splunk deployments?

[akondur]

Hey @gjanders , please let us know so we can help better with the issue. We are evaluating support for vault for password rotation right now.

[gjanders]

@akondur apologies about the delayed response. The documentation you have provided does work as expected, with the termination and re-creation of the pods with a new password.

Is there a way to avoid a restart of the pod to achieve this?
I could perhaps create a crontab that runs the CLI for splunk password changes:
splunk edit user admin -auth admin:<admin_password> -password newpassword
And then update the secret in K8s if the password change succeeds, however, I'd like to have the ability to rotate the password without restarting the pods.

Restarts cause various issues, in indexer clusters it can result in problems for the search tier related to bundle pushes and "generation" changes.

[akondur]

Hey @gjanders , currently we do not have a way of rotating passwords without pods being recycled. We are currently evaluating alternate options to improve the process of password rotation. We will keep the documentation updated on the same.

Marking this issue as an enhancement request to be looked at.