Smartstore/AppFramework: Azure AD Workload Identity Not Working

[dpericaxon]

Please select the type of request

Bug

Tell us more

Describe the request

• Currently we're using Azure AAD Pod Identity that works fine with the Splunk Operator and its components. It can read/write to storage buckets for storing apps and indexes in Azure blobs. Azure AAD pod identity has been deprecated so the the new solution is Azure-Workload-Identity. In an effort to move to Azure Workload Identity the indexer pods kepted failing because it was trying to reach the remote storage but it couldn't. Even after deleting the PVC's it still eventually started but then saw errors like "Failed to trigger replication (err='Cannot replicate remote storage enabled warm bucket,".

I believe the cause of this is that currently Splunk Operator is using Azure Instance Metadata Service (IMDS) which works with Azure AAD but not Azure Workload Identity since its point is to "Removes the need for Custom Resource Definitions and pods that intercept Instance Metadata Service (IMDS) traffic".

It looks like they have packages libraries available in GO: https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview?tabs=go#azure-identity-client-libraries

Expected behavior

• The Splunk Operator/Components should be able to work with Azure Workload Identity.

Splunk setup on K8S

• HA Splunk Operator/Components configuration with multiple indexers and searchheads.

Reproduction/Testing steps

• Follow most of this quickstart but apply it to the splunk components: https://azure.github.io/azure-workload-identity/docs/quick-start.html

K8s environment

• Azure/AKS

Proposed changes(optional)

• We should have an ability to leverage Workload Identity especiaily since Azure AAD Pod Identity is deprecated.

[dpericaxon]

Hey @vivekr-splunk @sgontla @gaurav-splunk do you know if this is something on the roadmap or could be fixed soon?

[vivekr-splunk]

@dpericaxon we will investigate and get back to soon