Mount host cluster service account to pod

[tsipinakis]

Is your feature request related to a problem?

I am trying to authenticate vcluster workloads to external services that use the kubernetes OIDC method and trust the host clusters service accounts, however even if I enable serviceAccount syncing to the host the token that is mounted in the pod is using the vcluster internal accounts and not the hosts one.

Which solution do you suggest?

I am not aware of any way currently to mount a token from the host cluster in the vclusters pods, (is there one?) If not it'd be nice to have a configuration option for that.

I tried to use

    volumeMounts:
    - mountPath: /var/run/secrets/tokens
      name: service-token
  volumes:
  - name: service-token
    projected:
      sources:
      - serviceAccountToken:
          path: token 

instead of the automounted token but it looks like vcluster is overriding that as well.

Which alternative solutions exist?

No response

Additional context

For this only need a single service account is needed to be mountable, so a filter to only allow one would be nice.

[janekbaraniewski]

Hey @tsipinakis ! Could you provide full vcluster.yaml configuration you used for installation? Also if you could please describe in a bit more detail your particular use-case - where are your workloads trying to authenticate, what environment you're running your host cluster in etc.

[tsipinakis]

Hi @janekbaraniewski

We deploy vcluster using helm directly on an on-premise host Kubernetes cluster, this is the configuration:

defaultImageRegistry: "registry.example.com"

syncer:
  extraArgs:
    - --tls-san=example.example.com
    - --out-kube-config-server=https://example.example.com
sync:
  ingresses:
    enabled: true
  serviceaccounts:
    enabled: true

ingress:
  enabled: true
  pathType: ImplementationSpecific
  ingressClassName: nginx
  host: example.example.com
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/ssl-passthrough: 'true'
    nginx.ingress.kubernetes.io/ssl-redirect: 'true'
    cert-manager.io/cluster-issuer: letsencrypt

We are trying to authenticate to Vault using the Kubernetes and/or JWT auth method. We have configured the host cluster as the authentication provider and we want to use the namespace field as the user identifier to identify the vcluster that is being authenticated. Then we want to give the token to an ArgoCD instance that'll be running in the vcluster to deploy our software stack. However the show-stopper is that I cannot see how to mount a token from the host cluster inside the vcluster.

[janekbaraniewski]

@tsipinakis in vCluster v0.20 we do have an option to sync resources from host cluster to vCluster instance, did you try this approach? This requires multi-namespace mode to be enabled:

experimental:
  multiNamespaceMode:
    enabled: true

  genericSync:
    import:
      - kind: ServiceAccount
        apiVersion: v1
      - kind: Secret
        apiVersion: v1

This will import ServiceAccounts and Secrets created in host cluster in namespaces that are part of given vCluster deployment into vCluster. You can specify selectors for each kind to import only specific resources. More info on this can be found here in docs

[tsipinakis]

Reading the docs I'm not sure that this would work, this would sync the service accounts from the host cluster into the vcluster, but still the ArgoCD pod will have a service account token generated by vcluster, not one generated by the host cluster. What I'm looking for is a way to mount a service account of the host cluster in a pod running in vcluster. aka, the token should be signed by the host clusters signing key.

[janekbaraniewski]

@tsipinakis in that case solution would be to instead of using volume projection for mounting the Secret, to create a service-account-token type Secret in your host cluster and sync it to vCluster instance. This way token in this secret will be signed by your host cluster and can be used by Pods running inside your virtual cluster. Otherwise, like you said, it will be vCluster doing the signing.

[tsipinakis]

This requires multi-namespace mode to be enabled:

Looking at multi-namespace mode currently,

Loft Labs discourages using this feature due to weaker isolation and more management overhead.

Seems like quite a big breaking feature to use for such a simple use-case. Additionally, having multiple different namespaces would break the ability to authenticate to Vault with the method mentioned above i.e. using the namespace name as the username