[bitnami/mongodb] Simplify and fix externalAccess configuration

[fmulero]

Description of the change

Allow the use of external certificates signed for external names. Current chart requires several internal names and IPs signed in the certificates provided by the user. That restriction makes almost impossible the use of this chart with external names and TLS enabled.

Benefits

Allow the use of external certificates and names in MongoDB chart.

Possible drawbacks

None

Applicable issues

• fixes [bitnami/mongodb] Enabling TLS using a custom certificate runs into multiple issues. - user creation / current_primary detection / all probes #16719

Additional information

These changes were tested in the following scenario:

1. Empty cluster created with k3d:

$ k3d cluster create

2. Certificates and secrets have been created with following script:

#!/bin/bash
if [[ ! -f  ca.crt ]]; then
  openssl genrsa -out ca.key 2048
  openssl req -x509 -new -nodes -key ca.key -subj "/CN=example.com" -sha256 -days 1024 -out ca.crt
fi

for i in 0 1 arbiter; do
  if [[ ! -f "mongodb-${i}.crt" ]]; then
    openssl genrsa -out "mongodb-${i}.key" 2048
    openssl req -new -key "mongodb-${i}.key" -subj "/CN=mongodb-${i}.example.com" -out "mongodb-${i}.csr"
    openssl x509 -req -in "mongodb-${i}.csr" -CA ca.crt -CAkey ca.key -CAcreateserial -out "mongodb-${i}.crt" -days 1024 -sha256
  fi
  kubectl create secret tls "mongodb-${i}-cert"  --cert="mongodb-${i}.crt" --key="mongodb-${i}.key"
  kubectl patch secret "mongodb-${i}-cert" -p="{\"data\":{\"ca.crt\": \"$(cat ca.crt | base64 -w0 )\"}}"
done

3. To avoid issues with name resolution this configuration has been added to the coredns configmap:

rewrite name mongodb-0.example.com mongodb-0.mongodb-headless.default.svc.cluster.local
rewrite name mongodb-1.example.com mongodb-1.mongodb-headless.default.svc.cluster.local

4. Values used:

architecture: replicaset
replicaCount: 2
tls:
  enabled: true
  autoGenerated: false
  replicaset:
    existingSecrets:
      - mongodb-0-cert
      - mongodb-1-cert
  arbiter:
    existingSecret: mongodb-arbiter-cert
externalAccess:
  enabled: true
  service:
    loadBalancerIPs:
      - 10.10.0.3
      - 10.10.0.4
    annotationsList:
      - external-dns.alpha.kubernetes.io/hostname: mongodb-0.example.com
      - external-dns.alpha.kubernetes.io/hostname: mongodb-1.example.com
  publicNames:
    - mongodb-0.example.com
    - mongodb-1.example.com

Checklist

• Chart version bumped in Chart.yaml according to semver. This is not necessary when the changes only affect README.md files.
• Variables are documented in the values.yaml and added to the README.md using readme-generator-for-helm
• Title of the pull request follows this pattern [bitnami/<name_of_the_chart>] Descriptive title
• All commits signed off and in agreement of Developer Certificate of Origin (DCO)

[rrileyca]

Hey @fmulero thanks for looking into this.

Does this work? One of the issues is that Mongo will see the external DNS name in the Replica config (which can be seen in mongosh with rs.config()) and that won't match its hostname, so it will think it's not in the cluster. Have you tested this?

[fmulero]

Hey @fmulero thanks for looking into this.

Does this work? One of the issues is that Mongo will see the external DNS name in the Replica config (which can be seen in mongosh with rs.config()) and that won't match its hostname, so it will think it's not in the cluster. Have you tested this?

At the moment I didn't face that issue, maybe because I am changing my coredns config to rewrite mongodb-0.example.com to mongodb-0.mongodb-headless.default.svc.cluster.local

[fmulero]

Hi @rrileyca, I think I reproduced the problem you mentioned. To fix it I configured the cluster using the external names and I added an init container to ensure that we can resolve the external name before init the cluster.

[rrileyca]

Hey @fmulero, thanks this is defnitely a useful feature.

Does it work without the coreDNS edits in the kubernetes cluster though? I think the external DNS name has to match the hostname of the Pod, and failing a hostname match it does a DNS lookup and tries to match the IP's of the Replicas to its own IP. I don't see any changes to the hostname of the Pod, just the services.

The reason changing the coreDNS config works (I believe) is because you can put the public DNS names like mongo-1.mydns.com in the Mongo ReplicaSet config, which coreDNS then translates to the service name like mongodb.svc.cluster.local and resolves the Pod IP. Mongo sees that IP address matches its local interface address and then assumes it is part of the cluster.

I think the problem here is that when you resolve the public DNS name of a node, it will resolve to the Loadbalancer IP address which is not the Pod IP. Mongo will not think it is part of the cluster in this case.

Does your PR assume that you have a DNS name that always translates to the Pod IP? If so, how do you keep this DNS entry up to date? I've only used external-dns for Loadbalancers and Ingresses.

[fmulero]

HI @rrileyca thanks for taking care of this. I've tested it following these steps:

1. Create the certificates with the script described in the description.
2. Run a regular installation with the values below:

architecture: replicaset
replicaCount: 2
tls:
  enabled: true
  autoGenerated: false
  replicaset:
    existingSecrets:
      - mongodb-0-cert
      - mongodb-1-cert
  arbiter:
    existingSecret: mongodb-arbiter-cert
externalAccess:
  enabled: true
  service:
    type: LoadBalancer
    publicNames:
      - 'mongodb-0.example.com'
      - 'mongodb-1.example.com'

That will create 2 external services and the pods will wait for the mongodb-0.example.com name resolution.

3. Once the external services have their external ip addresses we can upgrade the deployment adding hostAliases to resolve the external names:

architecture: replicaset
replicaCount: 2
tls:
  enabled: true
  autoGenerated: false
  replicaset:
    existingSecrets:
      - mongodb-0-cert
      - mongodb-1-cert
  arbiter:
    existingSecret: mongodb-arbiter-cert
externalAccess:
  enabled: true
  service:
    type: LoadBalancer
    publicNames:
      - 'mongodb-0.example.com'
      - 'mongodb-1.example.com'
hostAliases: &aliases
  - ip: 35.243.236.68
    hostnames:
      - mongodb-0.example.com
  - ip: 34.75.99.90
    hostnames:
      - mongodb-1.example.com
arbiter:
  hostAliases: *aliases

4. At this point our pods are stuck waiting for name resolution. In this case, we have changed the StatefulSet adding the hostAliases, we have to remove old pods.

$ kubectl delete pods mongodb-0 mongodb-arbiter-0
pod "mongodb-0" deleted
pod "mongodb-arbiter-0" deleted

5. New pods start running and get ready. Now if we run the following commands we can see how mongodb-0 is primary and mongodb-1 is secondary:

$ kubectl exec -it mongodb-0 -c mongodb -- mongosh --tls --tlsCertificateKeyFile=/certs/mongodb.pem --tlsCAFile=/certs/mongodb-ca-cert --tlsAllowInvalidHostnames --port $MONGODB_PORT_NUMBER  --eval "db.hello().isWritablePrimary"
true
$ kubectl exec -it mongodb-0 -c mongodb -- mongosh --tls --tlsCertificateKeyFile=/certs/mongodb.pem --tlsCAFile=/certs/mongodb-ca-cert --tlsAllowInvalidHostnames --port $MONGODB_PORT_NUMBER  --eval "db.hello().secondary"
false
$ kubectl exec -it mongodb-1 -c mongodb -- mongosh --tls --tlsCertificateKeyFile=/certs/mongodb.pem --tlsCAFile=/certs/mongodb-ca-cert --tlsAllowInvalidHostnames --port $MONGODB_PORT_NUMBER  --eval "db.hello().isWritablePrimary"
false
$ kubectl exec -it mongodb-1 -c mongodb -- mongosh --tls --tlsCertificateKeyFile=/certs/mongodb.pem --tlsCAFile=/certs/mongodb-ca-cert --tlsAllowInvalidHostnames --port $MONGODB_PORT_NUMBER  --eval "db.hello().secondary"
true

From my understanding everything looks good.

[rafariossaa]

LGTM.

[rrileyca]

thanks @fmulero. Should those instructions regarding the hostAliases be added to the README? As I understand it, there is no way to deploy using public names in a single helm install. You need to deploy it, let the IPs be allocated, and then add the hostAliases and run a helm update afterward.

[fmulero]

I've just added a note about hostAliases. From my understanding that is useful in testing environments. In the real world, developers or SREs should be able to update their DNS records, using external-dns (the chart allows it) or manually configuring the DNS server records. In that case the pods will wait for name resolution and then they'll continue with the startup process.