This guide doesn't try to be generic and there are indeed more ways of generating HTTPS/TLS certificates for kubernetes GKE clusters. Instead it provides (hopefully) a reproducible and complete guide for generating certificates according to the following:
- Google-managed certificate (not letsencrypt)
- Ingress via google load balancer
- Using "official" https://github.com/GoogleCloudPlatform/gke-managed-certs
Limitations of this approach:
- Ingress only applies to 1 namespace -> 1 load balancer per namespace
Tested on: v1.11.6-gke.3
Create a new GKE cluster (if needed) using cloud console
For quick testing you can use "compute engine admin".
Recommended, create custom role for creating TLS certs with access:
compute.sslCertificates.create
compute.sslCertificates.delete
compute.sslCertificates.get
compute.sslCertificates.list
Create a new service account and give apply the role above. Create and download json key. Save as gke-managed-certs.json - to be used below.
gcloud container clusters get-credentials <cert-test-cluster>
kubectl create namespace infra
kubectl config set-context $(kubectl config current-context) --namespace=infra # Make infra default namespace
gcloud compute addresses create tls-ingress-ip --global
kubectl run web --image=gcr.io/google-samples/hello-app:1.0 --port=8080
kubectl expose deployment web --target-port=8080 --type=NodePort
kubectl get service web
gcloud compute addresses describe tls-ingress-ip --global
Using the IP address from the last command, create 1 (or 2 to complete full guide) DNS names (with your DNS provider) with A records toward this IP.
e.g. web1.test.com, web2.test.com
Modify deploy/ingress-demo.yaml and specify your domain name and then deploy the ingress:
kubectl apply -f deploy/base/ingress-demo.yaml
After a while (5-15min) you should be able to access http://web1.example.com, feel free to continue with the guide in the meantime
kubectl get po -o wide # check status
NOTE: yamls taken from https://github.com/GoogleCloudPlatform/gke-managed-certs. You may want to use the most up to date there instead of from this project but I included here fore reproducibility and stability (latest sometimes broken)
NOTE2: You need to modify the managed-certificate-controller.yaml to add env variable for service account and mount secret (already done if you use the yaml from example folder)
kubectl create secret generic gcp-svc-acc-gke-managed-certs --from-file=/tmp/gke-managed-certs.json # File must be called "gke-managed-certs.json"
kubectl apply -f deploy/example1/managedcertificates-crd.yaml # Install CRDs
# Give yourself access to deploy cert manager, use real email
kubectl create clusterrolebinding make-me-cluster-admin --clusterrole=cluster-admin [email protected]
# Give gke-managed-certs controller cluster admin access
# Change namespace if deploying to another namespace
kubectl create clusterrolebinding managed-certificate-account-cluster-admin --clusterrole=cluster-admin --user=system:serviceaccount:infra:managed-certificate-account
kubectl apply -f deploy/example1/managed-certificate-controller.yaml # Install the "cert manager"
Modify managed-cert.yaml with your domain name, then apply that and annotate the ingress
kubectl apply -f deploy/example1/managed-cert.yaml
kubectl annotate ingress test-ingress networking.gke.io/managed-certificates=test-cert
You should se (may take some time) your ingress getting new annotations to look similar to:
$ kubectl describe ingress test-ingress
Name: test-ingress
Namespace: infra
Address: 35.201.65.x
Default backend: default-http-backend:80 (10.20.5.5:8080)
Rules:
Host Path Backends
---- ---- --------
web1.example.com
/ web:8080 (<none>)
Annotations:
ingress.kubernetes.io/https-target-proxy: k8s-tps-infra-test-ingress--ed7c677087a4323e
ingress.kubernetes.io/ssl-cert: mcrt-53a034a7-5dea-4980-83a8-aaf605e4589f
ingress.kubernetes.io/target-proxy: k8s-tp-infra-test-ingress--ed7c677087a4323e
ingress.kubernetes.io/url-map: k8s-um-infra-test-ingress--ed7c677087a4323e
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.global-static-ip-name":"tls-ingress-ip"},"name":"test-ingress","namespace":"infra"},"spec":{"rules":[{"host":"web1.example.com","http":{"paths":[{"backend":{"serviceName":"web","servicePort":8080},"path":"/"}]}}]}}
gke.googleapis.com/managed-certificates: test-cert
ingress.gcp.kubernetes.io/pre-shared-cert: mcrt-53a034a7-5dea-4980-83a8-aaf605e4589f
ingress.kubernetes.io/backends: {"k8s-be-30117--ed7c677087a4323e":"HEALTHY","k8s-be-31334--ed7c677087a4323e":"HEALTHY"}
kubernetes.io/ingress.global-static-ip-name: tls-ingress-ip
ingress.kubernetes.io/forwarding-rule: k8s-fw-infra-test-ingress--ed7c677087a4323e
ingress.kubernetes.io/https-forwarding-rule: k8s-fws-infra-test-ingress--ed7c677087a4323e
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ADD 6m38s loadbalancer-controller infra/test-ingress
Normal CREATE 5m26s loadbalancer-controller ip: 35.201.65.90
Provisioning the certificate may take some time and you can follow the progress in cloud console UI -> Network Services -> Load balancer -> your new lb -> mcrt-... (may say "provisioning for some time)
You can also check:
gcloud compute ssl-certificates list
gcloud compute ssl-certificates describe mcrt-<your cert>
kubectl get/describe mcrt test-cert
Even after everything looks good it might still take some time before the load balancer properly terminates https (you may see handshake errors)
Eventually it should work via https too!
NOTE: requires example 1
kubectl run web2 --image=gcr.io/google-samples/hello-app:2.0 --port=8080
kubectl expose deployment web2 --target-port=8080 --type=NodePort
# Modify domain name in deploy/example2/managed-cert-web2.yaml
kubectl apply -f deploy/example2/managed-cert-web2.yaml
kubectl apply -f deploy/example2/ingress-demo-v2.yaml
kubectl annotate ingress test-ingress networking.gke.io/managed-certificates=test-cert,web2-cert
Each load balancer can only be linked to one ingress and each ingress can only reference services in it's own namespace The certificate controller as installed above manages certs in all namespaces. To use it in a new namespace, the following is needed:
kubectl create ns apps
kubectl config set-context $(kubectl config current-context) --namespace=apps
gcloud compute addresses create apps-ingress-ip --global # Request IP
gcloud compute addresses describe apps-ingress-ip --global # Get IP
# Set up DNS record to web3.example.com to the IP
# Change domain in example3 ingress and managed cert
kubectl apply -f deploy/example3/managed-cert-3.yaml # Define cert to reference in ingress
kubectl run web2 --image=gcr.io/google-samples/hello-app:2.0 --port=8080 # Create a new "web2" in apps namespace
kubectl expose deployment web2 --target-port=8080 --type=NodePort
kubectl apply -f deploy/example3/ingress-demo-v3.yaml # incl. cert annotation, triggers complete setupThe controller pod doesn't log to stdout for some reason so you can't check it's logs in stackdriver so you need to look at the log file which is not so convenient in a container that doesn't even have bash nor basic tools like less.
kubectl exec -it --namespace infra managed-certificate-controller-54fbb9cf78-74jxl -- /bin/sh
# tail -n 30 /var/log/managed_certificate_controller.log