CoreDNS AWS NLB health check not getting healthy

[diego7marques]

Describe the issue
When k8gb is deployed on AWS, the k8gb-coredns service creates an AWS Network Load Balancer with TCP health check, but no TCP port is available, so the target group never gets healthy.

Generated service example:

apiVersion: v1
kind: Service
metadata:
  annotations:
    ...
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
  labels:
    ...
  name: k8gb-coredns
  namespace: k8gb
spec:
  ...
  ports:
  - name: udp-5353
    nodePort: 31412
    port: 53
    protocol: UDP
    targetPort: 5353
  selector:
    app.kubernetes.io/instance: k8gb
    app.kubernetes.io/name: coredns
  type: LoadBalancer

To Reproduce
Deploy k8gb following: https://www.k8gb.io/docs/deploy_route53.html

Expected behavior
The k8gb-coredns should expose a TCP port along with the UDP, so the target group gets healthy, as UDP health checks are not supported by AWS NLB.

Workaround
In case you are facing the same issue, you can overcome the error if you add the following annotation to your values.yaml within coredns:

service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "<nginx_service_port>"

The port must be filled with the Nginx service nodePort. As Nginx has TCP ports exposed, the health check will work.

Screenshots

Additional context
Me and @ytsarev tried to check if it was possible to add extra ports through CoreDNS helm chart, but the service ports are imported using a helper, and not .Values.xx. References:

• https://github.com/k8gb-io/coredns-helm/blob/master/stable/coredns/templates/service.yaml#L43-L45
• https://github.com/k8gb-io/coredns-helm/blob/master/stable/coredns/templates/_helpers.tpl#L60-L63

[abaguas]

We should try to use the upstream coredns chart where these options are available: https://github.com/coredns/helm/blob/master/charts/coredns/templates/_helpers.tpl#L101-L105