Why can I not resolve private EC2 DNS names in my pods when nodelocal dns is used?

[yorinasub17]

I am trying to resolve private EC2 DNS names from pods, but the DNS queries return no answer. The same DNS records resolve correctly when hitting it from the underlying instances. What could be the issue here?

Important note: I have NodeLocal DNS deployed

[autero1]

If you’re having issues with non-cluster lookups with NodeLocal DNS deployed, check your (Corefile) configuration:

1. Which DNS server the fallback .:53 queries are forwarded to:

 .:53 {
        ...
        bind 169.254.20.10 172.20.0.10
        forward . __PILLAR__UPSTREAM__SERVERS__ {
                prefer_udp
        }
        ...
}

Here, __PILLAR__UPSTREAM__SERVERS__ might resolve to 8.8.8.8 or some other custom DNS server, and subsequently fail to resolve the private EC2 DNS name.

2. Optionally add extra configuration the private EC2 stubdomain, e.g.

us-west-2.compute.internal:53 {
        errors
        cache 30
        reload
        loop
        bind 169.254.20.10 172.20.0.10
        forward . __PILLAR__CLUSTER__DNS__ {
                prefer_udp
        }
        prometheus :9253
}