Kubernetes DNS reliability: diagnose first, cache second

Kubernetes DNS incidents are rarely solved safely by adding replicas or a cache first. Trace one lookup from the application through the Pod's resolver configuration, DNS search expansion, the kube-dns Service, CoreDNS endpoints and any upstream resolver; then fix the layer that is actually failing. NodeLocal DNSCache can reduce cross-node latency, UDP loss and conntrack pressure, but it cannot repair missing Service records, unhealthy CoreDNS Pods, an unreachable upstream or an application that creates excessive lookups. This guide is for platform teams operating clusters where intermittent name-resolution delays affect availability or tail latency.

Kubernetes documents that `ClusterFirst` Pods send cluster-domain lookups to cluster DNS and forward non-cluster names upstream. Its official NodeLocal DNSCache design runs a caching agent as a DaemonSet on each node, avoiding part of the kube-proxy DNAT and connection-tracking path. CoreDNS documents that its cache TTL is a maximum: it does not extend a record's original TTL. Those are documented behaviors. The diagnostic order, SLOs, rollout gates and alert policy below are engineering recommendations verified against the primary documentation on 13 October 2026.

Map the lookup path before changing it

Start with a concrete failing tuple: workload, namespace, node, queried name, record type, timestamp and whether the failure was `NXDOMAIN`, `SERVFAIL`, timeout or an unexpectedly slow success. “DNS is down” merges different systems. A Service name missing in one namespace is different from an upstream timeout for a public API, and both are different from a client that retries every candidate search suffix.

For a normal Service, test the fully qualified name such as `payments.prod.svc.cluster.local`, the namespace-qualified name `payments.prod`, and the short name only where the caller shares the namespace. Kubernetes' DNS for Services and Pods explains that short-name resolution is namespace-scoped and that kubelet writes the Pod's DNS configuration. Record the actual cluster domain instead of assuming `cluster.local`.

Draw the dependency path explicitly:

application resolver
  → /etc/resolv.conf search + ndots
  → kube-dns Service or node-local address
  → CoreDNS / NodeLocal DNSCache
  → Kubernetes plugin for cluster records
  → forward plugin for external names
  → upstream resolver / authoritative server

A change at one hop should have a hypothesis and a metric. Increasing CoreDNS replicas does not change search expansion inside the client. Lowering `ndots` does not repair a missing EndpointSlice. A local cache does not make an authoritative `NXDOMAIN` become a valid record.

Diagnose from the Pod outward

Reproduce from a disposable Pod on the same node and namespace as the affected workload. The official Kubernetes DNS debugging guide uses a `dnsutils` Pod and checks `/etc/resolv.conf`, the DNS Pods, logs, the kube-dns Service and EndpointSlices. Follow that sequence because it separates client configuration, service routing and server behavior.

kubectl exec -n prod deploy/api -- cat /etc/resolv.conf
kubectl exec -n prod deploy/api -- nslookup payments.prod.svc.cluster.local
kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide
kubectl -n kube-system get svc kube-dns
kubectl -n kube-system get endpointslice -l kubernetes.io/service-name=kube-dns
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=200

Compare a failing node with a healthy node. If only one node fails, inspect its node-local DNS Pod, routing, NetworkPolicy and conntrack state before scaling the entire cluster DNS deployment. If every node fails for cluster names, inspect CoreDNS health, Kubernetes API access and the `kubernetes` plugin. If only public names fail, inspect the `forward` path and upstream resolver.

Do not run a permanent debug container with broad privileges. Use a bounded incident tool, capture the requested name and response code, and remove it. Avoid logging full internal domain names indefinitely when they can reveal tenant, service or environment identifiers.

Control search expansion and ndots deliberately

A Pod's `/etc/resolv.conf` commonly includes several search suffixes and `options ndots:5`. A name with fewer than the configured number of dots may be tried against multiple search domains before the absolute name. This behavior supports convenient Service discovery, but chatty clients that repeatedly resolve external or partially qualified names can amplify DNS traffic and tail latency.

Prefer fully qualified internal Service names when crossing namespaces. For external dependencies, an absolute name with a trailing dot can bypass search expansion when the application resolver supports it. Do not lower `ndots` cluster-wide as a reflex: it can change lookup order and break applications that depend on Kubernetes search behavior. Test per-workload `dnsConfig` in a canary and verify both internal and external names.

Host-networked Pods need deliberate policy. Kubernetes documents `ClusterFirstWithHostNet` for a Pod using `hostNetwork`; leaving such workloads on the wrong policy can route resolution differently than expected. A Pod with `dnsPolicy: None` takes full responsibility for `nameservers`, `searches` and `options`, so use it only for workloads that truly need isolated resolver configuration.

Decide whether NodeLocal DNSCache addresses the failure

NodeLocal DNSCache is a stable Kubernetes feature that places a caching agent on each node. Pods hit a local address, cache misses go to CoreDNS and external names follow the configured upstream path. Kubernetes lists reduced cross-node latency, less iptables DNAT and conntrack pressure, TCP upgrades from the local agent to CoreDNS, node-level metrics and negative caching as motivations.

That makes NodeLocal DNSCache a strong fit when many Pods repeat the same lookups, DNS packets cross nodes, UDP loss or conntrack saturation drives tail latency, or teams need node-level evidence. It is not automatically useful for low-query clusters, highly unique names with poor cache reuse, broken upstream resolvers or incorrect Service discovery. It also introduces a DaemonSet and node-local dependency that must be monitored and rolled back safely.

Installation differs by kube-proxy mode. Kubernetes documents that iptables mode can listen on both the kube-dns Service IP and node-local address, while IPVS mode requires the node-local address and a kubelet `--cluster-dns` change. Treat that difference as a rollout gate. Verify the actual proxy mode, reserve a non-colliding local address, test IPv4 or IPv6 behavior and confirm how the managed Kubernetes provider owns the configuration before editing it.

Cache enough, but preserve freshness and recovery

Caching converts upstream work into local memory and faster answers, but it also keeps success and denial responses for a period. CoreDNS' cache plugin documents separate success and denial caches, capacity controls and maximum TTLs. A configured maximum only caps duration; it does not lengthen a record with a shorter authoritative TTL.

Set cache policy from change behavior and incident objectives, not a generic “five minutes is fast” rule. Cluster Service records change during rollout, external providers may fail over, and negative answers can appear briefly during creation. Excessive minimum TTL values can retain stale or negative data longer than the source intended. Too little caching pushes repeated work upstream and reduces the value of the local agent.

Track cache hits and misses by layer. A high NodeLocal hit rate with continued latency can point to client or application behavior rather than CoreDNS. A sudden denial-cache increase can signal a deployment resolving a misspelled name. Low reuse with rising request volume suggests the workload is generating unique names; more cache capacity will not fix that pattern.

Scale CoreDNS around demand and failure domains

CoreDNS remains authoritative for Kubernetes Service discovery even when a node-local cache fronts it. Run enough replicas to survive a node or zone loss while serving cache misses and uncached requests. Spread replicas across failure domains, give them realistic CPU and memory requests, and ensure readiness reflects whether the server can answer—not merely whether the process exists.

Kubernetes provides an official DNS horizontal autoscaling mechanism. Autoscaling can help with cluster growth, but it is not a substitute for measuring QPS, response-code mix, latency and saturation. Scaling after a burst also cannot protect the first wave if capacity and cache warmth are both insufficient. Keep baseline headroom and test a CoreDNS replica loss during expected peak traffic.

Do not customize the Corefile without ownership and rollback. Kubernetes' DNS customization guide supports stub domains and upstream nameservers, but a syntax-valid change can still create loops, unexpected forwarding or a larger failure domain. Version the ConfigMap, validate it outside production, roll one replica at a time where possible and retain the last known-good configuration.

Define DNS SLOs and actionable telemetry

Measure success rate and latency at the client boundary, then add server evidence. At minimum separate cluster names from external names, success from `NXDOMAIN` and `SERVFAIL`, and node-local hits from upstream misses. Track request rate by record type, response-code ratio, request-duration percentiles, cache hit and eviction behavior, forwarder health, CoreDNS CPU throttling and memory, Pod restarts, node-local availability and kube-dns EndpointSlice count.

A useful DNS SLO is scoped: “99.95% of approved cluster-name queries succeed within 50 ms over five minutes” is diagnosable; “DNS is healthy” is not. Use values based on observed applications rather than copying this example. Alert on user-visible failure plus a likely cause. A page for every transient `NXDOMAIN` creates noise because negative answers can be legitimate. A rise in timeouts, `SERVFAIL` or high latency paired with forwarder or saturation signals is more actionable.

Keep a synthetic lookup from representative namespaces and nodes, but do not let it be the only signal. One cached name can stay green while uncached external names fail. Use a small matrix: same-namespace Service, cross-namespace FQDN, Kubernetes API Service and one approved external name. Record which layer answered each probe.

Roll out and prove the improvement

Baseline query volume, p50/p95/p99 latency, timeouts, response codes, conntrack pressure and CoreDNS resource use before changing anything. Deploy NodeLocal DNSCache to a small node pool or canary cluster when the platform permits it. Move a bounded workload, verify its resolver configuration and compare the same lookup matrix.

Test normal traffic, a CoreDNS Pod termination, an upstream timeout, a newly created Service, a deleted Service and node-local Pod restart. Define rollback before rollout: restore kubelet cluster DNS where changed, remove the DaemonSet and confirm Pods can resolve through the original Service. Keep the configuration versions and timestamps needed to distinguish a DNS rollout from an application release.

The executive rule is simple: prove the failing hop, control client amplification, add local caching only when it targets measured latency or conntrack pressure, keep CoreDNS redundant, and validate both positive and negative answers during failure. DNS reliability comes from an observable dependency chain—not from one larger cache.

Prepared by: Noor Yasser