CoreDNS and NodeLocal DNSCache: Troubleshoot and Monitor

DNS is the cornerstone of any Kubernetes cluster. CoreDNS answers cluster DNS queries, and NodeLocal DNSCache puts a caching resolver on every node in front of it. This guide explains how both work on RKE2, how to enable NodeLocal DNSCache with Canal or Cilium, how to troubleshoot failures and latency, and which Prometheus metrics to watch.

What is CoreDNS?

CoreDNS is a flexible, extensible DNS server and the default cluster DNS provider for Kubernetes. It provides:

• Service discovery via DNS for Kubernetes Services and Pods.
• Forwarding of external names to upstream resolvers.
• A plugin-based architecture for customization.

CoreDNS in RKE2

RKE2 installs CoreDNS from its packaged rke2-coredns Helm chart into kube-system, together with the cluster-proportional-autoscaler that scales CoreDNS replicas with cluster size. Pods keep the k8s-app=kube-dns label, the Deployment is rke2-coredns-rke2-coredns, and the Service is still named kube-dns. You change it with a HelmChartConfig named rke2-coredns, not by editing the ConfigMap, because the Helm controller reverts manual edits. Disable it with disable: rke2-coredns in /etc/rancher/rke2/config.yaml only if you deploy another cluster DNS.

Recent RKE2 releases (from the September 2026 releases, per the RKE2 networking docs) also allow Corefile customization through the chart. The docs warn that only the default Corefile is tested, so keep changes small.

For deeper Corefile topics, see CoreDNS with Rancher: Best Practices and Troubleshooting.

What is NodeLocal DNSCache?

NodeLocal DNSCache (often called NodeLocalDNS) runs a small CoreDNS-based cache as a DaemonSet on every node, listening on a link-local address (169.254.20.10 by default). Pods send queries to the local cache, which answers from cache or forwards cluster names to CoreDNS over TCP. Benefits:

• Lower DNS latency, since most answers come from the same node.
• Less load on the CoreDNS Deployment.
• Avoids conntrack races on UDP DNS that cause intermittent 5 second lookups.

NodeLocal DNSCache in RKE2

In RKE2, NodeLocal DNSCache is optional and is part of the rke2-coredns chart. Enable it with a HelmChartConfig, placed in /var/lib/rancher/rke2/server/manifests/ on a server node or applied with kubectl.

Canal or Calico with kube-proxy in iptables mode:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-coredns
  namespace: kube-system
spec:
  valuesContent: |-
    nodelocal:
      enabled: true

kube-proxy in IPVS mode: add ipvs: true under nodelocal. In IPVS mode the cache cannot intercept the Service IP, so every node’s kubelet must point at the node-local address instead, for example kubelet-arg: ["cluster-dns=169.254.20.10"] in the RKE2 config, followed by an RKE2 restart and a restart of running pods. Note that kube-proxy IPVS mode is deprecated starting in Kubernetes v1.35.

Cilium with kube-proxy replacement: Cilium’s eBPF datapath bypasses the iptables rules NodeLocal normally uses, so RKE2 uses a Cilium Local Redirect Policy instead (supported since v1.28.13+rke2r1, v1.29.8+rke2r1 and v1.30.4+rke2r1):

---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-cilium
  namespace: kube-system
spec:
  valuesContent: |-
    kubeProxyReplacement: true
    k8sServiceHost: <api-server-ip>
    k8sServicePort: 6443
    localRedirectPolicy: true
---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-coredns
  namespace: kube-system
spec:
  valuesContent: |-
    nodelocal:
      enabled: true
      use_cilium_lrp: true

Set k8sServiceHost to an address that reaches the API server from every node (RKE2’s docs use your API server IP). The RKE2 docs still show the older localRedirectPolicy: true key, while the current Cilium docs enable the feature with localRedirectPolicies.enabled: true. Whichever key your Cilium chart version accepts, confirm the result with kubectl -n kube-system get cm cilium-config -o yaml | grep local-redirect, which should show enable-local-redirect-policy: "true". For Cilium side debugging, see the Cilium Troubleshooting Guide.

Warning from the RKE2 docs: NodeLocal changes node iptables rules to intercept DNS, so enabling it and then disabling it without redeploying can break DNS until the rules are cleaned up.

Troubleshooting CoreDNS and NodeLocal DNSCache

Create a test pod first. The upstream dnsutils manifest uses registry.k8s.io/e2e-test-images/agnhost, which has reliable nslookup and dig (BusyBox nslookup gives misleading results with search domains):

kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
kubectl wait --for=condition=Ready pod/dnsutils
kubectl exec -it dnsutils -- cat /etc/resolv.conf

The nameserver line tells you which path the pod uses: the CoreDNS Service IP, or 169.254.20.10 in IPVS-mode NodeLocal setups.

Common CoreDNS issues and fixes

DNS resolution failures

• Symptoms: applications report “unknown host” or SERVFAIL, or fail to resolve Service names.
• Checks:
  • CoreDNS pods are running: kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide.
  • CoreDNS logs: kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50.
  • The Service has endpoints: kubectl get endpointslices -n kube-system -l k8s-app=kube-dns.
  • Internal name: kubectl exec -it dnsutils -- nslookup kubernetes.default.
  • Service in another namespace: kubectl exec -it dnsutils -- nslookup <service>.<namespace>.svc.cluster.local.
  • External name: kubectl exec -it dnsutils -- nslookup example.com.
• Fixes:
  • Validate the Corefile for syntax errors; CoreDNS logs plugin/... errors at startup.
  • Make sure the forward plugin points at working upstream resolvers. By default it uses the node’s /etc/resolv.conf. On hosts with systemd-resolved that file points at 127.0.0.53, which is unreachable from inside a pod; RKE2 and K3s detect this and hand kubelet /run/systemd/resolve/resolv.conf instead, and on other distributions you set kubelet’s --resolv-conf yourself.
  • If only pods on some nodes fail, test pod to pod connectivity across nodes (see the overlay test below); DNS is often the first symptom of a broken CNI overlay.
  • Turn on query logging temporarily by adding the log plugin to the Corefile.

High latency in DNS queries

• Symptoms: slow first requests, or lookups that take about 5 seconds and then succeed.
• Checks:
  • CPU throttling or OOM restarts in CoreDNS pods: kubectl top pods -n kube-system -l k8s-app=kube-dns.
  • Query time from a pod: kubectl exec -it dnsutils -- dig +stats kubernetes.default.svc.cluster.local.
  • The ndots:5 search path: an external name like api.example.com is first tried with every search domain. A trailing dot (api.example.com.) or a lower ndots in the pod dnsConfig avoids that.
• Fixes:
  • Keep the cache plugin enabled (it is in the default Corefile) and give CoreDNS enough CPU.
  • Scale CoreDNS (adjust the autoscaler parameters in the rke2-coredns HelmChartConfig).
  • Enable NodeLocal DNSCache to remove cross-node hops and the UDP conntrack race.

More tuning ideas are in Kubernetes DNS Optimization.

NodeLocal DNSCache troubleshooting

Local cache not responding

• Symptoms: DNS is slow or fails on some nodes, or pods bypass the cache.
• Checks:
  • The DaemonSet is running on every node: kubectl get ds -n kube-system | grep -i node-local and kubectl get pods -n kube-system -o wide | grep -i node-local.
  • Query the cache directly from a pod on the affected node: kubectl exec -it dnsutils -- nslookup kubernetes.default 169.254.20.10.
  • In iptables mode, check on the node that the link-local address exists: ip addr show nodelocaldns.
  • With Cilium, confirm the redirect policy exists: kubectl get ciliumlocalredirectpolicies -A.
• Fixes:
  • Restart the NodeLocal pod on the affected node.
  • If you just toggled NodeLocal on or off, restart RKE2 on the node and recreate pods so iptables rules and resolv.conf match.
  • With Cilium, make sure localRedirectPolicy is enabled in rke2-cilium and the Cilium agents were restarted after the change.

Overlay test

DNS failures on only some nodes usually mean pod to pod traffic between nodes is broken. Rancher’s support-tools overlay test checks that. Deploy its DaemonSet:

kubectl apply -f https://raw.githubusercontent.com/rancherlabs/support-tools/master/swiss-army-knife/overlaytest.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: overlaytest
spec:
  selector:
      matchLabels:
        name: overlaytest
  template:
    metadata:
      labels:
        name: overlaytest
    spec:
      tolerations:
      - operator: Exists
      containers:
      - image: rancherlabs/swiss-army-knife
        imagePullPolicy: IfNotPresent
        name: overlaytest
        command: ["sh", "-c", "tail -f /dev/null"]
        terminationMessagePath: /dev/termination-log

This will deploy a DaemonSet that will run on all nodes in the cluster. These pods will be running tail -f /dev/null, which will do nothing but keep the pod running.

To run the overlay test script, execute the following command:

curl -sfL https://raw.githubusercontent.com/rancherlabs/support-tools/master/swiss-army-knife/overlaytest.sh | bash

The script below is a simplified version that adds optional DNS checks with a --dns-check flag:

#!/bin/bash

DNS_CHECK=false

# Parse arguments
while [[ $# -gt 0 ]]; do
  case $1 in
    --dns-check)
      DNS_CHECK=true
      shift
      ;;
    *)
      echo "Unknown option: $1"
      exit 1
      ;;
  esac
done

echo "=> Start network overlay, DNS, and API test"

kubectl get pods -l name=overlaytest -o jsonpath='{range .items[*]}{@.metadata.name}{" "}{@.spec.nodeName}{" "}{@.status.podIP}{"\n"}{end}' | sort -k2 |
while read spod shost sip
do
  echo "Testing pod $spod on node $shost with IP $sip"

  # Overlay network test
  echo "  => Testing overlay network connectivity"
  kubectl get pods -l name=overlaytest -o jsonpath='{range .items[*]}{@.status.podIP}{" "}{@.spec.nodeName}{"\n"}{end}' | sort -k2 |
  while read tip thost
  do
    if [[ ! $shost == $thost ]]; then
      kubectl --request-timeout='10s' exec $spod -c overlaytest -- /bin/sh -c "ping -c2 $tip > /dev/null 2>&1"
      RC=$?
      if [ $RC -ne 0 ]; then
        echo "    FAIL: $spod on $shost cannot reach pod IP $tip on $thost"
      else
        echo "    PASS: $spod on $shost can reach pod IP $tip on $thost"
      fi
    fi
  done

  if $DNS_CHECK; then
    # Internal DNS test
    echo "  => Testing internal DNS"
    kubectl --request-timeout='10s' exec $spod -c overlaytest -- /bin/sh -c "nslookup kubernetes.default > /dev/null 2>&1"
    RC=$?
    if [ $RC -ne 0 ]; then
      echo "    FAIL: $spod cannot resolve internal DNS for 'kubernetes.default'"
    else
      echo "    PASS: $spod can resolve internal DNS for 'kubernetes.default'"
    fi

    # External DNS test
    echo "  => Testing external DNS"
    kubectl --request-timeout='10s' exec $spod -c overlaytest -- /bin/sh -c "nslookup rancher.com > /dev/null 2>&1"
    RC=$?
    if [ $RC -ne 0 ]; then
      echo "    FAIL: $spod cannot resolve external DNS for 'rancher.com'"
    else
      echo "    PASS: $spod can resolve external DNS for 'rancher.com'"
    fi
  else
    echo "  => DNS checks are skipped. Use --dns-check to enable."
  fi

done

echo "=> End network overlay, DNS, and API test"

This script allows for comprehensive testing of overlay network connectivity, internal DNS, and external DNS. The --dns-check flag enables DNS-specific tests, ensuring the cluster’s DNS functionality is verified.

Monitoring CoreDNS and NodeLocal DNSCache

CoreDNS metrics

CoreDNS exposes Prometheus metrics on port 9153 through the prometheus plugin. Many older guides use metric names that were removed in CoreDNS 1.7.0 (coredns_dns_request_count_total, coredns_dns_request_error_count_total). The current names are:

Useful queries:

# Queries per second
sum(rate(coredns_dns_requests_total[5m]))

# SERVFAIL ratio
sum(rate(coredns_dns_responses_total{rcode="SERVFAIL"}[5m]))
  / sum(rate(coredns_dns_responses_total[5m]))

# 99th percentile latency
histogram_quantile(0.99, sum by (le) (rate(coredns_dns_request_duration_seconds_bucket[5m])))

# Cache hit ratio
sum(rate(coredns_cache_hits_total[5m]))
  / (sum(rate(coredns_cache_hits_total[5m])) + sum(rate(coredns_cache_misses_total[5m])))

NodeLocal DNSCache metrics

The node cache is CoreDNS-based, so it exports the same coredns_* metric names. In the upstream NodeLocal DNSCache manifest the metrics listen on port 9253; check the DaemonSet’s container ports and annotations in your cluster before adding a scrape job, and add a job or instance filter so node cache metrics are not mixed with the CoreDNS Deployment.

Visualization and alerting with Grafana

• Import dashboard 14981 (“CoreDNS”, built for the metric names used by CoreDNS 1.7.0 and later). Dashboards made for the old names show empty panels.
• Alert on a sustained SERVFAIL ratio, on p99 latency above your target (for example 100 ms), and on CoreDNS pods not being ready.
• Rancher Monitoring (kube-prometheus-stack) already scrapes CoreDNS on RKE2 and ships CoreDNS alert rules; check them before adding your own.

FAQ

What IP does NodeLocal DNSCache use?

169.254.20.10 by default, set by nodelocal.ip_address in the RKE2 chart. In iptables mode the cache also intercepts traffic sent to the CoreDNS Service IP, so pods do not need a different nameserver. In IPVS mode you point kubelet at 169.254.20.10 instead.

Does NodeLocal DNSCache work with Cilium kube-proxy replacement?

Yes, but not through iptables. On RKE2 set nodelocal.use_cilium_lrp: true and enable Local Redirect Policies in Cilium, as shown above.

Why does DNS take 5 seconds and then work?

The usual causes are the UDP conntrack race on parallel A and AAAA queries, or a lost packet across a broken overlay. NodeLocal DNSCache fixes the first; the overlay test above finds the second.

Why do my CoreDNS Grafana panels show “No data”?

The dashboard uses metric names from before CoreDNS 1.7.0. Use coredns_dns_requests_total and coredns_dns_responses_total, or import dashboard 14981.

Related guides

• CoreDNS with Rancher: Best Practices and Troubleshooting
• Kubernetes DNS Optimization
• Deploying an EKS Cluster with NodeLocal DNSCache
• Cilium Troubleshooting Guide