A name resolves fine from your laptop and from the K3s node, but pods get NXDOMAIN. On K3s this almost always comes down to which upstream resolver CoreDNS forwards to. This guide walks through how to prove that and the two supported fixes.

The symptom

In this example an extra CoreDNS instance on the LAN serves *.k3s.differentpla.net, with LoadBalancer and Ingress names registered in it. Queries from outside the cluster work. Inside the cluster they fail: an Argo CD repository URL of https://git.k3s.differentpla.net/USER/REPO.git fails because the name does not resolve.

Step 1: Test from a pod

Use the DNS test pod from the Kubernetes DNS debugging guide:

bash
kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
kubectl exec -it dnsutils -- nslookup kubernetes.default
Server:		10.43.0.10
Address:	10.43.0.10#53

Name:	kubernetes.default.svc.cluster.local
Address: 10.43.0.1

Cluster DNS works. Now the internal name:

bash
kubectl exec -it dnsutils -- nslookup git.k3s.differentpla.net
Server:		10.43.0.10
Address:	10.43.0.10#53

** server can't find git.k3s.differentpla.net: NXDOMAIN

Step 2: Understand the DNS path

Pods use dnsPolicy: ClusterFirst, so their /etc/resolv.conf contains nameserver 10.43.0.10, the K3s CoreDNS Service. CoreDNS answers cluster.local names itself and sends everything else to the resolvers in its own /etc/resolv.conf.

CoreDNS runs with dnsPolicy: Default, so it inherits the resolver file that kubelet hands it:

bash
kubectl -n kube-system get pod -l k8s-app=kube-dns -o jsonpath='{.items[0].spec.dnsPolicy}{"\n"}'
kubectl -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' | grep -n forward
Default
    forward . /etc/resolv.conf

Step 3: See which upstream CoreDNS really uses

The CoreDNS image has no shell, so you cannot simply exec into it. The easiest check is a throwaway pod with the same dnsPolicy: Default, which receives the same resolver file from kubelet:

bash
kubectl run -i --restart=Never --rm test-${RANDOM} --image=busybox:1.36 \
  --overrides='{"apiVersion":"v1","spec":{"dnsPolicy":"Default"}}' -- cat /etc/resolv.conf
nameserver 8.8.8.8

That is the problem: CoreDNS forwards to Google Public DNS, which knows nothing about k3s.differentpla.net.

You can also confirm it with query logging. Add log to the Corefile (in K3s, prefer a coredns-custom key named log.override containing log, since K3s rewrites the main coredns ConfigMap), then read the logs:

bash
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=20
[INFO] 10.42.0.179:43855 - 12121 "A IN git.k3s.differentpla.net.default.svc.cluster.local. udp 68 false 512" NXDOMAIN qr,aa,rd 161 0.000657711s

Why K3s uses 8.8.8.8

Per the K3s advanced options docs, at startup each node checks /etc/resolv.conf and /run/systemd/resolve/resolv.conf. Nameservers that are loopback, multicast or link-local are treated as unusable, because pods cannot reach them. If neither file has a usable nameserver, K3s logs a warning and generates a stub resolver file with 8.8.8.8 and 2001:4860:4860::8888.

Typical triggers:

  • systemd-resolved with only the 127.0.0.53 stub and no real upstream in /run/systemd/resolve/resolv.conf.
  • A local dnsmasq or Pi-hole on 127.0.0.1.
  • A node that got its DNS server by DHCP after K3s started.

Check the K3s logs on the node for the warning:

bash
sudo journalctl -u k3s | grep -i resolv

Fix 1: Give K3s a real resolver file

Create a resolver file that lists your LAN DNS server:

bash
echo "nameserver 192.168.28.1" | sudo tee /etc/k3s-resolv.conf

Point K3s at it in /etc/rancher/k3s/config.yaml (on every node, servers and agents):

yaml
resolv-conf: /etc/k3s-resolv.conf

Then restart K3s (sudo systemctl restart k3s, or k3s-agent on agents) and restart CoreDNS so it picks up the new file:

bash
kubectl -n kube-system rollout restart deployment coredns

A file passed with resolv-conf (the --resolv-conf flag) skips the viability checks, so make sure the server in it is reachable from pods.

Fix 2: Forward only your internal zone with coredns-custom

If you want public names to keep using the default path, forward just the internal domain. K3s’s Corefile imports keys from a ConfigMap named coredns-custom in kube-system: keys ending in .override go inside the default .:53 server block, and keys ending in .server become extra server blocks.

yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  differentpla.server: |
    k3s.differentpla.net:53 {
        errors
        cache 30
        forward . 192.168.28.53
    }
bash
kubectl apply -f coredns-custom.yaml
kubectl -n kube-system rollout restart deployment coredns
kubectl exec -it dnsutils -- nslookup git.k3s.differentpla.net

Use coredns-custom rather than editing the coredns ConfigMap: K3s manages that ConfigMap and overwrites manual changes on restart and upgrade.

FAQ

Why does nslookup on the node work but not in a pod?

The node may resolve through 127.0.0.53 (systemd-resolved), which forwards to your LAN DNS. Pods cannot use a loopback address on the node, so K3s replaced it with 8.8.8.8 for them.

Do I need to restart pods after changing resolv-conf?

CoreDNS needs a restart to read the new file. Application pods with ClusterFirst do not, because their nameserver is still the CoreDNS Service.

Does the same apply to RKE2?

RKE2 performs similar resolver checks and supports resolv-conf in /etc/rancher/rke2/config.yaml. Its CoreDNS is customized through a HelmChartConfig for rke2-coredns instead of coredns-custom; see CoreDNS and NodeLocal DNSCache: Troubleshoot and Monitor.