Cilium Troubleshooting Guide: Commands That Work in 2026

Quick fix
Runcilium status --wait and cilium connectivity test from your workstation with the cilium CLI.
Then exec into the agent on the affected node (not a random ds/cilium pod):
kubectl -n kube-system exec <cilium-pod-on-that-node> -c cilium-agent -- cilium-dbg status --verbose.
Find the dropped packets with hubble observe --verdict DROPPED --pod <ns>/<pod> or cilium-dbg monitor --type drop; the drop reason tells you whether it is policy, routing or a missing endpoint.
Since Cilium 1.15 the in-pod CLI is cilium-dbg, not cilium.On this page
Troubleshooting Cilium is mostly about running the right command in the right place. This guide is a working reference for current Cilium releases (1.16 through 1.20, with 1.20.3 the current stable at the time of this update): the cilium CLI on your workstation, cilium-dbg inside the agent, and Hubble for flows. It builds on our earlier Cilium Troubleshooting Guide.
What changed since older Cilium guides
ciliuminside the agent pod is nowcilium-dbg. The in-agent debug CLI was renamed in Cilium 1.15. The nameciliumnow refers to the separate cilium CLI you install on your workstation (cilium status,cilium connectivity test,cilium sysdump,cilium hubble port-forward,cilium config).- Kernel requirement. Current Cilium requires Linux 5.10 or later, or an equivalent backport such as RHEL 8.10’s 4.18 kernel (system requirements).
- Sysdump. The old standalone
cilium-sysdumpzip is gone; usecilium sysdump. - Upgrades. Only consecutive minor upgrades are tested (1.19 to 1.20, not 1.18 to 1.20).
Many commands in older blog posts, including the 2025 version of this page, used flags and subcommands that do not exist (cilium kernel-check, cilium monitor -n <namespace>, cilium proxy get access-log). The commands below are taken from the current Cilium docs and CLI help.
Architecture in one minute
- cilium-agent (DaemonSet): programs eBPF on each node, manages endpoints, identities, policy and service load balancing.
- cilium-operator (Deployment): cluster-wide tasks such as IPAM allocation and identity garbage collection.
- cilium-envoy (DaemonSet in recent releases, or embedded): L7 policy, Ingress and Gateway API.
- Hubble (in the agent) plus hubble-relay and hubble-ui: flow visibility.
Packet paths to keep in mind:
- Pod egress: pod, veth, eBPF program on the host side of the veth, routing (native routing or VXLAN/Geneve tunnel), destination.
- Pod ingress: node interface, eBPF on the node interface, veth, pod.
- Service access: the destination ClusterIP is translated to a backend by eBPF at the socket or TC layer, before the packet leaves the pod’s node.
Step 0: Talk to the agent on the right node
kubectl exec ds/cilium picks an arbitrary agent pod, which is the wrong one for node-specific problems. Select the agent on the node that runs the affected pod:
NODE=$(kubectl get pod <pod> -n <namespace> -o jsonpath='{.spec.nodeName}')
CILIUM_POD=$(kubectl -n kube-system get pod -l k8s-app=cilium --field-selector spec.nodeName="$NODE" -o name)
cdbg() { kubectl -n kube-system exec "$CILIUM_POD" -c cilium-agent -- cilium-dbg "$@"; }
cdbg status --verboseThe rest of this guide uses cdbg for cilium-dbg on the selected node. On RKE2 Cilium also runs in kube-system, installed by the rke2-cilium chart.
Cluster-level diagnostics
Overall health
cilium status --wait
kubectl -n kube-system get pods -l k8s-app=cilium -o wide
kubectl -n kube-system get pods -l name=cilium-operator -o widecilium status summarizes agent, operator, Hubble and Envoy readiness and any errors per node. On a single node, cdbg status --verbose shows:
- Kubernetes API connectivity and KVStore (only relevant if you run an external etcd kvstore).
- Controller Status: should read
0/N failing. Any failing controller is named with its last error. - Proxy Status, IPAM allocation, Encryption, and Cluster health (node to node reachability).
Node to node health is checked by cilium-health:
kubectl -n kube-system exec "$CILIUM_POD" -c cilium-agent -- cilium-health status --verboseUnreachable nodes here usually mean a firewall blocking VXLAN (UDP 8472) or Geneve (UDP 6081), the health port (TCP 4240), or an MTU mismatch.
Operator problems
kubectl -n kube-system logs deploy/cilium-operator --tail=100
kubectl -n kube-system describe deploy/cilium-operatorCommon causes: RBAC or API connectivity errors, IPAM exhaustion (no more pod CIDRs to hand out), and OOMKilled pods on large clusters.
Logs and debug mode
kubectl -n kube-system logs "$CILIUM_POD" -c cilium-agent --tail=200 | grep -E 'level=(error|warn)'To enable debug logging, set it in the cilium-config ConfigMap through the cilium CLI (which restarts the agents) or through your Helm values (debug.enabled=true, debug.verbose=datapath):
cilium config set debug true
cilium config set debug-verbose datapath
kubectl -n kube-system get cm cilium-config -o yaml | grep debugTurn it off again afterwards; datapath debug is very noisy. On RKE2, set these through the rke2-cilium HelmChartConfig instead, or the Helm controller will revert them.
Services: Kubernetes vs the datapath
kubectl get svc,endpointslices -n <namespace>
cdbg service list
cdbg bpf lb listA Service missing from cdbg service list, or a backend missing from its list, means the agent has not processed the update. Check the agent log for API errors before restarting anything.
eBPF maps
cdbg map list
cdbg bpf ct list global | wc -lcdbg map list shows each map with its entry count. Watch the cilium_bpf_map_pressure metric for maps that are close to full; a full connection tracking or NAT map causes random new-connection failures. Size them with bpf.mapDynamicSizeRatio or the specific bpf.*Max Helm values.
Pod connectivity
Find the drop
Hubble is the fastest way to see what happens to a flow:
cilium hubble port-forward &
hubble observe --pod <namespace>/<pod> --last 50
hubble observe --verdict DROPPED --namespace <namespace> -fWithout Hubble, use the monitor on the node of the source or destination pod:
cdbg endpoint list
cdbg monitor --type drop
cdbg monitor --related-to <endpoint-id>The drop reason is the key:
| Drop reason | Usual cause |
|---|---|
Policy denied | A network policy does not allow the flow (see policy section) |
Stale or unroutable IP | Traffic to an IP that is no longer a known endpoint, often after pod churn |
CT: Map insertion failed | Connection tracking map full |
Invalid source ip or Unsupported L3 protocol | Spoofed or unexpected traffic, or a misconfigured multi-homed pod |
No tunnel/encapsulation endpoint | The agent does not know the remote node; check cilium-health and node CIDRs |
Structured connectivity test
cilium connectivity testThis creates test workloads in the cilium-test-* namespaces, runs pod to pod, pod to Service, node port, egress and policy scenarios, and cleans up after itself on success. Use --test <name> to run a subset. The troubleshooting docs still publish the older static connectivity-check.yaml manifest per release if you cannot run the CLI.
MTU and encapsulation
Large requests that hang while small ones work point to MTU. With VXLAN or Geneve, the pod MTU must be at least 50 bytes smaller than the underlay MTU. Check what Cilium chose:
cdbg status --verbose | grep -i mtu
kubectl -n kube-system get cm cilium-config -o yaml | grep -i mtuDNS problems
Cilium only proxies DNS when an FQDN policy (toFQDNs) or DNS visibility rule applies to the pod. Check CoreDNS first (see CoreDNS and NodeLocal DNSCache: Troubleshoot and Monitor), then Cilium:
hubble observe --pod <namespace>/<pod> --protocol dns
cdbg fqdn cache listIf pods with FQDN policies cannot resolve, make sure the policy also allows egress to kube-dns on port 53 with a dns rule, otherwise the DNS request itself is dropped. If you use NodeLocal DNSCache with kube-proxy replacement, it needs a Local Redirect Policy.
Policy troubleshooting
Audit mode
Policy audit mode reports what would be denied without dropping it. Enable it cluster-wide with cilium config set policy-audit-mode true (or the Helm value policyAuditMode=true), then look for audit verdicts:
hubble observe --verdict AUDIT --namespace <namespace>Turn it off when you are done; while it is on, policies do not block anything.
Step-by-step policy debugging
Labels on both pods:
bashkubectl get pod <source-pod> -n <ns> --show-labels kubectl get pod <destination-pod> -n <ns> --show-labelsIdentities and endpoints:
bashcdbg endpoint list cdbg identity listPolicies that apply:
bashkubectl get networkpolicy,ciliumnetworkpolicy -n <namespace> kubectl get ciliumclusterwidenetworkpolicy cdbg policy selectors cdbg endpoint get <endpoint-id> cdbg bpf policy get <endpoint-id>The short names are
cnpandccnp.Watch verdicts while you reproduce:
bashcdbg monitor --type policy-verdictTest with a narrow temporary allow policy:
yamlapiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: temp-allow namespace: <namespace> spec: endpointSelector: matchLabels: app: <destination-app> ingress: - fromEndpoints: - matchLabels: app: <source-app>If this fixes it, the original policy’s selectors or ports are wrong. Delete the temporary policy afterwards.
Remember that as soon as any policy selects an endpoint in a direction (ingress or egress), everything not allowed in that direction is denied.
Performance
Connection tracking
cdbg bpf ct list global | wc -l
kubectl -n kube-system get cm cilium-config -o yaml | grep -E 'ct-|conntrack'Flushing the table (cdbg bpf ct flush global) breaks established connections on that node; use it only as a last resort during an incident.
Agent resources
kubectl -n kube-system top pod -l k8s-app=cilium
kubectl -n kube-system describe pod "${CILIUM_POD#pod/}" | grep -A3 -i 'last state'OOMKilled agents or heavy CPU throttling show up as slow endpoint regeneration and delayed policy updates. Check cilium_endpoint_regeneration_time_stats_seconds and cilium_endpoint_regenerations_total in Prometheus.
Useful metrics
cilium_drop_count_totalbyreasonanddirectioncilium_bpf_map_pressurecilium_endpoint_regenerations_totalbyoutcomecilium_unreachable_nodescilium_datapath_errors_total
L7, Envoy, Ingress and Gateway API
cilium status | grep -i envoy
kubectl -n kube-system get pods -l k8s-app=cilium-envoy -o wide
kubectl -n kube-system logs ds/cilium-envoy --tail=100
hubble observe --type l7 --namespace <namespace>
cdbg monitor --type l7For HTTP policy problems: confirm that traffic reaches Envoy (L7 flows appear in Hubble), that the method, path and headers match the rule exactly, and that the L4 port in the rule matches the container port. If you manage Cilium with Argo CD on RKE2, see Fixing Argo CD Out-of-Sync Due to Cilium Identity on RKE2 for a related gotcha.
Upgrades and rollback
Before a minor upgrade, run the preflight check from the upgrade guide, which pre-pulls images and validates existing policies:
helm template cilium/cilium --version 1.20.3 \
--namespace kube-system \
--set preflight.enabled=true \
--set agent=false \
--set operator.enabled=false \
> cilium-preflight.yaml
kubectl create -f cilium-preflight.yaml
# wait until the preflight DaemonSet and Deployment are ready, then
kubectl delete -f cilium-preflight.yamlUpgrade with a values file, not --reuse-values, and set upgradeCompatibility to the version the cluster was first installed with:
helm upgrade cilium cilium/cilium --version 1.20.3 \
--namespace kube-system \
-f my-values.yaml \
--set upgradeCompatibility=1.XRead the upgrade notes for every minor version you cross. To roll back, helm rollback cilium <revision> -n kube-system, then cilium status --wait.
On RKE2 the Cilium version is tied to the RKE2 release and configured through the rke2-cilium HelmChartConfig, so upgrade RKE2 rather than running helm upgrade yourself.
Collect everything: sysdump
cilium sysdump
# Large clusters: limit nodes and log size
cilium sysdump --node-list <node1>,<node2> --logs-limit-bytes 1073741824The resulting zip contains agent and operator logs, cilium-dbg output from every node, BPF maps, Hubble flows and the relevant Kubernetes objects. Attach it to GitHub issues or support cases.
Case studies
Case 1: Intermittent pod connectivity
hubble observe --verdict DROPPED -fwhile reproducing.cdbg endpoint liston both nodes: look for endpoints not inreadystate.cilium-health status --verbosefor node to node problems.cdbg status --verbosefor failing controllers.
Common causes: full CT or NAT maps, MTU mismatch on a tunnel, an endpoint stuck regenerating, or a policy that only matches some replicas’ labels.
Case 2: Service load balancing failures
cdbg service listandcdbg bpf lb listfor the ClusterIP.kubectl get endpointslicesto compare with Kubernetes.hubble observe --to-ip <cluster-ip>or--to-service <ns>/<svc>.
Common causes: pods not Ready (so they are not in EndpointSlices), Service selector mismatch, externalTrafficPolicy: Local on a node without local backends, or a full LB map.
Case 3: Policy enforcement inconsistencies
cdbg policy selectorsto see which identities each selector matches.cdbg monitor --type policy-verdictduring traffic.cdbg identity listto confirm identities match current labels.
Common causes: label typos, namespace selectors missing the k8s:io.kubernetes.pod.namespace label, policies in both NetworkPolicy and CiliumNetworkPolicy form that combine unexpectedly, and stale identities after a label change (recreate the pod).
FAQ
Why does kubectl exec ds/cilium -- cilium status fail with “executable file not found”?
On Cilium 1.15 and later, the binary inside the agent is cilium-dbg. Use cilium-dbg status, or run cilium status with the cilium CLI on your workstation.
Which Cilium versions are supported?
The Cilium project supports the three most recent minor releases. With 1.20 current, that is 1.18, 1.19 and 1.20. Check the release notes for your exact patch level.
How do I debug Cilium on a node that is NotReady?
Use kubectl debug node/<node> -it --image=<image-with-tools> for host-level checks, and read the agent log with kubectl -n kube-system logs <cilium-pod> -c cilium-agent --previous. If the agent never starts, check the init containers (mount-cgroup, apply-sysctl-overwrites, mount-bpf-fs, clean-cilium-state) with kubectl describe pod.