VXLAN on Support Tools

Cilium MTU Black Hole: How a Linux Bridge and KubeVirt VMs Silently Broke VXLAN and Kept Longhorn Degraded

Sun, 25 Oct 2026 09:00:00 -0500

For about ten hours, roughly 45 Longhorn volumes on one of our production RKE2 clusters sat in a degraded state. Replicas flapped to ERR, rebuilds started and failed, and the Longhorn event stream logged more than 1,400 replica errors, nearly 800 of them in a single hour on one node. Small API calls worked. DNS worked. Health checks were green. Only bulk traffic between certain pairs of nodes was disappearing, and which pairs changed over the course of the day.

The cause turned out to be three ordinary behaviors stacked on top of each other. A Linux bridge takes the MTU of its smallest port. KubeVirt VMs attach 1500-byte veths to that bridge. And Cilium, with no MTU configured, does not just read the device MTU at startup: it follows it while running. Pods keep the MTU they were created with. So every time a VM moved onto or off a node, that node’s tunnel MTU changed underneath pods that still believed the old value, and their large VXLAN packets went into a black hole with no error anywhere.

This post walks through the mechanism, a reproducible demonstration of the bridge behavior, the commands that prove it on a live cluster, the fix (pin the MTU, flatten the underlay), and the planned full-cluster outage it took to apply that fix safely.

Networking

Fri, 25 Oct 2024 00:00:00 +0000

Welcome to our comprehensive networking training section. Here you’ll find detailed guides and tutorials covering everything from basic networking concepts to advanced topics in modern cloud-native networking.

Core Networking Concepts

Networking 101: TCP/IP, subnets, and fundamental concepts
Understanding VXLAN: Deep dive into Virtual Extensible LAN
Network Security Fundamentals: Firewalls, encryption, and security best practices

Container and Cloud Networking

Kubernetes VXLAN Networking: How Kubernetes uses VXLAN for overlay networks
Container Network Interface (CNI): Understanding Kubernetes networking plugins
Service Mesh Architecture: Modern microservices networking

Advanced Topics

Network Troubleshooting: Tools and techniques for debugging network issues
Network Performance: Optimization and monitoring
Software-Defined Networking: Modern approaches to network management

Hands-on Labs

Setting up a VXLAN Network
Kubernetes CNI Implementation
Network Monitoring Lab

Each section includes practical examples, diagrams, and real-world scenarios to help you understand and implement networking concepts effectively.

How Kubernetes Uses VXLAN for Overlay Networking

Thu, 24 Oct 2024 00:00:00 +0000

How Kubernetes Uses VXLAN for Overlay Networking

Kubernetes needs to create a virtual network that connects pods across multiple nodes seamlessly. Since these pods reside in isolated networks, overlay networks—like those implemented with VXLAN—allow them to communicate over a physical Layer 3 infrastructure.

In this post, we explore how Kubernetes uses VXLAN to build an overlay network, enabling communication between pods across nodes.

1. What is an Overlay Network?

In Kubernetes, an overlay network abstracts the physical network, allowing pods on different nodes to communicate as if they were on the same Layer 2 network. VXLAN is one of the most popular protocols for building this overlay.

Understanding VXLAN

Thu, 24 Oct 2024 00:00:00 +0000

Understanding VXLAN: Virtual Extensible LAN

Virtual Extensible LAN (VXLAN) is a network virtualization technology that helps extend Layer 2 networks over Layer 3 infrastructure. It’s a key enabler in modern data centers, especially in cloud environments, providing scalable overlay networks for virtual machines and containers.

1. What is VXLAN?

VXLAN is a network overlay protocol that allows you to create a virtual Layer 2 network over a physical Layer 3 network. It encapsulates Ethernet frames in UDP packets, enabling the deployment of virtual networks across large, distributed environments.