Rke2 on Support Tools

Ingress-NGINX Retirement on RKE2: Migrating to Traefik Without Downtime

Wed, 04 Feb 2026 00:00:00 -0500

The Kubernetes community announced in November 2025 that Ingress-NGINX is being retired. Maintenance halts completely in March 2026 — no more releases, no more bug fixes, no more security patches. If you’re running RKE2, which ships ingress-nginx by default, this directly affects you.

This post walks through the practical side of migrating RKE2 clusters from ingress-nginx to Traefik. The emphasis is on running both controllers side-by-side so you can migrate gradually without taking production offline.

How to Recover Longhorn Volume Data from a Single Replica in RKE2 Kubernetes

Fri, 11 Apr 2025 00:00:00 -0500

When disaster strikes and your entire RKE2 cluster or Longhorn control plane becomes unavailable, you can still recover critical data from a single Longhorn volume replica using a static pod definition. This emergency recovery method works even when the Kubernetes API server is completely offline and Docker is not available on your RKE2 nodes.

Running RKE2 Ingress-NGINX with an External LoadBalancer

Thu, 10 Apr 2025 00:00:00 -0500

This article provides step-by-step guidance for configuring RKE2’s built-in ingress-nginx controller to run behind an external LoadBalancer. It supports environments like AWS, GCP, Azure, and MetalLB, and replaces the default hostNetwork + DaemonSet configuration with a Deployment and LoadBalancer service.

RKE2 the Hard Way: Part 1 – Introduction and Prerequisites for Building a Kubernetes Cluster

Tue, 01 Apr 2025 00:00:00 +0000

Introduction

Welcome to the “RKE2 the Hard Way” training series! In this series, we’ll embark on a journey to build a Kubernetes cluster from scratch—mimicking the robust features of Rancher Kubernetes Engine 2 (RKE2)—without relying on any distribution-specific tools or installers.

It’s called the “hard way” because we’ll manually configure and set up each component of the Kubernetes cluster. This approach is designed to provide a deep understanding of how Kubernetes works under the hood and to highlight the automation and simplification RKE2 provides.

RKE2 the Hard Way: Part 10 – Installing Ingress Nginx

Tue, 01 Apr 2025 00:00:00 +0000

Part 10 – Installing Ingress Nginx

In this part of the “RKE2 the Hard Way” training series, we will install and configure Ingress Nginx for external access to services in our Kubernetes cluster. Ingress Nginx is a popular ingress controller for Kubernetes that uses NGINX as a reverse proxy and load balancer.

An ingress controller is essential because it:

Provides a single entry point to multiple services in your cluster
Handles SSL/TLS termination for your services
Enables host-based and path-based routing to backend services
Allows you to expose applications to the outside world without creating a LoadBalancer service for each one

At this point in our series, we have:

RKE2 the Hard Way: Part 11 – Cluster Verification and Access

Tue, 01 Apr 2025 00:00:00 +0000

Part 11 – Cluster Verification and Access

Congratulations! You’ve reached the final part of the “RKE2 the Hard Way” training series. At this point, we have:

Set up containerd and kubelet on all nodes
Generated all the necessary TLS certificates
Deployed etcd, kube-apiserver, kube-controller-manager, and kube-scheduler as static pods
Configured kubelet and kube-proxy on worker nodes
Installed Cilium CNI for pod networking
Deployed CoreDNS for cluster DNS resolution
Installed Ingress Nginx for external access

Now, let’s verify that everything is working correctly and set up kubectl for convenient cluster access.

RKE2 the Hard Way: Part 2 – Certificate Authority and TLS Certificates

Tue, 01 Apr 2025 00:00:00 +0000

Part 2 – Certificate Authority and TLS Certificates

In this part of the “RKE2 the Hard Way” training series, we will set up our own Certificate Authority (CA) and generate the necessary TLS certificates for securing our Kubernetes cluster. Kubernetes components communicate over TLS, and properly configured certificates are crucial for cluster security.

We’ll use cfssl, Cloudflare’s TLS toolkit, to create our CA and generate certificates.

✅ Assumption: All commands in this guide are run from node01, which is a Linux box.

RKE2 the Hard Way: Part 3 – Setting up containerd and kubelet

Tue, 01 Apr 2025 00:00:00 +0000

Part 3 – Setting up containerd and kubelet

In this part of the “RKE2 the Hard Way” training series, we will install and configure containerd (the container runtime) and kubelet (the Kubernetes node agent) on all our nodes. These components form the foundation of our Kubernetes cluster.

One of the key design decisions in RKE2 is to run the Kubernetes control plane components (etcd, kube-apiserver, kube-controller-manager, and kube-scheduler) as static pods managed by kubelet, rather than as systemd services. This approach simplifies deployment and maintenance, as kubelet manages the lifecycle of these critical components.

RKE2 the Hard Way: Part 4 – Setting up etcd Cluster as Static Pods

Tue, 01 Apr 2025 00:00:00 +0000

Part 4 – Setting up etcd Cluster as Static Pods

In this part of the “RKE2 the Hard Way” training series, we will set up etcd as a clustered key-value store using static pods managed by kubelet. etcd is a consistent and highly-available key-value store used by Kubernetes to store all cluster data, including the state of the cluster, configuration data, and metadata.

In RKE2, etcd is deployed as a static pod managed by kubelet, instead of running directly as a systemd service. This makes it easier to manage, as kubelet handles lifecycle operations, restarts, and monitoring.

RKE2 the Hard Way: Part 5 – Setting up kube-apiserver as Static Pods

Tue, 01 Apr 2025 00:00:00 +0000

Part 5 – Setting up kube-apiserver as Static Pods

In this part of the “RKE2 the Hard Way” training series, we will set up the Kubernetes API Server (kube-apiserver) as static pods managed by kubelet on each of our nodes. The API server is the central component of the Kubernetes control plane that exposes the Kubernetes API.

Using static pods for the API server is a key design principle in RKE2. This approach simplifies deployment and management, as kubelet will automatically start the API server and ensure it stays running. If an API server instance fails, kubelet will automatically restart it.

RKE2 the Hard Way: Part 6 – Setting up kube-controller-manager and kube-scheduler as Static Pods

Tue, 01 Apr 2025 00:00:00 +0000

Part 6 – Setting up kube-controller-manager and kube-scheduler as Static Pods

In this part of the “RKE2 the Hard Way” training series, we will set up the Kubernetes Controller Manager (kube-controller-manager) and Scheduler (kube-scheduler) as static pods managed by kubelet on each of our nodes.

The Controller Manager runs controller processes that regulate the state of the cluster (such as replication controllers, node controllers, and service account controllers)
The Scheduler is responsible for watching for newly created pods and assigning them to nodes

Using static pods for these components is a key design principle in RKE2, just like we did for etcd and kube-apiserver. This approach ensures high availability and automatic recovery in case of failures.

RKE2 the Hard Way: Part 7 – Setting up kubelet and kube-proxy on Worker Nodes

Tue, 01 Apr 2025 00:00:00 +0000

Part 7 – Setting up kubelet and kube-proxy on Worker Nodes

In this part of the “RKE2 the Hard Way” training series, we will configure kubelet to register with the Kubernetes API server and set up kube-proxy on our nodes. We already installed containerd and kubelet in Part 3, but now we need to finalize their configuration to connect to our Kubernetes control plane.

kubelet is the primary node agent that runs on each node. It ensures containers are running in a Pod and healthy.
kube-proxy maintains network rules on nodes. These network rules allow network communication to your Pods from inside or outside of your cluster.

✅ Note: In our setup, all nodes (node01, node02, node03) are both control plane and worker nodes. This is common for smaller clusters and provides high availability without needing separate worker nodes.

RKE2 the Hard Way: Part 8 – Installing Cilium CNI

Tue, 01 Apr 2025 00:00:00 +0000

Part 8 – Installing Cilium CNI

In this part of the “RKE2 the Hard Way” training series, we will install and configure the Cilium CNI (Container Network Interface). Cilium is a powerful, open-source CNI solution that provides advanced networking, security, and observability features for Kubernetes clusters.

At this point in our tutorial series, we have:

Set up all the core Kubernetes components (etcd, kube-apiserver, kube-controller-manager, kube-scheduler)
Configured kubelet and kube-proxy on our nodes

However, our cluster network is not yet functional for pod-to-pod communication. The CNI plugin is essential for implementing the Kubernetes networking model, which requires that:

RKE2 the Hard Way: Part 9 – Installing CoreDNS

Tue, 01 Apr 2025 00:00:00 +0000

Part 9 – Installing CoreDNS

In this part of the “RKE2 the Hard Way” training series, we will install and configure CoreDNS for Kubernetes cluster DNS resolution. CoreDNS is a flexible, extensible DNS server that serves as the cluster DNS provider in Kubernetes, allowing service discovery by DNS name.

DNS resolution is critical in a Kubernetes cluster because:

It enables services to be discovered by their names rather than IP addresses
It allows pods to find and communicate with other pods and services
It provides a stable naming scheme even when IPs change due to pod rescheduling

✅ Note: We already configured kubelet in Part 7 to use 10.43.0.10 as the DNS server, which is the IP we’ll use for the CoreDNS service.

Setting Up Equinix Metal Cloud Provider for RKE2

Wed, 29 Jan 2025 00:00:00 +0000

Introduction

Equinix Metal (formerly Packet) provides bare-metal cloud infrastructure for running Kubernetes. RKE2 supports Equinix Metal as an out-of-tree cloud provider, enabling integration with MetalLB for LoadBalancer services and CSI drivers for persistent storage.

This guide walks you through setting up the Equinix Metal Cloud Provider in RKE2.

Prerequisites

An active Equinix Metal account
A running RKE2 cluster deployed on Equinix Metal
Equinix Metal API Key for authentication
kubectl configured to interact with your cluster
Helm installed for deploying the cloud provider

Step 1: Generate Equinix Metal API Key

Log in to the Equinix Metal Console.
Navigate to API Keys.
Generate a read-write API key for Kubernetes integration.
Store this key securely, as it will be required for configuring the cloud provider.

Step 2: Configure the Cloud Provider

The Equinix Metal Cloud Provider requires a configuration file (cloud-provider-config) to be stored as a Kubernetes Secret.

Ingress-NGINX for RKE2: Overview, Troubleshooting, and Common Configurations

Wed, 22 Jan 2025 00:00:00 -0500

RKE2’s Ingress-NGINX controller is a powerful solution for managing incoming HTTP and HTTPS traffic in Kubernetes clusters. This guide explores its architecture, troubleshooting techniques, common configurations, and practical examples.

KubeBackup: Automating Kubernetes Cluster Backups

Sat, 19 Oct 2024 19:26:00 -0500

Backing up your Kubernetes cluster is critical for disaster recovery and operational continuity. KubeBackup is a tool designed to simplify cluster backups by exporting YAML configurations and uploading them to an S3 bucket. With KubeBackup, you can automate backups, ensuring you have all necessary data to redeploy or restore your cluster quickly.

Fixing Argo CD Out-of-Sync Due to Cilium Identity on RKE2

Tue, 15 Oct 2024 11:30:00 -0500

Fixing Argo CD Out-of-Sync Due to Cilium Identity on RKE2

If you’re running a Kubernetes cluster with RKE2 and Cilium as your networking solution, you might encounter a situation where Argo CD shows an out-of-sync status due to the presence of CiliumIdentity resources. Although the actual application is in sync and functioning normally, this false out-of-sync status can be frustrating to see in your CI/CD workflows.

Monitor Etcd Cluster with Grafana and Prometheus in RKE2

Tue, 17 Sep 2024 19:26:00 -0500

Monitoring your Etcd cluster is critical to ensure the health of your Kubernetes infrastructure, especially in an RKE2 environment. In this post, we’ll explore how to monitor an Etcd cluster using Grafana and Prometheus, giving you real-time insights and alerting capabilities to prevent issues before they impact your cluster’s availability.

Configuring Kube-apiserver Audit Logs in RKE2

Wed, 11 Sep 2024 03:26:00 -0500

Audit logs in Kubernetes are vital in monitoring and securing your cluster. They help you track API requests, identify configuration changes, and trace actions to their source. In RKE2 (Rancher Kubernetes Engine 2), you can configure kube-apiserver audit logs to capture high-level and detailed information about what happens in your cluster.

In this post, we’ll walk through how to set up basic and detailed audit logging for the kube-apiserver in RKE2, providing examples to help you monitor API activities effectively.

Running RKE2 Clusters on-top Harvester Using the Cloud Provider and Rancher

Wed, 14 Feb 2024 10:00:00 -0500

Learn how to seamlessly integrate RKE2 clusters with Harvester’s infrastructure capabilities using the Harvester cloud provider. This workshop guide covers the entire process from initial setup to cluster deployment and configuration.

Monitoring RKE2 Certs with x509-certificate-exporter

Thu, 08 Feb 2024 03:24:00 -0500

Monitoring SSL/TLS certificate validity is crucial to maintaining the security and reliability of any Kubernetes cluster. For Rancher Kubernetes Engine (RKE2) clusters, tracking certificate expiration helps prevent unexpected outages and ensures your applications run smoothly.

In this post, we’ll guide you through deploying the x509-certificate-exporter using Helm to monitor your RKE2 certificates effectively. We’ll also demonstrate how to monitor the expiration of TLS secrets stored as Kubernetes secrets, including cert-manager and imported certificates.

Rancher Monitoring v2 Workshop: Comprehensive Guide

Wed, 10 Jan 2024 09:00:00 -0500

Rancher Monitoring v2 Workshop

Welcome to the Rancher Monitoring v2 Workshop! This workshop is designed to provide a comprehensive guide to deploying, managing, and troubleshooting Rancher Monitoring v2 in Kubernetes environments.

Overview

Rancher Monitoring v2 offers advanced monitoring capabilities for Kubernetes clusters managed by Rancher. It is a powerful tool that integrates Prometheus, Grafana, and other monitoring components to provide deep insights into the health and performance of clusters.

Goals of the Workshop

Understanding the architecture of Rancher Monitoring v2
Learning how to deploy and configure Rancher Monitoring v2
Best practices for monitoring Kubernetes clusters
Troubleshooting common issues in Rancher Monitoring v2
Hands-on experience with Rancher Monitoring v2
Setting up alerts and notifications
Setting up ingress for Grafana and Prometheus
Setting up Grafana dashboards

RKE2 with Rancher - Disaster Recovery Guide

Wed, 18 Oct 2023 01:15:00 -0600

RKE2 with Rancher - Disaster Recovery Guide

Introduction

The purpose of this guide is to provide a comprehensive guide for disaster recovery of RKE2 clusters managed by Rancher. RKE2, a CNCF-certified Kubernetes distribution, is a fully conformant Kubernetes distribution focusing on security and simplicity. Rancher is a CNCF-certified Kubernetes management platform that provides a single pane of glass for managing multiple Kubernetes clusters. Rancher can manage RKE2 clusters and other Kubernetes distributions such as RKE, EKS, AKS, GKE, and more.

Please note that this guide is intended to be something other than a replacement for the official documentation. This guide is designed to supplement the official documentation and provide a comprehensive guide for disaster recovery of RKE2 clusters that Rancher manages. This guide also assumes a basic understanding of Kubernetes, RKE2, and Rancher.

RKE2 Standalone - Disaster Recovery Guide

Wed, 11 Oct 2023 00:00:00 -0600

RKE2 Standalone - Disaster Recovery Guide

Introduction

The purpose of this guide is to provide comprehensive instructions for recovering a standalone RKE2 cluster that Rancher does not manage. RKE2, a lightweight Kubernetes distribution, is a robust solution, but disasters can happen. In this guide, we’ll address common scenarios like recovering after etcd quorum loss, restoring a cluster from a backup, and troubleshooting issues when nodes cannot join the cluster.

Please note that this guide assumes a basic familiarity with RKE2 and Kubernetes concepts. If you encounter any challenges or have questions, please ask for assistance.

RKE2: A Deep Dive into CNI options

Tue, 08 Nov 2022 01:30:00 -0500

RKE2’s CNI Options: Canal, Cilium, Calico, and Multus
Conclusion

RKE2: A Deep Dive into Rancher's Kubernetes Distribution

Mon, 24 Oct 2022 21:01:00 -0500

service rancher-monitoring-operator not found

Thu, 29 Sep 2022 00:51:00 -0500

Error:

Error: Internal error occurred: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://rancher-monitoring-operator.cattle-monitoring-system.svc:443/admission-prometheusrules/mutate?timeout=10s": service "rancher-monitoring-operator" not found

This error can be caused by a number of different issues. But first we need to understand what this error means.

The TLDR is that the webhook is not available. The webhook is a service that is used to validate and mutate resources. In this case, the webhook is used to validate and mutate PrometheusRule resources.

Harbor Redis Crashlooping

Tue, 12 Jul 2022 18:21:00 -0500

My Harbor server went offline when I patched one of my RKE2 clusters, which caused pods to go into ImagePullBackoff. Within a few minutes, I noticed that Redis was crashlooping with the following error:

bash
mmattox@a1ubthorp01:~/$ kubectl -n harbor logs -f harbor-redis-0
1:C 12 Jul 2022 23:15:36.208 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 12 Jul 2022 23:15:36.208 # Redis version=6.0.16, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 12 Jul 2022 23:15:36.208 # Configuration loaded
 _._ 
 _.-``__ ''-._ 
 _.-`` `. `_. ''-._ Redis 6.0.16 (00000000/0) 64 bit
 .-`` .-```. ```\/ _.,_ ''-._ 
 ( ' , .-` | `, ) Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
 | `-._ `._ / _.-' | PID: 1
 `-._ `-._ `-./ _.-' _.-' 
 |`-._`-._ `-.__.-' _.-'_.-'| 
 | `-._`-._ _.-'_.-' | http://redis.io 
 `-._ `-._`-.__.-'_.-' _.-' 
 |`-._`-._ `-.__.-' _.-'_.-'| 
 | `-._`-._ _.-'_.-' | 
 `-._ `-._`-.__.-'_.-' _.-' 
 `-._ `-.__.-' _.-' 
 `-._ _.-' 
 `-.__.-' 

1:M 12 Jul 2022 23:15:36.210 # Server initialized
1:M 12 Jul 2022 23:15:36.210 # Can't handle RDB format version 10
1:M 12 Jul 2022 23:15:36.210 # Fatal error loading the DB: Invalid argument. Exiting.
mmattox@a1ubthorp01:~/$

How to install system-upgrade-controller for RKE2

Fri, 24 Jun 2022 01:20:00 -0500

One of the core ideas with RKE2 is that the cluster should be able to manage itself. This means that the cluster should be able to upgrade itself. RKE2 uses a tool called system-upgrade-controller to do this.

kernel: rc rc0: receive overflow

Thu, 23 Jun 2022 23:52:00 -0500

After upgrade my build nodes from Ubuntu 20.04 to Ubuntu 22.04, I got a lot of spamming syslog messages on my Intel NUC nodes, model NUC5i5RYB. Both these nodes are running RKE2 v1.24.1+rke2r2. I have them tainted so that they are only used by Drone CI to run Kubernetes runners.

Error message

Jun 24 04:35:27 a0ubnucp02 kernel: [ 2025.557663] rc rc0: receive overflow
Jun 24 04:35:27 a0ubnucp02 kernel: [ 2025.561258] rc rc0: receive overflow
Jun 24 04:35:27 a0ubnucp02 kernel: [ 2025.566120] rc rc0: receive overflow
Jun 24 04:35:27 a0ubnucp02 kernel: [ 2025.569739] rc rc0: receive overflow
Jun 24 04:35:27 a0ubnucp02 kernel: [ 2025.618179] rc rc0: receive overflow
Jun 24 04:35:27 a0ubnucp02 kernel: [ 2025.637622] rc rc0: receive overflow
Jun 24 04:35:27 a0ubnucp02 kernel: [ 2025.645784] rc rc0: receive overflow
Jun 24 04:35:27 a0ubnucp02 kernel: [ 2025.649311] rc rc0: receive overflow
Jun 24 04:35:27 a0ubnucp02 kernel: [ 2025.656808] rc rc0: receive overflow
Jun 24 04:35:27 a0ubnucp02 kernel: [ 2025.660897] rc rc0: receive overflow

Rke2 on Support Tools

Ingress-NGINX Retirement on RKE2: Migrating to Traefik Without Downtime

How to Recover Longhorn Volume Data from a Single Replica in RKE2 Kubernetes

Running RKE2 Ingress-NGINX with an External LoadBalancer

RKE2 the Hard Way: Part 1 – Introduction and Prerequisites for Building a Kubernetes Cluster

Introduction

RKE2 the Hard Way: Part 10 – Installing Ingress Nginx

Part 10 – Installing Ingress Nginx

RKE2 the Hard Way: Part 11 – Cluster Verification and Access

Part 11 – Cluster Verification and Access

RKE2 the Hard Way: Part 2 – Certificate Authority and TLS Certificates

Part 2 – Certificate Authority and TLS Certificates

RKE2 the Hard Way: Part 3 – Setting up containerd and kubelet

Part 3 – Setting up containerd and kubelet

RKE2 the Hard Way: Part 4 – Setting up etcd Cluster as Static Pods

Part 4 – Setting up etcd Cluster as Static Pods

RKE2 the Hard Way: Part 5 – Setting up kube-apiserver as Static Pods

Part 5 – Setting up kube-apiserver as Static Pods

RKE2 the Hard Way: Part 6 – Setting up kube-controller-manager and kube-scheduler as Static Pods

Part 6 – Setting up kube-controller-manager and kube-scheduler as Static Pods

RKE2 the Hard Way: Part 7 – Setting up kubelet and kube-proxy on Worker Nodes

Part 7 – Setting up kubelet and kube-proxy on Worker Nodes

RKE2 the Hard Way: Part 8 – Installing Cilium CNI

Part 8 – Installing Cilium CNI

RKE2 the Hard Way: Part 9 – Installing CoreDNS

Part 9 – Installing CoreDNS

Setting Up Equinix Metal Cloud Provider for RKE2

Introduction

Prerequisites

Step 1: Generate Equinix Metal API Key

Step 2: Configure the Cloud Provider

Ingress-NGINX for RKE2: Overview, Troubleshooting, and Common Configurations

KubeBackup: Automating Kubernetes Cluster Backups

Fixing Argo CD Out-of-Sync Due to Cilium Identity on RKE2

Fixing Argo CD Out-of-Sync Due to Cilium Identity on RKE2

Monitor Etcd Cluster with Grafana and Prometheus in RKE2

Configuring Kube-apiserver Audit Logs in RKE2

Running RKE2 Clusters on-top Harvester Using the Cloud Provider and Rancher

Monitoring RKE2 Certs with x509-certificate-exporter

Rancher Monitoring v2 Workshop: Comprehensive Guide

Rancher Monitoring v2 Workshop

Overview

Goals of the Workshop

RKE2 with Rancher - Disaster Recovery Guide

RKE2 with Rancher - Disaster Recovery Guide

Introduction

RKE2 Standalone - Disaster Recovery Guide

RKE2 Standalone - Disaster Recovery Guide

Introduction

RKE2: A Deep Dive into CNI options

Table of Contents

RKE2: A Deep Dive into Rancher's Kubernetes Distribution

service rancher-monitoring-operator not found

Harbor Redis Crashlooping

How to install system-upgrade-controller for RKE2

kernel: rc rc0: receive overflow