Upgrading a Kubernetes cluster is essential for maintaining security, gaining access to new features, and ensuring compatibility with modern workloads. However, it requires meticulous planning and execution to avoid disruptions. This guide provides best practices and considerations for upgrading Kubernetes clusters, whether self-managed or hosted.

Upgrading Kubernetes Versions

Planning and Preparation

A successful Kubernetes upgrade starts with careful planning and thorough preparation.

1. Review Release Notes and Change Logs

Understand what’s new, deprecated, or removed in the target version. Check the official Kubernetes release notes for:

  • API changes and deprecations.
  • New features and enhancements.
  • Known issues or breaking changes.

2. Perform Compatibility Checks

Ensure compatibility between the new Kubernetes version and the following components:

  • Add-ons (e.g., CoreDNS, kube-proxy, VPC CNI).
  • Custom applications and workloads.
  • Third-party tools (e.g., monitoring, logging, storage drivers).

For Amazon EKS users, detailed add-on compatibility information is available in AWS documentation.

3. Test in Non-Production Environments

Simulate the upgrade process in a staging or test cluster to identify and resolve issues. Use continuous integration workflows to validate application behavior against the new version.

Upgrade Approaches

The upgrade process varies depending on how the Kubernetes cluster is deployed.

1. Using kubeadm

For clusters deployed with kubeadm, follow the official kubeadm upgrade guide. Steps include:

Upgrade Control Plane:

# Upgrade kubeadm
sudo apt-mark unhold kubeadm
sudo apt-get update
sudo apt-get install -y kubeadm=1.29.x-

# Apply the upgrade
sudo kubeadm upgrade apply v1.29.x

Upgrade Worker Nodes:

# Drain the node
kubectl drain <node-name> --force --ignore-daemonsets --delete-emptydir-data

# Upgrade kubelet
sudo apt-get update
sudo apt-get install -y kubelet=1.29.x-

# Uncordon the node
kubectl uncordon <node-name>

2. Manually Deployed Clusters

For manually configured clusters, upgrade control plane components in the following order:

  1. etcd
  2. kube-apiserver
  3. kube-controller-manager
  4. kube-scheduler
  5. cloud-controller-manager (if used)

Upgrade worker nodes one at a time, draining, updating, and uncordoning each.

3. Amazon EKS Clusters

For Amazon EKS, upgrades can be performed using eksctl or AWS CLI:

# Upgrade with eksctl
eksctl upgrade cluster --name my-cluster --version 1.30 --approve

# Upgrade with AWS CLI
aws eks update-cluster-version --region <region> --name my-cluster --kubernetes-version 1.30

Ensure worker nodes are updated to match the control plane version.

Post-Upgrade Tasks

After completing the upgrade, additional steps are necessary to ensure cluster functionality and performance.

1. Update Cluster Components

Ensure all nodes are running the same Kubernetes version as the control plane. For EKS, update managed and self-managed node groups.

2. Update Storage API Versions

If the storage API version has changed, rewrite objects in the new API format:

kubectl get <resource> -o yaml | kubectl apply -f -

3. Convert Manifests

Update YAML manifests to the latest supported API versions:

kubectl convert -f pod.yaml --output-version v1

4. Update Device Plugins

If device plugins are used, update them to support both the old and new Kubernetes versions before upgrading nodes.

Monitoring and Disaster Recovery

Monitor the Upgrade

Use tools like Prometheus and Grafana to track cluster health and performance during and after the upgrade. Look for any abnormalities in logs, resource utilization, or API server metrics.

Disaster Recovery Planning

Have a well-documented recovery plan in place before initiating the upgrade.

  • Create detailed upgrade runbooks.
  • Regularly test your recovery procedures to ensure they are effective.

Compatibility and Deprecation

Understand Deprecation Policies

Kubernetes’ deprecation policy ensures stability:

  • GA APIs are not removed until a major version change.
  • Beta APIs are supported for at least three minor versions after deprecation.

Assess Add-On Compatibility

Update or replace cluster add-ons to ensure compatibility with the new Kubernetes version. For managed services like EKS, these updates are often manual.

Upgrade Strategies

1. Incremental Upgrades

Upgrade one minor version at a time to ensure stability and compatibility. This approach requires frequent upgrades but minimizes risk.

2. Long Jumps

Skip multiple minor versions to reduce the frequency of upgrades. While efficient, this approach requires rigorous testing due to potential breaking changes.

Conclusion

Upgrading a Kubernetes cluster is a complex yet essential task that demands careful planning, testing, and execution. By following best practices—such as reviewing release notes, validating compatibility, and monitoring the process—you can minimize disruptions and maintain a robust Kubernetes environment.

Regular upgrades not only enhance security but also enable access to new features, aligning your cluster with the latest technological advancements. Always maintain a comprehensive disaster recovery plan to handle unexpected issues effectively.

For more Kubernetes insights and platform engineering tips, visit Platform Engineers Blog.