“No One Has Ever Gotten Fired for Having Too Many Backups”

When running Rancher or RKE2, there’s one mantra to live by: “Always back it up before making a change.” Backups are not just about recovering from disasters—they’re about giving you the confidence to experiment, upgrade, and troubleshoot without fear. This post covers essential strategies for Kubernetes backups, with practical advice and links to tools and methods to make backup processes effortless.

For deep dives into specific tools, check out our related posts:

Why Kubernetes Backups are Critical

In dynamic environments like Rancher or RKE2 clusters, backups are essential for several reasons:

  • Recover from Failures: Accidental deletions, misconfigurations, or system failures are common in large deployments.
  • Disaster Recovery: Ensures quick recovery from hardware failures, node outages, or security incidents.
  • Simplifies Upgrades: Upgrades are smoother when you know you have working backups in place.
  • Experiment Safely: With reliable backups, you can make configuration changes without fear.

Etcd Backups: Quick and Simple

Etcd stores the state of your entire Kubernetes cluster, including workloads, configurations, and networking details. Thankfully, taking an etcd snapshot is quick and easy:

rke2 etcd-snapshot save --name snapshot-$(date +%Y%m%d-%H%M%S)

When making any major change, such as upgrading the control plane or modifying critical resources, always take a snapshot first. Remember: “Better safe than sorry.”

Restoring an etcd snapshot is just as straightforward:

rke2 etcd-snapshot restore --name snapshot-YYYYMMDD-HHMMSS

For more on working with etcd snapshots, see our post on backup tools.

YAML Exports: Your Best Friend Before Any Change

Another essential practice is exporting Kubernetes objects to YAML before editing them. With this approach, you can easily roll back if something goes wrong.

kubectl get deployment my-app -n my-namespace -o yaml > my-app-backup.yaml

If you need to revert to the previous version:

kubectl apply -f my-app-backup.yaml

This small habit will save you from countless headaches when making on-the-fly changes. For a deeper dive into YAML-based backups, check out my post on KubeBackup.

Automating Backups with Tools

To ensure backups are consistent and automated, use tools like Velero, Longhorn, and KubeBackup:

  • Velero:
    Great for application-level backups and cluster migrations. Works well with S3-compatible storage.

  • Longhorn:
    A distributed block storage system that excels at PVC snapshots and volume backups.

  • KubeBackup:
    Automates YAML exports of Kubernetes resources and uploads them to S3. Ideal for maintaining configuration backups over time.

  • Kasten K10:
    A data management platform that provides backup, disaster recovery, and mobility for Kubernetes applications.

  • Rancher Backup Operator:
    A Rancher-native backup solution that automates backups of Rancher.

With these tools in place, you can automate backups to run on schedules, offload them to S3 or cloud storage, and rest easy knowing your cluster is safe.

Backup Best Practices

  1. Backup Before Every Change:
    Whether it’s upgrading Rancher, changing cluster configurations, or modifying resources—always take a backup first.

  2. Use Multiple Backup Methods:
    Combine etcd snapshots, YAML exports, and Velero/Longhorn backups to ensure you’re covered on all fronts. Remember, “No one has ever gotten fired for having too many backups.”

  3. Offsite Storage:
    Always upload backups to an external location, such as S3, to protect against hardware failures or catastrophic events.

  4. Test Your Restores:
    A backup is only as good as your ability to restore it. Regularly test your restore processes to avoid surprises during real emergencies.

  5. Monitor Your Backups:
    Use Prometheus and AlertManager to monitor backup health and ensure your backups are being created on schedule.

Conclusion

Running Kubernetes with Rancher or RKE2 requires you to be proactive about backups. Whether it’s an etcd snapshot, a YAML export, or an automated backup using Velero, Longhorn, or KubeBackup, the key is to always have a backup before making a change. With these strategies and tools in place, you can ensure smooth operations, quick recoveries, and peace of mind.

To learn more about specific tools and backup methods, check out our other posts:

By following these best practices, you’ll be well-prepared for anything that comes your way—because “better safe than sorry” should always be your motto.