Production Incidents
Reddit Kubernetes Cluster Upgrade Failure: When Updates Take Down Production
19 min readOn March 14, 2023, Reddit experienced a multi-hour production outage affecting their entire platform during what should have been a routine Kubernetes cluster upgrade. The incident …
Kubernetes Pod Startup Failures: The Too Many Services Incident
15 min readAt 3:47 AM on a Tuesday morning, our production monitoring system lit up with critical alerts: dozens of pods across multiple namespaces were failing to start. The error message …
Honeycomb Certificate Expiry Outage: 2.5-Hour Production Incident Analysis
20 min readOn April 27, 2022, Honeycomb experienced a 2.5-hour production outage that affected their entire Kubernetes infrastructure. The root cause was deceptively simple: expired …