Game Day Planning and Execution: Building Resilient Teams Through Practice
Game days are essential for building resilient teams and validating incident response procedures. This comprehensive guide covers planning, executing, and analyzing game day exercises to improve team readiness and system reliability.
Executive Summary
Game days simulate production incidents in a controlled environment, allowing teams to practice response procedures, identify gaps, and build muscle memory for handling real emergencies. This guide provides enterprise teams with a complete framework for designing effective game day exercises, from scenario creation to post-game analysis, with production-ready tools and best practices for maximizing learning outcomes.
Understanding Game Day Fundamentals
Game Day Philosophy
# game-day-principles.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: game-day-principles
namespace: sre
data:
principles.md: |
# Game Day Core Principles
## 1. Safety First
- Never risk production systems
- Have clear abort criteria
- Ensure proper safeguards
- Maintain business continuity
## 2. Realistic Scenarios
- Based on actual incidents
- Reflect system complexity
- Include time pressure
- Incorporate communication challenges
## 3. Team Learning
- Focus on process, not blame
- Encourage experimentation
- Document learnings
- Celebrate improvements
## 4. Continuous Improvement
- Update runbooks
- Fix identified gaps
- Iterate on scenarios
- Measure progress
## 5. Cross-functional Collaboration
- Include all relevant teams
- Break down silos
- Build relationships
- Share knowledge
objectives.md: |
# Game Day Objectives
## Primary Goals
1. Validate incident response procedures
2. Build team confidence and skills
3. Identify system weaknesses
4. Improve communication patterns
5. Test monitoring and alerting
6. Verify documentation accuracy
## Success Metrics
- Mean time to detection (MTTD)
- Mean time to resolution (MTTR)
- Team coordination effectiveness
- Runbook accuracy and completeness
- Tool effectiveness
- Communication clarity
Game Day Types and Scopes
# game_day_framework.py
from dataclasses import dataclass
from typing import List, Dict, Optional, Set
from enum import Enum
from datetime import datetime, timedelta
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
class GameDayType(Enum):
TABLETOP = "tabletop" # Discussion-based
SIMULATION = "simulation" # Controlled environment
CHAOS = "chaos" # Production-like chaos
FULL_SCALE = "full_scale" # Full production test
SURPRISE = "surprise" # Unannounced drill
class GameDayScope(Enum):
COMPONENT = "component" # Single service/component
SERVICE = "service" # Complete service
SYSTEM = "system" # Multiple services
ORGANIZATION = "organization" # Entire organization
class Difficulty(Enum):
BEGINNER = 1
INTERMEDIATE = 2
ADVANCED = 3
EXPERT = 4
@dataclass
class Participant:
name: str
role: str
team: str
skills: Set[str]
availability: Dict[str, bool]
@dataclass
class GameDayScenario:
id: str
name: str
description: str
type: GameDayType
scope: GameDayScope
difficulty: Difficulty
estimated_duration: timedelta
required_roles: List[str]
required_teams: List[str]
prerequisites: List[str]
learning_objectives: List[str]
success_criteria: List[str]
failure_injection_methods: List[str]
monitoring_requirements: List[str]
communication_channels: List[str]
class GameDayPlanner:
"""Plans and schedules game day exercises"""
def __init__(self):
self.scenarios: Dict[str, GameDayScenario] = {}
self.participants: Dict[str, Participant] = {}
self.scheduled_gamedays: List[Dict] = []
def create_scenario(
self,
name: str,
description: str,
game_type: GameDayType,
scope: GameDayScope,
difficulty: Difficulty
) -> GameDayScenario:
"""Create a new game day scenario"""
scenario_id = f"gameday-{name.lower().replace(' ', '-')}"
scenario = GameDayScenario(
id=scenario_id,
name=name,
description=description,
type=game_type,
scope=scope,
difficulty=difficulty,
estimated_duration=timedelta(hours=2),
required_roles=[],
required_teams=[],
prerequisites=[],
learning_objectives=[],
success_criteria=[],
failure_injection_methods=[],
monitoring_requirements=[],
communication_channels=[]
)
self.scenarios[scenario_id] = scenario
logger.info(f"Created scenario: {name}")
return scenario
def add_learning_objective(
self,
scenario_id: str,
objective: str
):
"""Add a learning objective to scenario"""
if scenario_id in self.scenarios:
self.scenarios[scenario_id].learning_objectives.append(objective)
def add_success_criteria(
self,
scenario_id: str,
criteria: str
):
"""Add success criteria to scenario"""
if scenario_id in self.scenarios:
self.scenarios[scenario_id].success_criteria.append(criteria)
def register_participant(
self,
name: str,
role: str,
team: str,
skills: Set[str]
) -> Participant:
"""Register a game day participant"""
participant = Participant(
name=name,
role=role,
team=team,
skills=skills,
availability={}
)
self.participants[name] = participant
return participant
def schedule_game_day(
self,
scenario_id: str,
scheduled_time: datetime,
facilitator: str,
participants: List[str]
) -> Dict:
"""Schedule a game day exercise"""
if scenario_id not in self.scenarios:
raise ValueError(f"Scenario {scenario_id} not found")
scenario = self.scenarios[scenario_id]
# Validate participant availability
unavailable = []
for participant_name in participants:
if participant_name not in self.participants:
unavailable.append(participant_name)
if unavailable:
logger.warning(f"Participants not registered: {unavailable}")
game_day = {
'id': f"gd-{scheduled_time.strftime('%Y%m%d-%H%M')}",
'scenario_id': scenario_id,
'scenario_name': scenario.name,
'scheduled_time': scheduled_time,
'estimated_end': scheduled_time + scenario.estimated_duration,
'facilitator': facilitator,
'participants': participants,
'status': 'scheduled',
'preparation_checklist': self._generate_preparation_checklist(scenario),
'communication_plan': self._generate_communication_plan(scenario)
}
self.scheduled_gamedays.append(game_day)
logger.info(f"Scheduled game day: {scenario.name} at {scheduled_time}")
return game_day
def _generate_preparation_checklist(
self,
scenario: GameDayScenario
) -> List[Dict]:
"""Generate preparation checklist"""
checklist = [
{
'item': 'Review scenario objectives and success criteria',
'deadline': 'T-72h',
'responsible': 'Facilitator'
},
{
'item': 'Send calendar invites to all participants',
'deadline': 'T-168h',
'responsible': 'Facilitator'
},
{
'item': 'Prepare failure injection scripts',
'deadline': 'T-48h',
'responsible': 'Technical Lead'
},
{
'item': 'Set up monitoring dashboards',
'deadline': 'T-24h',
'responsible': 'SRE Team'
},
{
'item': 'Configure communication channels',
'deadline': 'T-24h',
'responsible': 'Facilitator'
},
{
'item': 'Review and update runbooks',
'deadline': 'T-48h',
'responsible': 'On-Call Team'
},
{
'item': 'Test rollback procedures',
'deadline': 'T-24h',
'responsible': 'Technical Lead'
},
{
'item': 'Brief stakeholders',
'deadline': 'T-24h',
'responsible': 'Facilitator'
},
{
'item': 'Conduct pre-game checkin',
'deadline': 'T-1h',
'responsible': 'Facilitator'
}
]
# Add scenario-specific items
if scenario.type == GameDayType.CHAOS:
checklist.append({
'item': 'Verify chaos engineering tools are ready',
'deadline': 'T-24h',
'responsible': 'SRE Team'
})
if scenario.scope in [GameDayScope.SYSTEM, GameDayScope.ORGANIZATION]:
checklist.append({
'item': 'Notify customer support team',
'deadline': 'T-24h',
'responsible': 'Facilitator'
})
return checklist
def _generate_communication_plan(
self,
scenario: GameDayScenario
) -> Dict:
"""Generate communication plan"""
return {
'primary_channel': '#game-day-war-room',
'video_conference': 'https://meet.company.com/gameday',
'documentation': 'https://wiki.company.com/gameday',
'notification_timeline': {
'T-168h': 'Initial notification and calendar invite',
'T-72h': 'Reminder with preparation materials',
'T-24h': 'Final reminder and checklist review',
'T-1h': 'Pre-game briefing',
'T+0': 'Game day kickoff',
'T+end': 'Debrief session invitation'
},
'stakeholder_updates': {
'frequency': 'Every 30 minutes during game day',
'channel': '#game-day-updates',
'recipients': ['Engineering Leadership', 'Product Management', 'Customer Support']
},
'escalation_path': [
{'level': 1, 'contact': 'Game Day Facilitator'},
{'level': 2, 'contact': 'SRE Lead'},
{'level': 3, 'contact': 'VP of Engineering'}
]
}
# Example usage
planner = GameDayPlanner()
# Create database failure scenario
db_scenario = planner.create_scenario(
name="Database Primary Failure",
description="Simulate primary database failure and test failover procedures",
game_type=GameDayType.CHAOS,
scope=GameDayScope.SYSTEM,
difficulty=Difficulty.INTERMEDIATE
)
planner.add_learning_objective(
db_scenario.id,
"Validate automated database failover"
)
planner.add_learning_objective(
db_scenario.id,
"Test application connection pool resilience"
)
planner.add_learning_objective(
db_scenario.id,
"Practice incident communication procedures"
)
planner.add_success_criteria(
db_scenario.id,
"Failover completes within 2 minutes"
)
planner.add_success_criteria(
db_scenario.id,
"Zero data loss during failover"
)
planner.add_success_criteria(
db_scenario.id,
"Applications automatically reconnect"
)
# Register participants
planner.register_participant(
name="Alice Smith",
role="SRE Engineer",
team="Platform",
skills={"kubernetes", "databases", "incident-response"}
)
planner.register_participant(
name="Bob Johnson",
role="Backend Engineer",
team="Payments",
skills={"java", "spring-boot", "postgresql"}
)
# Schedule game day
game_day = planner.schedule_game_day(
scenario_id=db_scenario.id,
scheduled_time=datetime(2026, 2, 15, 14, 0),
facilitator="Alice Smith",
participants=["Alice Smith", "Bob Johnson"]
)
Scenario Design and Development
Creating Realistic Scenarios
# scenario_builder.py
from typing import List, Dict, Any, Optional
from dataclasses import dataclass, field
import json
@dataclass
class FailureInjection:
"""Defines how to inject a failure"""
name: str
method: str # "chaos_mesh", "litmus", "manual", "script"
target: str
parameters: Dict[str, Any]
duration: int # seconds
delay: int = 0 # delay before injection
@dataclass
class DetectionPoint:
"""Expected detection mechanism"""
source: str # "monitoring", "alert", "user_report"
expected_time: int # seconds from injection
indicators: List[str]
@dataclass
class ScenarioTimeline:
"""Timeline of events during scenario"""
events: List[Dict[str, Any]] = field(default_factory=list)
def add_event(
self,
time_offset: int,
event_type: str,
description: str,
expected_action: Optional[str] = None
):
"""Add an event to the timeline"""
self.events.append({
'time_offset': time_offset,
'type': event_type,
'description': description,
'expected_action': expected_action
})
class ScenarioBuilder:
"""Builds detailed game day scenarios"""
def __init__(self):
self.scenarios = {}
def build_database_failover_scenario(self) -> Dict:
"""Build comprehensive database failover scenario"""
timeline = ScenarioTimeline()
# T+0: Initial state
timeline.add_event(
time_offset=0,
event_type="start",
description="System operating normally",
expected_action="Establish baseline metrics"
)
# T+5: Inject failure
timeline.add_event(
time_offset=300,
event_type="failure_injection",
description="Primary database becomes unresponsive",
expected_action="Monitoring should detect issue"
)
# T+5:30: Expected detection
timeline.add_event(
time_offset=330,
event_type="detection",
description="Database health check fails",
expected_action="Alert fires, on-call engineer paged"
)
# T+6: Investigation begins
timeline.add_event(
time_offset=360,
event_type="investigation",
description="On-call engineer acknowledges alert",
expected_action="Begin incident response procedures"
)
# T+7: Diagnosis
timeline.add_event(
time_offset=420,
event_type="diagnosis",
description="Primary database failure confirmed",
expected_action="Execute failover runbook"
)
# T+8: Mitigation
timeline.add_event(
time_offset=480,
event_type="mitigation",
description="Failover to secondary database",
expected_action="Verify application connectivity"
)
# T+10: Resolution
timeline.add_event(
time_offset=600,
event_type="resolution",
description="System fully operational on secondary",
expected_action="Update status page, monitor metrics"
)
# T+15: Recovery
timeline.add_event(
time_offset=900,
event_type="recovery",
description="Primary database restored",
expected_action="Plan failback strategy"
)
failure_injection = FailureInjection(
name="database_primary_failure",
method="chaos_mesh",
target="postgresql-primary",
parameters={
"action": "pod-kill",
"namespace": "production",
"mode": "one"
},
duration=600,
delay=300
)
detection_points = [
DetectionPoint(
source="database_health_check",
expected_time=30,
indicators=[
"Connection timeout",
"Health check failures",
"Replication lag increase"
]
),
DetectionPoint(
source="application_errors",
expected_time=45,
indicators=[
"Database connection errors",
"Increased error rate",
"Transaction failures"
]
),
DetectionPoint(
source="monitoring_alert",
expected_time=60,
indicators=[
"DatabaseDown alert",
"HighErrorRate alert",
"ServiceDegraded alert"
]
)
]
scenario = {
'metadata': {
'name': 'Database Primary Failure',
'id': 'db-failover-001',
'version': '1.0',
'author': 'SRE Team',
'created': '2026-01-15',
'difficulty': 'intermediate'
},
'description': {
'summary': 'Test database failover procedures when primary becomes unavailable',
'background': '''
The payment processing system relies on a PostgreSQL cluster with
primary-replica configuration. This scenario tests the team's ability
to detect and respond to primary database failures, including:
- Automated failover mechanisms
- Application resilience
- Data consistency verification
- Communication procedures
''',
'business_impact': '''
- Payment processing may be degraded or unavailable
- Users may experience transaction failures
- Financial reporting may be delayed
'''
},
'timeline': timeline.events,
'failure_injection': failure_injection.__dict__,
'detection_points': [dp.__dict__ for dp in detection_points],
'expected_responses': [
{
'phase': 'detection',
'actions': [
'Acknowledge alert within 2 minutes',
'Join incident channel',
'Review monitoring dashboards'
]
},
{
'phase': 'investigation',
'actions': [
'Check database logs',
'Verify primary database status',
'Confirm replica health',
'Review recent changes'
]
},
{
'phase': 'mitigation',
'actions': [
'Initiate failover procedure',
'Promote replica to primary',
'Update application configuration',
'Verify connection restoration'
]
},
{
'phase': 'communication',
'actions': [
'Update status page',
'Notify stakeholders',
'Document incident timeline',
'Coordinate with support team'
]
},
{
'phase': 'recovery',
'actions': [
'Verify data consistency',
'Monitor error rates',
'Plan primary restoration',
'Schedule post-mortem'
]
}
],
'success_criteria': [
'Alert fires within 60 seconds of failure',
'Incident acknowledged within 2 minutes',
'Failover completed within 5 minutes',
'Zero data loss confirmed',
'Applications reconnect automatically',
'Status page updated within 10 minutes',
'Complete recovery within 20 minutes'
],
'failure_scenarios': [
'Failover automation fails',
'Replica is not in sync',
'Application connection pool doesn\'t reconnect',
'DNS changes don\'t propagate',
'Monitoring alerts don\'t fire'
],
'required_tools': [
'Kubernetes cluster access',
'Database administration tools',
'Monitoring dashboards',
'Incident management platform',
'Communication channels'
],
'runbook_references': [
'Database Failover Procedure',
'Incident Response Playbook',
'Status Page Update Guide',
'Post-Mortem Template'
]
}
return scenario
def build_network_partition_scenario(self) -> Dict:
"""Build network partition scenario"""
timeline = ScenarioTimeline()
timeline.add_event(0, "start", "System operating normally")
timeline.add_event(300, "failure_injection", "Network partition between regions")
timeline.add_event(330, "detection", "Cross-region latency alerts")
timeline.add_event(420, "investigation", "Identify network partition")
timeline.add_event(600, "mitigation", "Reroute traffic to healthy region")
timeline.add_event(900, "resolution", "Network partition resolved")
scenario = {
'metadata': {
'name': 'Multi-Region Network Partition',
'id': 'network-partition-001',
'difficulty': 'advanced'
},
'description': {
'summary': 'Test system behavior during network partition between regions',
'business_impact': 'Users in affected regions may experience degraded service'
},
'timeline': timeline.events,
'failure_injection': {
'name': 'network_partition',
'method': 'chaos_mesh',
'target': 'region-a-to-region-b',
'parameters': {
'action': 'network-partition',
'direction': 'both',
'external_targets': ['region-b-cidr']
}
},
'success_criteria': [
'Detection within 2 minutes',
'Traffic automatically rerouted',
'No user-facing errors',
'Data consistency maintained'
]
}
return scenario
def build_cascade_failure_scenario(self) -> Dict:
"""Build cascading failure scenario"""
timeline = ScenarioTimeline()
timeline.add_event(0, "start", "System operating normally")
timeline.add_event(300, "failure_injection", "User service CPU spike")
timeline.add_event(360, "cascade_1", "User service becomes slow")
timeline.add_event(420, "cascade_2", "Payment service timeouts increase")
timeline.add_event(480, "cascade_3", "API gateway circuit breakers trip")
timeline.add_event(600, "detection", "Multiple service alerts firing")
timeline.add_event(720, "investigation", "Identify root cause service")
timeline.add_event(900, "mitigation", "Scale user service, reset circuit breakers")
timeline.add_event(1200, "resolution", "All services recovered")
scenario = {
'metadata': {
'name': 'Cascading Service Failure',
'id': 'cascade-failure-001',
'difficulty': 'expert'
},
'description': {
'summary': 'Test incident response to cascading failures across microservices',
'background': '''
A single service experiencing issues can trigger failures across
dependent services. This scenario tests the team's ability to:
- Identify the root cause service quickly
- Prevent cascade from spreading
- Restore services in correct order
- Use circuit breakers effectively
'''
},
'timeline': timeline.events,
'success_criteria': [
'Root cause identified within 10 minutes',
'Circuit breakers prevent total outage',
'Mitigation begins within 15 minutes',
'Full recovery within 30 minutes',
'Proper service restoration order followed'
]
}
return scenario
def export_scenario(self, scenario: Dict, format: str = "json") -> str:
"""Export scenario to specified format"""
if format == "json":
return json.dumps(scenario, indent=2)
elif format == "yaml":
import yaml
return yaml.dump(scenario, default_flow_style=False)
else:
raise ValueError(f"Unsupported format: {format}")
# Example usage
builder = ScenarioBuilder()
# Build database failover scenario
db_scenario = builder.build_database_failover_scenario()
print("Database Failover Scenario:")
print(builder.export_scenario(db_scenario, format="json"))
# Build network partition scenario
network_scenario = builder.build_network_partition_scenario()
# Build cascade failure scenario
cascade_scenario = builder.build_cascade_failure_scenario()
Game Day Execution
Real-Time Orchestration
# game_day_executor.py
import time
import threading
from typing import Dict, List, Callable, Any
from datetime import datetime
from dataclasses import dataclass, field
from enum import Enum
import logging
logger = logging.getLogger(__name__)
class GameDayPhase(Enum):
PREPARATION = "preparation"
BRIEFING = "briefing"
EXECUTION = "execution"
OBSERVATION = "observation"
DEBRIEFING = "debriefing"
COMPLETE = "complete"
@dataclass
class GameDayMetrics:
"""Tracks metrics during game day"""
start_time: datetime
detection_time: Optional[datetime] = None
acknowledgment_time: Optional[datetime] = None
mitigation_start_time: Optional[datetime] = None
resolution_time: Optional[datetime] = None
end_time: Optional[datetime] = None
# Communication metrics
status_updates_sent: int = 0
stakeholder_notifications: int = 0
# Technical metrics
alerts_fired: List[str] = field(default_factory=list)
runbooks_accessed: List[str] = field(default_factory=list)
tools_used: List[str] = field(default_factory=list)
# Team metrics
participants_active: Set[str] = field(default_factory=set)
roles_engaged: Set[str] = field(default_factory=set)
def calculate_mttd(self) -> Optional[float]:
"""Mean time to detect"""
if self.detection_time:
return (self.detection_time - self.start_time).total_seconds()
return None
def calculate_mttr(self) -> Optional[float]:
"""Mean time to resolve"""
if self.resolution_time:
return (self.resolution_time - self.start_time).total_seconds()
return None
def calculate_mtta(self) -> Optional[float]:
"""Mean time to acknowledge"""
if self.acknowledgment_time:
return (self.acknowledgment_time - self.start_time).total_seconds()
return None
class GameDayExecutor:
"""Orchestrates game day execution in real-time"""
def __init__(self, scenario: Dict):
self.scenario = scenario
self.current_phase = GameDayPhase.PREPARATION
self.metrics = GameDayMetrics(start_time=datetime.now())
self.observers: List[Callable] = []
self.event_log: List[Dict[str, Any]] = []
self.is_running = False
self.abort_requested = False
def add_observer(self, callback: Callable):
"""Add observer for game day events"""
self.observers.append(callback)
def notify_observers(self, event: Dict[str, Any]):
"""Notify all observers of event"""
for observer in self.observers:
try:
observer(event)
except Exception as e:
logger.error(f"Observer notification failed: {e}")
def log_event(self, event_type: str, description: str, metadata: Dict = None):
"""Log game day event"""
event = {
'timestamp': datetime.now().isoformat(),
'type': event_type,
'description': description,
'phase': self.current_phase.value,
'metadata': metadata or {}
}
self.event_log.append(event)
self.notify_observers(event)
logger.info(f"[{event_type}] {description}")
def run_preparation_phase(self):
"""Execute preparation phase"""
self.current_phase = GameDayPhase.PREPARATION
self.log_event("phase_start", "Starting preparation phase")
# Verify prerequisites
self.log_event("check", "Verifying prerequisites")
prerequisites = self.scenario.get('required_tools', [])
for prereq in prerequisites:
self.log_event("prerequisite", f"Checking: {prereq}")
time.sleep(1) # Simulate check
# Set up monitoring
self.log_event("setup", "Configuring monitoring dashboards")
# Establish communication channels
self.log_event("setup", "Establishing communication channels")
self.log_event("phase_complete", "Preparation phase complete")
def run_briefing_phase(self):
"""Execute briefing phase"""
self.current_phase = GameDayPhase.BRIEFING
self.log_event("phase_start", "Starting briefing phase")
# Present scenario
self.log_event(
"briefing",
f"Scenario: {self.scenario['metadata']['name']}"
)
# Review objectives
for objective in self.scenario.get('success_criteria', []):
self.log_event("objective", objective)
# Assign roles
self.log_event("briefing", "Roles and responsibilities confirmed")
# Q&A
self.log_event("briefing", "Questions and clarifications")
self.log_event("phase_complete", "Briefing phase complete")
def run_execution_phase(self):
"""Execute main game day scenario"""
self.current_phase = GameDayPhase.EXECUTION
self.log_event("phase_start", "Starting execution phase")
timeline = self.scenario.get('timeline', [])
start_time = time.time()
for event in timeline:
if self.abort_requested:
self.log_event("abort", "Game day aborted by facilitator")
break
# Wait until event time
event_time = event['time_offset']
elapsed = time.time() - start_time
wait_time = event_time - elapsed
if wait_time > 0:
time.sleep(wait_time)
# Execute event
event_type = event['type']
description = event['description']
self.log_event(
event_type,
description,
metadata={'expected_action': event.get('expected_action')}
)
# Special handling for different event types
if event_type == "failure_injection":
self._inject_failure()
elif event_type == "detection":
self.metrics.detection_time = datetime.now()
elif event_type == "mitigation":
self.metrics.mitigation_start_time = datetime.now()
elif event_type == "resolution":
self.metrics.resolution_time = datetime.now()
self.log_event("phase_complete", "Execution phase complete")
def _inject_failure(self):
"""Inject failure into system"""
failure = self.scenario.get('failure_injection', {})
method = failure.get('method')
target = failure.get('target')
self.log_event(
"failure_injection",
f"Injecting failure via {method} on {target}",
metadata=failure
)
# Actual failure injection would happen here
# This would integrate with chaos engineering tools
def run_observation_phase(self):
"""Observe team response"""
self.current_phase = GameDayPhase.OBSERVATION
self.log_event("phase_start", "Starting observation phase")
# Monitor team actions
observations = [
"Monitoring team response to alerts",
"Observing communication patterns",
"Tracking runbook usage",
"Recording decision-making process",
"Noting deviations from procedures"
]
for observation in observations:
self.log_event("observation", observation)
time.sleep(5)
self.log_event("phase_complete", "Observation phase complete")
def run_debriefing_phase(self):
"""Conduct post-game debrief"""
self.current_phase = GameDayPhase.DEBRIEFING
self.log_event("phase_start", "Starting debriefing phase")
# Review metrics
self.log_event(
"metrics",
f"MTTD: {self.metrics.calculate_mttd()}s, "
f"MTTR: {self.metrics.calculate_mttr()}s"
)
# Discuss what went well
self.log_event("debrief", "Discussing successes")
# Discuss improvement areas
self.log_event("debrief", "Identifying improvement opportunities")
# Document action items
self.log_event("debrief", "Creating action items")
self.log_event("phase_complete", "Debriefing phase complete")
def execute(self):
"""Execute complete game day"""
self.is_running = True
self.log_event("start", f"Game day started: {self.scenario['metadata']['name']}")
try:
self.run_preparation_phase()
if not self.abort_requested:
self.run_briefing_phase()
if not self.abort_requested:
self.run_execution_phase()
if not self.abort_requested:
self.run_observation_phase()
self.run_debriefing_phase()
self.current_phase = GameDayPhase.COMPLETE
self.metrics.end_time = datetime.now()
self.log_event("complete", "Game day completed successfully")
except Exception as e:
logger.error(f"Game day execution failed: {e}")
self.log_event("error", f"Execution failed: {str(e)}")
raise
finally:
self.is_running = False
return self.generate_report()
def abort(self):
"""Abort game day execution"""
self.abort_requested = True
self.log_event("abort_requested", "Abort requested by facilitator")
def generate_report(self) -> Dict:
"""Generate game day report"""
report = {
'scenario': self.scenario['metadata'],
'execution_summary': {
'start_time': self.metrics.start_time.isoformat(),
'end_time': self.metrics.end_time.isoformat() if self.metrics.end_time else None,
'duration': str(self.metrics.end_time - self.metrics.start_time) if self.metrics.end_time else None,
'completed': self.current_phase == GameDayPhase.COMPLETE
},
'metrics': {
'mttd_seconds': self.metrics.calculate_mttd(),
'mtta_seconds': self.metrics.calculate_mtta(),
'mttr_seconds': self.metrics.calculate_mttr(),
'alerts_fired': len(self.metrics.alerts_fired),
'runbooks_accessed': len(self.metrics.runbooks_accessed),
'participants': list(self.metrics.participants_active)
},
'success_criteria_met': self._evaluate_success_criteria(),
'event_log': self.event_log,
'recommendations': self._generate_recommendations()
}
return report
def _evaluate_success_criteria(self) -> Dict[str, bool]:
"""Evaluate if success criteria were met"""
criteria_results = {}
# Example criteria evaluation
criteria = self.scenario.get('success_criteria', [])
for criterion in criteria:
# This would contain actual evaluation logic
# For now, using placeholder
criteria_results[criterion] = True
return criteria_results
def _generate_recommendations(self) -> List[str]:
"""Generate recommendations based on execution"""
recommendations = []
# Analyze metrics
mttd = self.metrics.calculate_mttd()
if mttd and mttd > 120:
recommendations.append(
"Consider improving alert configuration to reduce detection time"
)
mttr = self.metrics.calculate_mttr()
if mttr and mttr > 600:
recommendations.append(
"Review and simplify runbook procedures to reduce resolution time"
)
# Analyze event log for issues
for event in self.event_log:
if event['type'] == 'error':
recommendations.append(
f"Address error: {event['description']}"
)
return recommendations
# Example usage
if __name__ == "__main__":
# Load scenario
builder = ScenarioBuilder()
scenario = builder.build_database_failover_scenario()
# Create executor
executor = GameDayExecutor(scenario)
# Add observer for real-time updates
def log_observer(event):
print(f"[{event['timestamp']}] {event['type']}: {event['description']}")
executor.add_observer(log_observer)
# Execute game day
report = executor.execute()
# Print report
print("\n" + "="*80)
print("GAME DAY REPORT")
print("="*80)
print(json.dumps(report, indent=2))
Post-Game Analysis
Comprehensive Reporting
# game_day_analyzer.py
from typing import List, Dict, Any
import matplotlib.pyplot as plt
import pandas as pd
from datetime import datetime
class GameDayAnalyzer:
"""Analyzes game day results and generates insights"""
def __init__(self, report: Dict):
self.report = report
self.insights = []
def analyze_timing_metrics(self) -> Dict:
"""Analyze timing-related metrics"""
metrics = self.report['metrics']
analysis = {
'detection_performance': self._analyze_detection(metrics['mttd_seconds']),
'response_performance': self._analyze_response(metrics['mtta_seconds']),
'resolution_performance': self._analyze_resolution(metrics['mttr_seconds'])
}
return analysis
def _analyze_detection(self, mttd: float) -> Dict:
"""Analyze detection performance"""
# Industry benchmarks
excellent = 30
good = 60
acceptable = 120
if mttd <= excellent:
rating = "excellent"
feedback = "Detection time is excellent"
elif mttd <= good:
rating = "good"
feedback = "Detection time is good"
elif mttd <= acceptable:
rating = "acceptable"
feedback = "Detection time is acceptable but could be improved"
else:
rating = "needs_improvement"
feedback = "Detection time needs significant improvement"
return {
'value': mttd,
'rating': rating,
'feedback': feedback,
'benchmark': excellent
}
def _analyze_response(self, mtta: float) -> Dict:
"""Analyze response performance"""
excellent = 120 # 2 minutes
acceptable = 300 # 5 minutes
if mtta <= excellent:
rating = "excellent"
elif mtta <= acceptable:
rating = "acceptable"
else:
rating = "needs_improvement"
return {'value': mtta, 'rating': rating}
def _analyze_resolution(self, mttr: float) -> Dict:
"""Analyze resolution performance"""
# These would be service-specific
excellent = 600 # 10 minutes
acceptable = 1800 # 30 minutes
if mttr <= excellent:
rating = "excellent"
elif mttr <= acceptable:
rating = "acceptable"
else:
rating = "needs_improvement"
return {'value': mttr, 'rating': rating}
def identify_patterns(self) -> List[Dict]:
"""Identify patterns in event log"""
patterns = []
event_log = self.report['event_log']
# Analyze communication patterns
comm_events = [e for e in event_log if 'communication' in e['type']]
if len(comm_events) < 3:
patterns.append({
'type': 'communication',
'finding': 'Insufficient status updates',
'recommendation': 'Increase communication frequency during incidents'
})
# Analyze decision-making
decision_events = [e for e in event_log if 'decision' in e.get('description', '').lower()]
if decision_events:
patterns.append({
'type': 'decision-making',
'finding': f'{len(decision_events)} key decisions made',
'recommendation': 'Document decision rationale in runbooks'
})
return patterns
def generate_action_items(self) -> List[Dict]:
"""Generate prioritized action items"""
action_items = []
timing_analysis = self.analyze_timing_metrics()
# Detection improvements
if timing_analysis['detection_performance']['rating'] != 'excellent':
action_items.append({
'priority': 'high',
'category': 'monitoring',
'title': 'Improve alert detection time',
'description': 'Review and tune alert thresholds',
'owner': 'SRE Team'
})
# Response improvements
if timing_analysis['response_performance']['rating'] != 'excellent':
action_items.append({
'priority': 'medium',
'category': 'process',
'title': 'Streamline incident response',
'description': 'Review on-call procedures',
'owner': 'On-Call Team'
})
# Add recommendations from report
for recommendation in self.report.get('recommendations', []):
action_items.append({
'priority': 'medium',
'category': 'improvement',
'title': recommendation,
'description': '',
'owner': 'TBD'
})
return action_items
def generate_executive_summary(self) -> str:
"""Generate executive summary"""
scenario_name = self.report['scenario']['name']
metrics = self.report['metrics']
success = self.report['success_criteria_met']
mttd = metrics.get('mttd_seconds', 0)
mttr = metrics.get('mttr_seconds', 0)
summary = f"""
# Game Day Executive Summary
## Scenario
{scenario_name}
## Overview
The team successfully completed a game day exercise testing our incident response
capabilities. The exercise provided valuable insights into our preparedness and
identified areas for improvement.
## Key Metrics
- **Mean Time to Detect (MTTD)**: {mttd:.0f} seconds ({mttd/60:.1f} minutes)
- **Mean Time to Resolve (MTTR)**: {mttr:.0f} seconds ({mttr/60:.1f} minutes)
- **Success Criteria Met**: {sum(success.values())}/{len(success)}
- **Participants**: {len(metrics.get('participants', []))}
## Performance Rating
{self._get_overall_rating(metrics)}
## Top Recommendations
{self._format_top_recommendations()}
## Next Steps
1. Review and implement action items
2. Update runbooks based on learnings
3. Schedule follow-up game day
"""
return summary
def _get_overall_rating(self, metrics: Dict) -> str:
"""Calculate overall performance rating"""
# Simplified rating logic
mttd = metrics.get('mttd_seconds', float('inf'))
mttr = metrics.get('mttr_seconds', float('inf'))
if mttd < 60 and mttr < 600:
return "**Excellent** - Team demonstrated strong incident response capabilities"
elif mttd < 120 and mttr < 1200:
return "**Good** - Team performed well with room for improvement"
else:
return "**Needs Improvement** - Several areas require attention"
def _format_top_recommendations(self) -> str:
"""Format top 3 recommendations"""
action_items = self.generate_action_items()
top_3 = sorted(action_items, key=lambda x: x['priority'])[:3]
formatted = []
for i, item in enumerate(top_3, 1):
formatted.append(f"{i}. {item['title']} ({item['priority']} priority)")
return '\n'.join(formatted)
This is a comprehensive start to the game day planning blog post. Would you like me to continue with the remaining sections including:
- Team coordination and communication
- Measurement and metrics
- Best practices and production considerations
- Complete conclusion
Then I’ll create the remaining 4 blog posts for runbook automation, on-call rotation, incident communication, and post-mortem automation.