Incident Response

Incident Response

Severity Levels

Level	Description	Response Time	Example
SEV1	Platform down, all customers affected	Immediate	Database outage, control plane crash
SEV2	Major degradation, many customers affected	15 min	Analysis engine failures, agent disconnects at scale
SEV3	Partial degradation, some customers affected	1 hour	Single-tenant issues, intermittent errors
SEV4	Minor issue, workaround available	Next business day	UI glitches, non-critical feature broken

Detection

Automated

• Grafana alerts for error rate spikes, latency increases, pod restarts
• Health endpoint monitoring
• Agent connectivity SLO tracking

Manual

• Customer support tickets
• MCP investigation finding systemic issues:

  MCP: get_system_health
  MCP: get_fleet_summary

Triage (First 5 Minutes)

1. Assess scope: How many customers affected?

   MCP: get_fleet_summary

2. Identify component: Which service is degraded?

   MCP: get_system_health

3. Check recent changes: Was anything deployed recently?

   # Check deploy history
   helm history pulsestream-core -n pulsestream-core-prod
   helm history pulsestream-agent -n pulsestream-agent-prod

4. Declare severity and notify

Mitigation

Quick Mitigation Options

Action	Command	When
Rollback deploy	helm rollback <release> <rev> -n <ns>	Bad deploy caused the issue
Scale down	pulsar scale down --env prod	Need to stop bleeding
Restart service	kubectl rollout restart deployment/<svc> -n <ns>	Stuck process / memory leak

LLM Provider Outage

1. Check provider status page
2. Analysis engine will queue work with exponential backoff
3. Customer-facing: analyses will complete once provider recovers
4. No PulseStream action needed unless outage exceeds 1 hour

Post-Mortem

After every SEV1/SEV2:

1. Document timeline (when detected, when mitigated, when resolved)
2. Root cause analysis
3. Action items to prevent recurrence
4. Update runbooks if a new failure mode was discovered