Prompt
How do I handle escalation paths for on-call response?
Latest observation
A good escalation path for on-call is simple, explicit, and time-based. The goal is to make sure an incident always has an owner and never sits unresolved because people are unsure who should act next.
A practical escalation structure
1. Define severity levels
Create clear incident severities, such as:
- SEV1: Critical outage / major customer impact
- SEV2: Significant degradation / partial impact
- SEV3: Limited impact / workaround exists
- SEV4: Minor issue / no immediate impact
Each severity should map to:
- who is paged
- response time expectation
- whether escalation is automatic or manual
2. Set response time thresholds
Decide how long the current on-call person has before escalation happens.
Example:
- SEV1: acknowledge in 5 minutes, escalate if no action in 10
- SEV2: acknowledge in 10 minutes, escalate if no action in 20
- SEV3: acknowledge in 30 minutes, escalate if no action in 60
Make this explicit in your runbooks and paging tool.
3. Use a layered escalation chain
A common pattern:
- Primary on-call
- Secondary on-call / backup
- Team lead / incident commander
- Engineering manager / director
- Cross-functional support if needed
(SRE, DBAs, network, security, vendor support, etc.)
Escalation should be based on:
- no acknowledgment
- no progress
- incident severity increasing
- required expertise unavailable
4. Separate alert handling from incident management
The person paged first may not be the person who coordinates the entire incident.
Good roles:
- Responder: fixes the immediate problem
- Incident commander: coordinates and keeps the incident moving
- Communicator: updates stakeholders
- Subject matter expert: consulted as needed
For larger incidents, handoff coordination to an incident commander early.
5. Automate where possible
Use paging and incident tools to automate:
- retries to primary after X minutes
- page secondary after Y minutes
- notify manager after Z minutes
- open an incident channel automatically
- attach runbook links to alerts
Automation reduces hesitation and makes escalation consistent.
6. Establish clear “wake up” rules
Decide what justifies paging someone outside working hours. This avoids alert fatigue.
Example paging criteria:
- customer-facing outage
- security incident
- data loss risk
- revenue-impacting issue
- production system unavailable
- SLO burn rate above threshold
If it’s not urgent, route it to ticketing instead of paging.
7. Document the escalation policy
Every alert should have:
- owner
- severity
- first responder
- backup responder
- escalation contacts
- response deadlines
- runbook link
- customer communication expectations
Keep this documentation short and easy to find during an incident.
8. Review and improve after incidents
After each incident, ask:
- Did escalation happen fast enough?
- Did the right people get paged?
- Were there unnecessary pages?
- Was the backup effective?
- Did handoffs cause delays?
Use postmortems to refine thresholds and routing.
Example escalation policy
Here’s a simple model:
- 0 min: alert pages primary on-call
- 5 min: if no acknowledgment, page secondary
- 10 min: if still unresolved and SEV1/2, notify incident commander
- 15 min: if customer impact continues, escalate to manager/director
- 30 min: engage additional specialists or vendor support
Best practices
- Keep escalation paths short
- Make ownership unambiguous
- Escalate based on impact, not just time
- Avoid paging too many people at once
- Include a fallback if the primary and secondary are unavailable
- Rehearse the process with game days or simulations
If you want, I can also give you:
- a sample on-call escalation policy,
- a RACI table for on-call roles, or
- a PagerDuty/Opsgenie-style escalation flow template.