On-Call Handbook Template
A living reference for whoever's holding the pager this week. Copy it into your own workspace and fill in the bracketed parts.
Keep it updated as your systems change. A handbook that's six months stale is worse than none at all.
How to use this document
- Owner: whoever runs on-call rotation (usually an EM or a rotating "on-call lead")
- Update cadence: review after every incident that surfaces a gap, and do a full pass quarterly
- Audience: anyone in the rotation, including new hires seeing it for the first time at 3am
1. Rotation basics
| Field | Value |
|---|---|
| Rotation tool | [PagerDuty / Opsgenie / homegrown] |
| Shift length | [e.g. 1 week, Mon 10:00 to Mon 10:00] |
| Primary / secondary | [describe escalation between the two] |
| Handoff time & ritual | [e.g. Monday 10:00 standup, 5 min sync on open issues] |
| Who to page for escalation beyond secondary | [name/role, e.g. EM on duty] |
Write this so a brand-new engineer can find the pager schedule without asking anyone.
2. Before your shift starts
- Read the handoff notes from the outgoing on-call
- Confirm your phone/laptop actually receives pages (test alert if unsure)
- Check open incidents / unresolved alerts from the last 48 hours
- Know today's deploys and any flagged risky changes
- Confirm you have access to: [prod SSH/kubectl, admin dashboard, cloud console, VPN, status page admin]
3. Severity levels
Define these once, use them consistently everywhere (alerts, incident docs, postmortems).
| Severity | Definition | Response time | Who's notified |
|---|---|---|---|
| SEV1 | [e.g. full outage, data loss risk, customer-facing and revenue-impacting] | [e.g. page immediately, ack in 5 min] | [on-call + EM + exec channel] |
| SEV2 | [e.g. major feature degraded, workaround exists] | [e.g. ack in 15 min] | [on-call + team channel] |
| SEV3 | [e.g. minor/internal, no customer impact] | [e.g. next business day] | [team channel, no page] |
4. What to do when paged
- Acknowledge the page within [SLA] so escalation doesn't fire.
- Assess, is this SEV1/2/3? Check the relevant dashboard first (link it below).
- Communicate, post in [#incidents channel] even before you know the cause: "Investigating [alert name], will update in 10 min."
- Mitigate before you diagnose, if there's a known fast mitigation (rollback, feature flag off, scale up), do that first. Root cause can wait.
- Escalate if you're stuck after [15-20 min] or it's outside your expertise, paging in help early is always right, never a failure.
- Declare an incident if customer-facing impact crosses [threshold], this starts the postmortem process.
- Resolve and write a handoff note, even for alerts that self-resolved, note what happened so the pattern is visible.
5. Key dashboards & links
Fill in real links. This is the single most valuable part of the doc.
An on-call engineer half-asleep should be able to click straight to the right graph.
| System | Dashboard | Alert source |
|---|---|---|
| [API service] | [link] | [link] |
| [Database] | [link] | [link] |
| [Queue / workers] | [link] | [link] |
| [Third-party dependency, e.g. payment provider status page] | [link] | n/a |
6. Common alerts & first response
This is your living runbook index. Add a row every time an alert pages someone who didn't know what to do.
| Alert name | Likely cause | First response | Runbook |
|---|---|---|---|
HighErrorRate-checkout-api | Deploy regression or downstream dependency timeout | Check last deploy in [last 30 min]; if correlated, roll back | [link to runbook] |
DBConnectionPoolExhausted | Traffic spike or slow query holding connections | Check slow query log; consider bumping pool size or killing long queries | [link to runbook] |
QueueBacklogGrowing | Worker crash loop or upstream burst | Check worker pod status; scale workers if healthy but backlogged | [link to runbook] |
7. Escalation contacts
| Role | Name/Handle | When to page |
|---|---|---|
| Secondary on-call | [handle] | Primary unresponsive after [X] min |
| Team EM | [handle] | SEV1, or any customer commitment at risk |
| Security lead | [handle] | Suspected security incident |
| Infra/platform team | [handle] | Cloud provider issue, network partition |
8. End-of-shift handoff template
Post this in [#oncall-handoff] at the end of every shift, even a quiet one:
On-call handoff, [date]
Shift: [name] → [next name]
Incidents this shift: [none / link to postmortems]
Open issues to watch: [e.g. elevated latency on X since Tuesday, not yet root-caused]
Deploys pending / risky changes coming up: [...]
Anything I'd want to know if I were starting this shift: [...]Example: filled row
Alert:
HighErrorRate-checkout-apiLikely cause: new payment provider client library shipped in yesterday's deploy has a retry bug under timeout.
First response: checked the deploy log and confirmed it lined up with the 14:02 deploy. Rolled back with
[deploy tool] rollback checkout-api. Error rate was back to baseline in 4 minutes. Filed a follow-up ticket to fix the retry logic before re-deploying.Runbook:
runbooks/checkout-api-rollback.md
이 템플릿을 팀 워크스페이스로 복제하려면 → rootr.io