Runbook Template

A runbook is a procedure someone can follow at 3am, half-awake, without needing to understand the whole system first. Write it as literal steps and copy-pasteable commands, not prose explanations.

Keep it to one scenario per runbook. Don't try to make one runbook cover everything.

How to use this document

• Title it by the specific trigger/symptom, not a vague system name. "Checkout API rollback" not "Checkout API stuff."
• Every command should be copy-pasteable as-is (fill in real placeholders, don't leave <your-service-name> ambiguous).
• Test it: have someone unfamiliar with the system follow it literally and see where they get stuck. Fix those spots.
• Link it from the alert itself and from the on-call handbook's alert table.

Runbook: [specific scenario, e.g. "Checkout API elevated error rate"]

When to use this runbook

[Specific symptom description: what dashboard, alert, or error message points here. Include what this runbook is NOT for, if there's a similar-looking scenario with a different cause.]

Prerequisites

• Access to [system/tool, e.g. prod kubectl context, deploy tool]
• Permissions: [role needed]

Diagnosis steps

1. Check [dashboard link], look for [specific signal, e.g. "error rate above 5% on the /checkout endpoint"].
2. Check recent deploys:

[exact command, e.g. `deploytool history checkout-api --last 3`]

1. Check [dependency/downstream system] health: [link/command].
2. Confirm this matches the pattern this runbook addresses, [specific signature, e.g. "error is ConnectionPoolExhausted, not Timeout"]. If it doesn't match, stop and escalate instead of continuing.

Resolution steps

Ordered, specific, copy-pasteable. Mark anything destructive or hard to reverse.

1. [Step 1, e.g. "Identify the last known-good deploy version"]

[exact command]

1. [Step 2, e.g. "Roll back"]

[exact command]

⚠️ [Any warning, e.g. "This does not roll back the DB migration, see 'if a migration was involved' below."]

1. [Step 3, e.g. "Confirm recovery"]
   • Check [dashboard], error rate should return to baseline within [X min].
   • If it doesn't, [next step / escalation].

If a database migration was involved

[Separate, explicit sub-section. This is the #1 place runbooks fall short. State clearly whether rollback is safe, and if not, what the recovery path is.]

Verification

• [Metric/dashboard] back to baseline
• No new alerts firing
• [Any manual smoke test, e.g. "Complete one test checkout end-to-end in staging/prod"]

Escalation

If steps above don't resolve within [time], escalate to [team/person] via [channel/page].

Related

• Postmortem(s) that led to this runbook: [links]
• Related runbooks: [links]

Example: filled runbook (abbreviated)

Runbook: Checkout API elevated error rate (connection pool exhaustion)Triggers: HighErrorRate-checkout-api, DBConnectionPoolExhausted

Diagnosis:

Check [Grafana: checkout-api overview]. Look at the error rate and pool utilization panels.

deploytool history checkout-api --last 3. Check for a deploy in the last 30 min.

If pool utilization is above 90% and it correlates with a recent deploy, move to resolution.

Resolution:

deploytool rollback checkout-api --to-previous

Confirm pool utilization drops below 50% within 5 minutes on [Grafana link].

Post confirmation in #incidents. Open a follow-up ticket for the actual code fix. Don't try to hotfix the retry logic live.

이 템플릿을 팀 워크스페이스로 복제하려면 → rootr.io