Why Root Cause Analysis Always Takes More Than Three Hours
It’s 2:14 AM. An alert fires. By the time the incident channel goes quiet again, the sun is up, three people are exhausted, and the postmortem is already on the calendar for “root cause: TBD.”
Ask anyone who’s carried a pager: fixing the problem is rarely what eats three hours. Once you know what’s wrong, the fix is often five minutes — restart a service, roll back a deploy, bump a limit. The three hours go somewhere else entirely. They go into finding out what’s wrong in the first place.
Where the Three Hours Actually Go
Here’s where that time actually goes.
The Archaeology Tax
Something broke, but the evidence is scattered across six places: a metrics dashboard that only goes back 24 hours, a log aggregator with its own query syntax, a deploy history in a different tool, a config change buried in a pull request from two weeks ago, a support ticket that mentioned “this feels slower” three days before anyone paged. None of these tools know the others exist. You are the integration layer, manually, at 2 AM, holding six browser tabs open and trying to line up timestamps by eye.
The Tribal Knowledge Tax
Somewhere in the org is one person who remembers that this exact symptom happened eight months ago, and it was actually caused by a batch job that runs on the first of the month, and it only affects the eastern cluster. That person is asleep, or on vacation, or has left the company and taken the knowledge with them. Everyone else is re-deriving what they should already know.
The Reconstruction Tax
Once you find a suspect, you still have to prove it. That means manually rebuilding a timeline: what changed, in what order, and did the timing actually line up with when things went bad — or is it just a coincidence you’re pattern-matching onto because it’s 4 AM and you want an answer. Most of the “three hours” isn’t spent being wrong. It’s spent being unsure, and checking, and re-checking, because nothing hands you the timeline pre-built.
None of this shows up on an incident timer. The timer says “3h 12m to resolution.” What it doesn’t say is that 2h 40m of that was spent finding the haystack before anyone could even look for the needle — and that this exact search happens again next week, from scratch, because nothing that was learned this time got carried forward.
This is the part of on-call nobody puts in the job description: you are not an engineer during an incident. You are a detective with no case file, reconstructing a crime scene that six different witnesses each saw a fragment of, using six different languages.
Why Runbooks Don’t Fix This
Teams don’t get faster at this by hiring more people or writing more runbooks — runbooks answer the incidents you’ve already had, not the one happening right now. They get faster when the fragments stop being fragments: when the evidence is already connected before the pager goes off, and the person on call is spending their three hours confirming an answer instead of assembling one.
If any of this sounds like last Tuesday, you’re not doing it wrong. This is just what root cause analysis costs when every system that holds a piece of the story refuses to talk to the others.