Incident Postmortems for Staff Engineers

A good incident postmortem fixes the system, not the person. The single most important property is that it is blameless: it assumes a competent engineer acted reasonably inside a system that allowed the failure, and it asks what about the system let this happen. Get that culture right and people surface the honest detail you need to actually prevent recurrence. Get it wrong and every postmortem becomes theater, because nobody tells the truth when truth gets punished.

The postmortem is where an organization either learns from failure or merely survives it. The difference is not how serious the meeting feels; it is whether the action items get done and whether people felt safe enough to tell you what really happened.

Why postmortems are a staff engineer’s responsibility

Incidents are inevitable in any system of real complexity. What separates a maturing engineering org from a stagnant one is whether each incident makes the system stronger or just exhausts the people who fought it. Driving that learning is squarely staff-level work, because it requires both technical depth and the influence to get systemic fixes prioritized.

A staff engineer running a postmortem is doing two jobs at once: finding the true root cause, and protecting the culture that makes finding it possible. Both matter, and the second is the one that quietly fails first. This post is part of the Staff engineer craft series.

What is a blameless postmortem?

A blameless postmortem examines an incident by focusing on the systemic conditions that allowed it, not on the individual who triggered it. Its premise is that good engineers cause incidents inside imperfect systems, so the productive question is never “who did this” but “what about our system made this possible, and how do we change it.” Blame is replaced by systemic curiosity.

The reasoning is practical, not just kind. If “root cause: human error” is an acceptable conclusion, you stop looking exactly when the useful analysis would begin, because human error is the start of the investigation, not the end. Why was that error possible? Why did nothing catch it? Those questions lead to fixes; “be more careful” leads nowhere.

Why blameless? Doesn’t someone need to be accountable?

Accountability belongs to fixing the system, not to punishing the person. Blame does not improve reliability; it teaches people to hide mistakes, downplay incidents, and withhold the candid detail a postmortem depends on, which makes the next failure both more likely and harder to learn from. Blameless cultures surface more truth and therefore fix more root causes.

This is the part leaders find counterintuitive: the way to get more accountability is to remove blame. The accountability is real, it just attaches to the system and to completing the action items, not to shaming whoever was on the keyboard when a latent flaw finally surfaced.

What should an incident postmortem include?

A complete postmortem has a summary and impact, a factual timeline, the root cause, the detection and mitigation story, and concrete action items with owners and due dates. Everything except the action items is context; the owned, tracked action items are the only part that actually changes the future.

A dependable structure:

1. Summary and impact. What happened, for how long, and who or what was affected.
2. Timeline. A factual, timestamped sequence: detection, diagnosis, mitigation, resolution.
3. Root cause. The systemic condition that allowed it, not just the trigger.
4. Detection and response. How you found out, how fast, and what slowed the response.
5. What went well. The mitigations and instincts worth reinforcing.
6. Action items. Specific, owned, dated changes that prevent recurrence.

How do you find the real root cause of an incident?

Keep asking “why” past the first technical trigger until you reach a condition a system change can fix. The bad deploy, the wrong config, the unhandled input, those are triggers. The root cause is why that trigger was possible and why nothing caught it. Stop only when the answer is something you can actually change about the system.

The classic example: the incident was “a deploy took the service down.” Ask why. The deploy changed a config past a limit. Why did that ship? There was no validation on the value. Why did it reach production? The pipeline had no check for it. The root cause is the missing guardrail, not the engineer who typed the value, and the fix is a guardrail, not a reprimand.

This is also where the postmortem connects back to design: many root causes are exactly the failure modes the Distributed systems patterns series exists to prevent, from a missing timeout budget to absent backpressure to a readiness probe that lied. A good postmortem frequently produces a future RFC.

Is there really a single root cause?

Usually not, and “root cause” is the most misleading phrase in incident analysis. It implies a single defective component that, once repaired, restores correctness. Real incidents in distributed systems are almost never shaped like that.

What you actually find is a set of contributing factors that were each individually survivable and became an outage in combination. A config change that was fine. A retry policy that was fine. A dependency that was slightly slower than usual. A dashboard that did not surface the queue depth. None of those alone causes an incident; together they do.

The reason this matters practically rather than philosophically: searching for “the” root cause makes you stop too early. The moment someone identifies a plausible single cause, analysis ends and the remediation targets that one thing. The other four factors remain, and they recombine with a different fifth factor six weeks later — which is why teams experience “different” incidents that feel eerily familiar.

The better framing is to ask two questions of every incident:

1. What had to be true for this to happen? Enumerate every condition. You will typically find four to eight, not one.
2. What stopped it from being worse? This is the question nobody asks, and it identifies the defences that are silently holding your system together. Those are the ones you must not accidentally remove in a future refactor.

The second question is disproportionately valuable. Incidents reveal your system’s real safety mechanisms, and quite often the thing that limited the blast radius was accidental — a rate limit set for an unrelated reason, a cache that happened to serve stale data. Naming it converts an accident into a deliberate control.

A related discipline: resist “human error” as a terminal explanation. It is where analysis goes to die. If an engineer ran the wrong command, the useful questions are why that command was runnable, why it was not obviously wrong at the time, and why nothing caught it before impact. Every one of those has an engineering answer; “be more careful” does not.

How do you write an incident timeline?

The timeline is the load-bearing part of the document and the part most often written badly. Done well it makes the analysis obvious; done badly it is an unreadable log dump.

Three rules make the difference.

Record what people knew, not just what was true. There is always a gap between when a system broke and when anyone understood what was broken. That gap is where your detection and observability gaps live, and it is invisible if the timeline only lists what was actually happening. Write both: “14:02 — error rate begins climbing (discovered in review; nothing alerted).”

Timestamp four specific moments. Impact start, detection, mitigation, resolution. Everything else is detail; these four give you the metrics that matter, and they immediately expose which phase is your real problem.

Interval	What it measures	What a bad number tells you
Impact → detection	Time to detect	Alerting gap: you found out from users or luck
Detection → mitigation	Time to mitigate	Runbook, tooling, or escalation gap
Mitigation → resolution	Time to fully resolve	Usually fine to be long; mitigation is what users feel
Impact → mitigation	The number users experienced	The one to actually drive down

Most teams optimise the wrong interval. If detection takes 40 minutes and mitigation takes 4, no amount of runbook polish helps — the work is in alerting, and specifically in alerting on symptoms users feel rather than on causes.

Keep interpretation out of it. The timeline states what happened and when. Analysis belongs in its own section. Mixing them produces a document where the conclusion is smuggled into the evidence, and reviewers cannot evaluate whether the reasoning holds.

Action items that actually get done

The graveyard of postmortems is full of action items nobody ever completed. An action item prevents recurrence only if it is specific, owned by a named person, given a due date, and tracked like any other prioritized work. “We should improve monitoring” is not an action item; “add an alert on config values exceeding the broker limit, owned by X, due next sprint” is.

The staff engineer’s real contribution here is getting those items prioritized against feature work. A postmortem whose action items lose every sprint-planning fight has not made the system safer; it has documented a failure you will repeat. Tracking completion, and following up when items slip, is the unglamorous part that determines whether the whole exercise was worth anything.

Not all action items are equal, and the difference is whether they depend on someone remembering. Ranked by how durably they prevent recurrence:

Class	Example	Why it ranks there
Eliminate	Make the invalid state unrepresentable; delete the dangerous code path	The failure cannot recur, and no one has to remember anything
Guardrail	Validation that rejects the bad config; a CI check that blocks the change	Recurrence is blocked automatically at the point of the mistake
Automate	Replace the manual runbook step with a script	Removes the variance a tired human introduces at 3 a.m.
Detect	An alert on the condition that preceded impact	Does not prevent it, but collapses time-to-detection
Document	A runbook entry, a wiki page	Depends entirely on someone finding and following it under stress
Educate	”Brief the team”, “be careful with this command”	Decays with every new hire and every passing month

Push every action item as far up that table as it will go. The instinct after an incident is to write documentation, because it is fast and feels responsible — but a runbook is the second-weakest class of fix available. Ask each time: could this be a guardrail instead? Could the dangerous thing simply not exist?

A worked example of the escalation. A bad config value takes down the service. Document: add a runbook noting the safe range. Detect: alert when the value exceeds the limit. Guardrail: reject the config at deploy time so it never reaches production. Eliminate: derive the value from the broker’s advertised limit so a human never sets it at all. Same incident, four wildly different levels of future safety, and the last one costs perhaps a day more than the first.

Two counts worth tracking across postmortems, because they predict whether the practice is working: the proportion of action items that are Eliminate or Guardrail, and the completion rate of items older than 30 days. If the first number is low, you are documenting rather than fixing. If the second is low, your postmortems are theatre regardless of how good the analysis was.

Finally, cap the list. A postmortem with fourteen action items will complete three. Pick the two or three that most reduce the chance or the blast radius of recurrence, get those genuinely done, and let the rest go explicitly rather than by attrition — an item silently abandoned teaches the team that the list is optional.

When should you write a postmortem?

Write one for any incident that had real user impact, exceeded a severity threshold, or surprised you, even a near-miss that could have been severe. The trigger should be defined in advance (for example, any SEV-1 and SEV-2, or any customer-visible outage) so the decision is not relitigated emotionally after each event. A clear, pre-agreed threshold is what keeps postmortems consistent rather than political.

Near-misses deserve special attention, and teams routinely skip them. An incident that almost took down production but was caught in time exposed the same systemic weakness as a full outage would have, minus the damage. Writing the postmortem anyway means you fix the weakness before it actually fires, which is the cheapest possible time to learn the lesson. A culture that only writes postmortems for incidents that already hurt is one that waits for the damage before it learns.

The flip side is not to over-do it: a postmortem for a trivial, well-understood blip with no systemic cause is busywork that dilutes the practice. Reserve the full ritual for incidents that actually have something to teach, define the threshold up front, and include near-misses that crossed it. The goal is consistent learning from events worth learning from, not a document for every hiccup.

One nuance that resolves most arguments about the threshold: severity should be set by impact, but postmortem depth should be set by surprise. A large outage from a well-understood cause with a known fix may warrant only a short write-up. A brief, minor incident whose cause nobody can explain deserves a deep one, because unexplained behaviour in production is a much stronger signal than magnitude. The question that decides depth is not “how bad was this” but “how much of this did we already understand?”

That framing also protects against the most common distortion, where postmortem effort tracks how visible the incident was to leadership rather than how much the team could learn from it. The incidents with the highest learning value are frequently the ones no executive noticed.

How do you run the postmortem meeting?

The document is not the practice; the conversation is. A few mechanics make the difference between a review that surfaces truth and one that performs it.

Circulate the draft timeline before the meeting. Reading a timeline aloud consumes the entire session and produces no analysis. Send the facts in advance so the meeting starts at “why,” not “what.”

Have someone other than the responder write it up. The person who fought the incident has the deepest context and the strongest unconscious bias toward the explanation they formed at 3 a.m. under stress. A second author asks the naive questions that surface assumptions.

Watch the language actively. Blamelessness erodes through phrasing before it erodes through policy. “Why did you deploy without checking?” and “what made it possible to deploy without a check?” seek the same information; only one of them keeps people honest next time. Correcting that phrasing in the moment is a genuine part of the staff engineer’s job in the room.

Invite the people who were not paged. Adjacent teams often hold the missing detail — a change they shipped, an alert they silenced, a dependency they know is fragile. Incidents rarely respect team boundaries even when the pager does.

End by assigning owners aloud. An action item without a name attached in the room reliably has no name attached a week later.

Spend real time on what went well. This gets treated as a morale formality and skipped when the meeting runs long, which wastes the most reusable output of the whole exercise. If a rollback took ninety seconds, that is a capability worth naming, protecting, and copying to services that lack it. If someone found the cause quickly because of a dashboard they built months earlier, that dashboard just justified its existence and probably deserves siblings. Incidents are the only honest test your response tooling ever gets, and the parts that held are as informative as the parts that broke.

An incident postmortem checklist

Before you call a postmortem done:

• The analysis is blameless, focused on systemic conditions, not individuals.
• There is a factual, timestamped timeline everyone agrees on.
• Root cause goes past the trigger to a condition a system change can fix.
• Detection and response gaps are named (how long to detect, what slowed mitigation).
• Action items are specific, owned, dated, and entered into the real backlog.
• What went well is captured, so good instincts are reinforced.
• The action items are actually prioritized, and someone tracks them to completion.

What I’d do differently

The failure I have watched most is the postmortem that is procedurally perfect and practically useless: a thorough document, a serious meeting, and a list of action items that quietly die in the backlog while the team moves on. The ritual was performed; the system did not change; the incident recurs.

If I were running postmortems again, I would spend less energy on the document and more on two things: protecting blamelessness ruthlessly so the truth comes out, and treating action items as real, prioritized work with owners and follow-through. A short postmortem whose fixes actually ship beats a beautiful one whose fixes never do. The point was never the write-up; it was a system that fails less next quarter than it did this one.

Sources

• Google SRE Book, Postmortem Culture: Learning from Failure: sre.google/sre-book/postmortem-culture
• Google SRE Workbook, Example postmortem and templates: sre.google/workbook/postmortem-culture
• Atlassian, Incident postmortem guide: atlassian.com/incident-management/postmortem