---
title: "How to do root cause analysis: A step-by-step process for incidents"
description: "Root cause analysis steps: capture the timeline automatically, find proximate cause, isolate system failure, fix causes not symptoms."
url: "https://incident.io/blog/how-to-do-root-cause-analysis"
date: "2026-10-28T12:47:41.208Z"
---

# How to do root cause analysis: A step-by-step process for incidents

*October 28, 2026*

> **TL;DR:** Root cause analysis is most likely to go wrong when you reconstruct the timeline from memory or Slack scroll-back. Capture the timeline automatically during the incident, then apply the five-step RCA process this guide adapts for incident work. The process separates proximate from root causes, so you fix the system instead of the symptom. Run a full RCA on P0 and P1 incidents; use a lighter review for P2, P3, and near-misses. incident.io auto-captures the timeline and gives you a post-mortem draft built from it, cutting slow manual reconstruction down to minutes. Use the checklist at the end on your next incident.

Post-mortems slow down when the timeline has to be rebuilt from Slack threads after the incident is over. Root cause analysis isn't hard because the causes are complex. It's hard because the evidence is often scattered across tools like PagerDuty, Datadog, Slack, and Jira, plus whatever lives in one engineer's memory. This guide walks through a repeatable RCA procedure that starts with an automatically captured timeline, follows five structured steps to separate proximate from root causes, and ends with a blameless post-mortem finished in minutes instead of hours spent reconstructing events from memory.

## Defining root cause analysis for incident response

Root cause analysis (RCA) is a structured process for identifying the underlying system failure that allowed an incident to occur, not just the immediate technical fault that triggered it. [Google's SRE Book](https://sre.google/sre-book/postmortem-culture/) sets the goal: "to ensure that the incident is documented, that all contributing root cause(s) are well understood, and, especially, that effective preventive actions are put in place to reduce the likelihood and/or impact of recurrence." For SREs carrying a pager through weekly on-call rotations, RCA is how you stop fighting the same fire twice.

### Distinguishing root from proximate causes

The proximate cause is the immediate operational fault, for example an exhausted connection pool, a changed config value, or a deploy that introduced a bug. The root cause is the latent system condition that let that fault become an outage, such as no connection pool monitoring, no config validation in CI, or no capacity alert at 80% utilization.

The [SRE Book example postmortem](https://sre.google/sre-book/example-postmortem/) separates the two. Its trigger is a latent bug triggered by a sudden increase in traffic. Its root causes are the conditions that turned that bug into a cascading failure: exceptionally high load combined with a resource leak. A post-mortem that names only the trigger fixes one instance and leaves the conditions in place. Restart a service after its connection pool runs out, and three weeks later the pool runs out again because nothing monitors it.

| Proximate cause | Root cause |
| --- | --- |
| Connection pool exhausted | No connection pool metrics or alert thresholds configured |
| Deploy changed a config value | No config validation step in CI |
| Retry storm overloaded a downstream service | No retry limits or backoff in the client library |

Teams stop at the proximate cause because it's easier to find and it feels like a fix. The root cause requires asking why the system allowed that event to cascade.

### Improving reliability with better RCA

Action items that address system failures rather than symptoms are how you reduce the likelihood of recurrence. Teams that treat RCA as a documentation exercise end up with post-mortems nobody reads and incidents that keep repeating. The fix is structural: capture the evidence automatically, follow a consistent process, and track the action items to completion. The data collection step is where the time goes.

## Deciding when to run a root cause analysis

Not every incident needs the full five-step RCA procedure. The depth of analysis should match the severity of the incident and the potential for recurrence. [Google's SRE Book](https://sre.google/sre-book/postmortem-culture/) lists common post-mortem triggers, and notes that teams have some internal flexibility to adapt them. Treat them as a starting point, not a hard requirement: any incident with user-visible downtime or degradation beyond a threshold, data loss of any kind, on-call engineer intervention (a rollback, a traffic reroute), resolution time above a threshold, or a monitoring failure.

### Defining P0 and P1 incident scope

Severity scales vary by team. In a typical scheme, P0 incidents are critical, customer-facing outages that displace all other work, while P1 incidents are major degradations that displace planned work for the owning team. Run a full RCA on both, with a complete timeline, documented root causes, contributing factors, and tracked action items. Give P2 and P3 incidents a lighter-weight review, or group them for pattern analysis. Match the depth of the review to the cost of the incident repeating.

### Analyzing common incident signatures

Some incident patterns warrant RCA even at lower severity, because they signal systemic issues building toward a bigger failure:

* **Repeated failures in the same service:** Multiple P3s in the payments service in a month signal a pattern worth investigating before it escalates.
* **Self-resolving failures:** An incident that cleared without intervention means you don't know why it recovered, which means you can't count on it recovering next time.
* **Error budget burn approaching SLO limits:** When your error budget burns faster than expected, the next incident is already forming.
* **Log warnings that precede outages:** A warning that shows up before each outage is an early signal you can alert on.

### Running root cause analysis on near-misses

A near-miss is an incident caught before customer impact, or a failure that self-corrected. These are the cheapest RCA opportunities you get, because the system already told you something is wrong and you can fix it before it becomes a P0.

A connection pool that hit 90% capacity but recovered on its own is a near-miss. A deploy that caused elevated error rates for 90 seconds before the canary analysis rolled it back is a near-miss. Run a lightweight RCA (a timeline plus a short causal chain) for these events rather than no analysis at all. The alternative is waiting for the near-miss to become a real incident and doing the same analysis under pressure.

## Finding incident root causes in five steps

This five-step process maps to how SRE teams work through an incident. The steps are stable, but the speed and accuracy of each step depend entirely on the quality of the evidence you feed them.

1. **Build the timeline.** The timeline is the foundation. Without it, every subsequent step is guesswork. A good timeline contains alert firings, severity changes, role assignments, key Slack messages, deploy events, and config changes. Reconstructing this from Slack scroll-back is slow, and the record comes out incomplete because details fade once the incident is over.
2. **Find the proximate cause.** This is the immediate operational fault: what broke first. Identify it from the timeline by looking for the first anomalous event, the first alert, the first error spike. Don't stop here.
3. **Isolate the root cause.** Ask what system allowed the proximate cause to occur. What was missing: monitoring, alerting, validation, capacity planning? This step separates proximate from root cause. The 5 Whys technique works here, but with structure: each "why" should point to a system gap, not a person's mistake.
4. **Look past symptoms.** Turn each root cause into a system-level action item. Restarting the service treats the symptom. Adding pool metrics and an alert threshold fixes the cause, so you get an alert at 70% utilization before the pool runs out.
5. **Track contributing factors.** Contributing factors made the incident worse or longer without causing it: the page went unacknowledged for 10 minutes, the runbook was outdated, the escalation path was unclear. These go in the post-mortem as separate action items, not as root causes.

| Step | Question it answers | Output |
| --- | --- | --- |
| Build the timeline | What happened, in what order? | Chronological event list |
| Find the proximate cause | What broke first? | The triggering fault |
| Isolate the root cause | What system gap allowed it? | The latent condition |
| Look past symptoms | What fix makes recurrence less likely? | System-level action items |
| Track contributing factors | What made it worse or longer? | Process-level action items |

## Building incident timelines automatically

The timeline is where RCA succeeds or fails. A complete, accurate timeline speeds up the analysis. An incomplete one leaves gaps in every step after it.

### Consolidate alert and log history

The manual approach means pulling alert history from PagerDuty, logs from Datadog, and deploy events from GitHub, then correlating timestamps across those tools by hand. This can take longer than the analysis itself and often misses the Slack message that connects this incident to a pattern, buried in a thread you'll never find three days later.

### Automate incident timeline assembly

[incident.io builds the timeline](https://docs.incident.io/articles/7767182003-generating-your-incident-timeline) from the moment the alert fires. A Datadog alert triggers incident.io, which creates a dedicated Slack channel, pages on-call responders, and starts the timeline without anyone leaving chat. When you type `/inc severity P1`, the timeline records the severity change with a timestamp. When you pin a key message, the timeline captures it. Role changes such as `/inc role lead @sarah` sit in the incident activity log, and responders can add them to the timeline from there. On the Pro or Enterprise plan, once you [turn Scribe on](https://docs.incident.io/ai/scribe), it joins incident calls and posts key decisions to the incident channel as they happen, so the verbal "let's roll back the deploy" gets written down instead of living only in memory. Pin that message and it lands in the timeline.

> "It removes all friction and lets engineers focus on the incident while the integration handles updating the status page and taking notes for post mortems." - [Jérémy C. on G2](https://www.g2.com/products/incident-io/reviews/incident-io-review-13203870)

### Keep incident and configuration history

For RCA, you need two kinds of history. The [incident activity log](https://docs.incident.io/articles/7767182003-generating-your-incident-timeline) captures day-to-day incident work, such as role changes, summary updates, actions, GitHub PRs and commits. It has an API, so you can pull history into your own systems for pattern analysis. On the Enterprise plan, incident.io's [audit log](https://docs.incident.io/admin/audit-logs) captures configuration, permission, and security changes with 12-month retention, CSV export, and SIEM streaming to Datadog, Splunk, or S3. The activity log gives you the history of the incident itself. The audit log shows who changed your incident.io setup, such as a workflow, a role, or a permission.

### Build incident history automatically

Automatic timeline capture builds searchable incident history over time. When similar incidents occur, you pattern-match against past timelines instead of starting from scratch. [Investigations handles the first 80%](https://incident.io/resolve-incidents-while-you-sleep) of incident response, analyzing telemetry, code changes, and past incidents to surface likely root causes and draft fix PRs. Investigations is a paid add-on available on Pro and Enterprise and requires Response.

## Drafting a blameless post-mortem in minutes

The post-mortem is where the RCA process produces its output. A [blameless post-mortem](https://incident.io/blog/sre-incident-postmortem-best-practices) documents what happened, why it happened, and what you're doing about it, without assigning fault to any individual.

### Writing a blameless report

Blameless means the report focuses on system failures and process gaps, not individual mistakes. The language shift is direct: "the on-call engineer forgot to check the pool size" becomes "the runbook did not include a pool size check." [Google's SRE Book](https://sre.google/sre-book/postmortem-culture/) says a blameless post-mortem identifies contributing causes without indicting any individual or team, because you can fix systems and processes but you can't "fix" people. On the Pro and Enterprise plans, incident.io drafts the post-mortem automatically on resolution, pulling from the timeline, Slack threads, PRs, and custom fields. You edit and refine the draft instead of reconstructing from memory.

> "Our team also appreciated the AI features, such as postmortem generation, incident interactions that help manage the full lifecycle, and the integration with the Scribe agent." - [Guilherme R. on G2](https://www.g2.com/products/incident-io/reviews/incident-io-review-13182654)

### Tracking action items after incidents

[Action items](https://incident.io/blog/why-post-mortem-action-items-die) are the point of the post-mortem. They're what reduces the chance of recurrence. Every action item needs a named owner (a person, a team, or both) and a due date. Add a priority level if you enforce follow-ups by priority, as below. incident.io's [Jira integration](https://docs.incident.io/integrations/jira) exports follow-ups from Slack or the dashboard, then syncs each ticket's title, assignee, and completion back to incident.io. The [Follow-ups API](https://docs.incident.io/api-reference/follow-ups-v3) lets you pull follow-up data into your own dashboards.

For teams that need enforcement, you can set a [follow-up policy by priority](https://docs.incident.io/admin/enforcing-follow-ups-are-completed-based-on-priority), for example requiring urgent follow-ups to be completed within 3 days of resolution. Tag incidents by service, root cause category, or contributing factor to enable pattern analysis across incidents. The Insights dashboard shows Mean Time To Resolution (MTTR) trend and incident-pattern data, giving you a concrete answer when leadership asks if the SRE function is working.

## Diagnosing connection pool exhaustion: A worked example

**Hypothetical example:** The incident below is illustrative, not a real customer incident. The channel name, timestamps, and percentages are placeholders that show the pattern, not sourced figures. Here's the five-step process applied to it, with incident.io building the timeline as the team responds.

### Mapping the incident timeline

The timeline for this example:

* **First alert:** Datadog alert fires: connection pool at 95% on the payments service
* **1 minute later:** incident.io creates `#inc-2847-payments-pool` and pages the on-call engineer
* **4 minutes in:** Severity set to P1 with `/inc severity P1`
* **8 minutes in:** Engineer pins the error "Connection pool exhausted: max=200, active=200, idle=0"
* **11 minutes in:** Second responder joins, then shares and pins a Datadog graph showing connection count climbing for the past 35 minutes
* **12 minutes in:** Responder adds the deploy, which went out about half an hour before the first alert, as a custom timeline event
* **16 minutes in:** Service restarted, pool resets
* **21 minutes in:** Incident resolved with `/inc resolve`

### Pinpointing the failure source

The proximate cause is the connection pool exhaustion. In this example, a deploy about half an hour before the first alert shipped a feature that increased concurrent connections by roughly 40%. The pool then climbed past 95% and ran out. The pinned Datadog graph shows when the climb started, and the deploy sits in the timeline as a custom event, so the team can line the two up straight away.

### Conducting a root cause analysis

Applying the five steps:

1. **Build the timeline:** Captured during the incident, with the deploy added as a custom event.
2. **Find the proximate cause:** Connection pool exhausted because the new feature increased concurrent connections by roughly 40%.
3. **Isolate the root cause:** The only pool alert fired at 95%, minutes before exhaustion, so the team had no early warning. The deploy review process didn't include a connection capacity check.
4. **Look past symptoms:** Restarting the service fixed the symptom. Earlier alert thresholds and a deploy capacity check fix the cause.
5. **Track contributing factors:** The on-call runbook didn't include pool metrics. The escalation path for database issues was unclear, adding roughly 4 minutes to the response.

### Implementing long-term incident fixes

The action items from this RCA:

* Set alert thresholds, for example at 70% and 85% pool utilization (owner: SRE team, due: this week)
* Add a connection capacity check to deploy review (owner: platform team, due: next sprint)
* Update the runbook with pool size checks (owner: on-call lead, due: this week)
* Clarify the database escalation path (owner: engineering manager, due: next sprint)

The outcome: the next time the pool approaches capacity, you get an alert at 70%, investigate before exhaustion, and the incident never becomes a P1. This is the difference between RCA as documentation and RCA as a reliability investment.

According to [Favor's engineering team, MTTR dropped 37%](https://incident.io/customers/favor) after adopting incident.io. Before the switch, spinning up an incident took Favor's team 20-30 minutes of manual setup, such as creating a channel and pulling in engineers. incident.io automates that channel creation and responder paging, so troubleshooting starts sooner.

## Using a root cause analysis checklist on your next incident

Use this checklist on your next incident as a quick reference for any on-call engineer.

### Confirming the analysis is complete

When you finish the five-step process, confirm:

* **Timeline captured:** Is the incident timeline complete, from first alert to resolution?
* **Proximate cause identified:** What broke first?
* **Root cause isolated:** What system failure allowed the proximate cause to occur?
* **Contributing factors listed:** What made the incident worse or longer, without causing it?
* **Action items assigned:** Does each item have a named owner and a due date?

### Drafting the final incident report

* **Timeline section:** Pasted from the captured timeline, not reconstructed from memory.
* **Root cause section:** Describes the system failure, not the individual mistake.
* **Contributing factors section:** Lists what made it worse, without blame.
* **Action items section:** Each item has a named owner and a due date.
* **Blameless language:** No individual names attached to failures. System and process gaps named instead.

## Making root cause analysis a habit

Root cause analysis works when the evidence is complete before the analysis starts. Capture the timeline during the incident, separate the proximate cause from the system gap that allowed it, and give every action item a named owner with a due date. Repeat it on every P0 or P1 so each fix targets the system instead of the symptom.

If you're spending more time reconstructing timelines than analyzing causes, [book a demo of incident.io](https://incident.io/demo) and see your next incident's timeline captured automatically, with a post-mortem drafted from it in seconds.

## Key terms glossary

**Root cause:** The system failure that allowed the proximate cause to occur. Fixing the root cause reduces the chance of recurrence. Fixing the proximate cause only treats the symptom.

**Proximate cause:** The immediate operational fault that triggered the incident, such as an exhausted connection pool or a bad config deploy.

**Contributing factors:** Conditions that made the incident worse or longer without causing it, such as an outdated runbook or an unclear escalation path.

**Post-mortem:** A written record of an incident's timeline, root cause, contributing factors, and action items. A blameless post-mortem focuses on system failures, not individual mistakes.

**MTTR:** Mean Time To Resolution. The average time from incident detection to resolution, a key reliability metric.

**SLO:** Service Level Objective. A measurable reliability target for a service, such as 99.9% availability, that error budgets and alert thresholds tie back to.

**SIEM:** Security Information and Event Management. A platform that aggregates security and event data for analysis, such as Splunk or Datadog.