A guide to post-mortem meetings and how we run them at incident.io

You've just made it through a particularly tough incident.

It was a short outage affecting a subset of customers, so not exactly the end of the world, but bad enough that it involved multiple people across a number of teams to resolve. Either way, the incident was well managed, and the dust has settled.

Now what?

Most guidance would say that putting together a post-mortem document is a good idea, given the severity of the incident. You've also done this, so what's next?

It's time to prepare and run your post-mortem meeting! But it isn't enough to go into a room with the people involved and wish for the best. No, there are a few things that you need to keep in mind before, during, and after your post-incident meeting to get the most out of it.

What is a post-mortem meeting?

An incident post-mortem meeting is held after an incident. This is a space where a small group of responders and other stakeholders discuss the incident, its contributing factors, and its impacts.

The goal is to go beyond the post-mortem document and discuss the incident in a live setting. It's much less reactive and focused on creating better long-term outcomes.

Setting the scene for a productive post-incident meeting

To facilitate a productive post-mortem meeting, it's important to establish a few ground rules. These can help promote open and honest discussion, give everyone equal time to raise any concerns and prioritize addressing critical issues.

• Set the tone of blameless accountability: First, creating an environment of blameless accountability is crucial since it allows for a focus on understanding contributing factors rather than finger-pointing anyone. Remember—if discussing the actions of a specific person is being done for the sake of better learning; don't shy away from it.
• Take notes: Second, staying on topic and taking good notes during the meeting helps maintain focus and makes capturing important details a bit easier. Pro tip: the person running the meeting should not be the person taking notes. Their focus should be as a facilitator.
• Set expectations for learning: Lastly, setting expectations at the beginning of the meeting, emphasizing learning and improvement, and promoting open and respectful discussion can help create a positive and productive environment for everyone.

General guidance for running post-mortem meetings

There’s a myriad of ways you can run a post-mortem meeting, but let’s look at some best practices that most folks agree are a good starting point.

Get the right people together

First things first—it's important to include all relevant stakeholders in the post-mortem meeting, including any incident responders, key decision-makers, and individuals directly involved in the incident. The goal is to make sure that every perspective and nuance is accounted for. Remember, everyone will have a unique take on the incident worth factoring in.

Have an agenda

Before the meeting, set a clear agenda outlining some of the objectives and topics you want to discuss as a team. This helps keep the meeting focused and ensures that all agreed-upon talking points are covered.

It doesn't need to be a rigid agenda, and it's entirely expected (and encouraged!) that you'll end up talking about other topics, but going in with some pre-work and thoughts on areas of interest can be incredibly helpful.

Your post-mortem document is a good place to start!

Review your incident, preferably using a post-mortem document

During the meeting, the incident should be reviewed from end to end, including the timeline of events, the impact on the organization, and the actions taken to mitigate the incident. This allows the team to gain a complete understanding of what happened.

Discussion should be actively encouraged during the review. Remember, the goal is to learn, not recite the contents of the document. If people are less forthcoming with contributions, it can be helpful to ask open questions to gather additional perspectives.

Note any contributing factors

Next, try to identify any contributing factors to the incident. This involves analyzing technical issues, human error, or process gaps. By identifying these, teams can implement process and technical improvements to help prevent similar incidents in the future.

Discuss lessons learned

The post-mortem meeting should provide an opportunity for open and honest discussions about the lessons learned from the incident. This includes identifying areas for improvement, discussing best practices for resolving incidents like these, and sharing knowledge among team members.

Align on follow-up actions

Based on the insights gained from the meeting, a follow-up plan should be created to address the identified issues and prevent future incidents. This plan should include specific tasks, owners, and timelines.

How we run post-mortem meetings at incident.io

Here at incident.io, we run a debrief meeting (our name for a post-mortem) for every major or critical incident that we respond to. We also occasionally run them for lower-severity incidents too, at the discretion of the incident lead or by request from anyone involved.

For less severe incidents, it's common for the lead to write the postmortem and run the debrief. For bigger incidents, someone else, like a tech lead, will run the debrief. This frees up other key players to contribute to the conversation.

We also record the meetings using Google Meet, so they’re available for future reference and to share with anyone interested. This is a great way for folks to benefit from hearing the conversation in the room, without needing to inflate the invite list.

To make it easier to prepare for the meeting, we use the post-mortem template built into incident.io, which pulls specific information from the incident into a debrief document that we use to steer our conversation.

That said, we don't require a super polished post-mortem document before running the debrief. One of our company values is to trust by default, so we let folks choose what’s important to include and go from there.

When going through the meeting, this is the structure we usually stick with:

1. The facilitator walks through the post-mortem document, i.e., briefly summarizes the incident and walks through the timeline.
2. The discussion happens, normally based on a set of "talking points" or "learning points."
3. New follow-up actions are created during the discussion to make sure we’re applying our learnings and building better resilience.
4. After the meeting, the follow-ups are completed or pushed into an issue tracker, and the notes are sent around to all relevant stakeholders

We're good at stopping ourselves from going into rabbit holes or overrunning during these meetings. If it looks like something needs to be investigated or a particular problem needs to be solved, we'll create a follow-up for someone to keep the session focused on learning.

But we also make sure that we aren’t ever “overcorrecting” either. If we’re setting any follow-up actions, we aim for them to be proportionate to the severity of the incident.

In the end, what we find most important is having the space to discuss the incident at length. The goal is always to come out of the meeting with a shared understanding of how an incident happened and an agreement on whether there are any follow-up actions we need to take—and who’s doing them.

For us, this is what a successful post-incident meeting looks like.