Further reading

Level up your incident response with incident.io

Learn more
Fostering a transparent culture

War story – The power of transparency

Before I was an engineer, I used to work in consulting. I was on a project with a client who was notoriously tough. I represented our team at the weekly update meeting where I talked through what had happened during the week and where we were against our goals.

One week, someone in my team made a mistake, and it lost us a couple of days on a really time-sensitive part of the project. I turned up to the weekly update meeting, pretty nervous, and delivered the bad news.

I gave a detailed explanation of what had happened, why that lost us time, and then what steps we’d taken to be confident that a similar error wouldn’t happen again.

I was honest and I put my hands up - we had made a mistake, and that meant our project would be delayed. It was our fault - no one else’s.

In that same meeting, there was another supplier with a similar problem, but they were evasive when the client tried to find out more. They either didn’t understand what had actually gone wrong, or wasn’t willing to discuss it. Our client grew increasingly frustrated, almost to the point of being angry. They asked more questions: How do you know this won’t happen again? What happens if there are further delays? How confident are you that you can meet the new deadline?

At the end of the meeting, the client was unhappy, and asked the other supplier to send them a detailed report of exactly what had happened and what steps had been taken to prevent similar issues in the future. We didn’t receive the same request.

That meant that while we were getting on with the next task, the other supplier was having to gather information and write this report. In all likelihood, the only thing that achieved was to make the client feel better about the situation.

~ Lisa Karlin Curtis

Why transparency is important #

Transparency gives you breathing room to fix what’s actually broken #

When you communicate clearly, it’s easier for others to understand the problem and make good choices about when it’ll be useful for them to contribute.

Instead of spending lots of time persuading people that you’re doing the right thing (and building trust), you can invest that effort into whatever it is you’re supposed to be doing.

Transparency enables teams to trust each other. If teams haven’t built that trust, incidents become a black hole that sucks more and more people into them. As each new team is impacted, they too want to have a seat at the table so they can be sure it gets managed ‘correctly’ and their interests are represented.

Transparency builds relationships #

When things go wrong, people are challenged and often stressed. But it’s also the perfect opportunity to build relationships.

Being transparent when things are going wrong, taking the time to explain why something failed and what you’ve done, as a result, pays dividends both in future incidents and also outside.

Finding boundaries between teams is difficult: usually there’s lots of overlap between the various parts of your organisation. That means, to be successful, teams need to get really good at collaborating and relying on each other. If other teams don’t think you’re good at what you do, or don’t think your goals are aligned with theirs, these initiatives become almost impossible to deliver.

Why transparency is difficult #

Transparency is not a human default. When things go wrong, we all experience fear.

Is something bad about to happen? Has it already?
Could the company be in trouble? Am I in trouble?

Even once that has subsided, and it’s clear the sky isn’t falling, there’s still fear left over.

Did I mess up here? What will people think of me?

Humans feel shame incredibly acutely. Evolutionarily speaking, we’re not built to survive on our own - we needed to be part of a group. We might like to think we’ve outgrown those instincts, but there’s lots of evidence to suggest that’s not true. This fear is what makes it so uncomfortable to admit your mistakes, and be honest about what’s happening.

In reality, the opposite is true. Being honest and admitting your mistakes will build social capital. People will respect you more for being honest, and trust you more in the future. To foster a transparent culture, we need to build a team where everyone feels comfortable admitting to their mistakes, and being open when things go wrong.

Fostering a transparent culture #

Be explicit #

  • Talk about blamelessness - what that means for your company and for your team.
  • Explain in your onboarding materials how you think about blame. It can be useful at a system-level - perhaps to try and understand the root cause of an incident. But at an individual level, as a mechanism for assessing your colleagues, it’s not productive.
  • Share reading materials – maybe a link to this guide 😉 – or the numerous great articles that have been written on the subject.

Fail together #

The best teams succeed together, and fail together.

It’s very rare that something is solely one person’s fault: technology is usually more complicated than that.

War Story

A junior engineer is working on a new feature that sends emails to customers who aren’t paying, asking them to set up their billing. Their code isn’t quite right - it emails all the customers who are paying.

They deploy their code to staging and run it, just as they’ve been told to do as ‘good practice’ for this kind of feature.

Suddenly, customer support are inundated with questions. Why are you asking me to pay when I’m already paying? Is something wrong? Should I update my details?

The team gets involved, they write some comms to tell support what’s happened - that it’s all a misunderstanding - and the support team gets to work responding to the confused customers.

The junior engineer is mortified. How have they made such a foolish mistake?

It turns out that the staging environment had recently been seeded with production data, to enable some load testing that another team wanted to do.

This isn’t the engineer’s fault. This has happened because of a number of mistakes conspiring together to this unfortunate outcome.

Why was staging seeded with production data? Why wasn’t that data anonymised?
Should staging be set up to send unlimited emails to arbitrary email addresses?

When something goes wrong, it’s important to take accountability as a group, rather than either leaving it with a single team member.

This doesn’t mean ‘everything is fine’. Something bad has happened, and we need to learn from it. But we’ll do that as a team, together.

Example setting matters #

When anyone joins a company, they are heavily influenced by the people around them - particularly senior folks, or those who’ve been at the company for a long time.

If you’re one of those people, you’re in a unique position to set an example. Shout about the mistakes that you make - it’ll help shatter the illusion of perfection.

Additionally, be careful about how you talk about other people’s mistakes. War stories are popular – there’s plenty in this guide – and they’re a nice way to share some company history, but there’s a delicate balance the struck. Focus on stories about teams, not individuals, and (hopefully obviously) always avoid being unkind, even if it’s to someone you know well!