Over the past 15 years of working on and running infrastructure and engineering teams, I’ve led countless critical incidents affecting millions of users.
I’ve even been in incidents across a variety of industries. Some even generated memes—Diablo III’s Error 37 comes to mind. Not fun. But many of the biggest incident challenges weren’t in debugging the problem and getting a fix out. They were about coordinating a team, getting the right people in the room, and keeping customers in the loop.
It’s been many years now since I was running around a data center to find a spare server, but solving the unique challenges that arise in emergencies still excites me.
A few months back it hadn’t occurred to me that I might be joining incident.io as VP Engineering but when Pete and I started talking about the next stage of the company's journey, I knew I had to join. Being a very happy customer made it all the more easy to make the decision.
Complementary, highly collaborative, and incredibly enthusiastic about the people they managed and the work they were doing. What else could I ask for?
Two weeks after joining and I wanted to take some time to take stock, write about why I joined and reflect on how that’s marrying up with expectations.
The main reasons that stood out when deciding to join were: the team, the problem space, the product, and the pace.
The team
I met with a wide range of people before I joined—engineers, product managers, designers, data leaders and more. The calibre of every single individual was extremely high and the teams’ customer obsession combined with their product focus made it clear that this was a special place.
Everyone had intelligent questions, a story about smart approach they’d taken to solve a problem, a thorny issue they had to overcome, and all that was delivered with a healthy dose of pragmatism and humor.
My own experience interacting with the team as a customer had already given me glimpses of this: I once submitted a bug through Slack, had it acknowledged within minutes, and twenty minutes later another message alerted me to the fact that it was fixed.
I was always curious about how they managed this, and now I’m excited to be a part of the culture behind it!
A clearly articulated vision helps drive a team more autonomously through purpose and I’m a believer that management and leadership have an important role to play in the execution of that, too. I was particularly impressed and delighted when I met the Engineering Managers, Chris, and Alicia, and realized they had a magic of their own.
Complementary, highly collaborative, and incredibly enthusiastic about the people they managed and the work they were doing. What else could I ask for?
Dealing with incidents is hard. I know that incident.io makes handling them much easier.
Last but not least, I had worked with a few members of the team at GoCardless who I respected immensely, something that provided me with additional confidence that I wasn’t just seeing things through rose-tinted glasses.
The problem space
There’s nothing quite like the adrenaline of digging through a thorny problem to ensure that service comes back up as soon as possible, especially in an “always connected” world, where customer expectations are higher than ever.
I will never forget the time when I was woken up to sort out a cluster promotion failure.
I went from bed, to washing my face, to figuring out there was no failover server in the disaster recovery site in a handful of minutes. I remember thinking, “Wait, I think there’s no physical server for the failover and that’s why it failed. But that makes no sense. Did I miss some communication? Some decision and piece of work? Who do I escalate this to?”
A day later we learned someone had physically moved said server a few weeks before without broader communication and when this database server failed over, it wasn’t able to recover.
Dealing with incidents is hard. I know that incident.io makes handling them much easier.
Another piece of the problem space that I find so captivating is learning through incidents. In case you think that’s hyperbole, below is a picture I just took of my bookshelf to prove my incident-nerd credentials.
I particularly recommend Command and Control by Eric Schlosser for anyone looking to learn more about the complexity of the world, from operators to strategic decisions and how they affect the likelihood and the management of incidents. It’s also far enough from any SaaS story that it won’t feel personal, but with lots of lessons to take.
The incident.io founders and team have this belief that delivering something comparable to the status quo isn’t enough, which I share. This space is ripe for innovation and as you’ll see from the rest of this post, I believe we're uniquely positioned to do just that.
The product
I’ve been a fan of the product from its very early days. When I had to introduce some structure to incident management at a previous startup, Pete, Chris, and Stephen had just started incident.io. I immediately saw the value, and soon we were their first paying customer.
It is quite incredible the number of high-quality features built and how much polish they added to the product in such a short amount of time.
At this point, the product had the absolute barest bones to be called an incident management tool though. You could declare incidents through Slack and ... that was about it!
If you’re wondering, that top navbar is now a sidebar including Incidents, Alerting, Status Pages, Insights, Catalog, Workflows and more.
It is quite incredible the number of high-quality features built and how much polish they added to the product in such a short amount of time. Every time I used the product, I came out of it feeling like incident.io was increasingly the obvious way to manage incidents in this modern, changing world.
The pace
The pace has previously been blogged about by one of our product engineers, Macey.
One would be forgiven for thinking that making the product high quality and feature rich, would make the team be slow, or overthink what’s coming next.
The Changelog paints a different picture.
It shows me that even though the product is high quality and full of features, it also evolves rapidly, with lots of bugs fixed, enough improvements to make your head spin and a collection of magical product moments.
All of this happens at least daily. Not monthly, not quarterly or weekly, daily!
From engineers, product managers, to customer success, and sales, no one stops at, “It's good enough.”
Last year, the team set themselves a goal of a Changelog post every week, which impressively, they exceeded! The things they shipped were great, but this level of commitment and consistency is also extremely impressive on it’s own. It signals a company dedicated to excellence and innovation in the space; and ultimately it’s a reflection of the trajectory and pace of delivery.
In my first week, and while I was meeting people, teams delivered better AI suggestions (AI now knows what retrospective incidents are!), dozens of improvements and bug fixes, and more. See for yourself!
I’m really excited for the future
Joining such a brilliant, cohesive, product-focused team was a no brainer. I’ve only been here a couple weeks but have already found the team truly inspiring.
I’m still blown away by the pace and customer focus from everyone: not just the product managers and customer success folks you’d normally expect. “Not my job” aren’t words I’ve heard. Every customer, new deal and product release matters.
Listening to customers, delighting them on a daily basis, and sprinkling ‘a-ha’ moments in very serendipitous ways. I can think of the first time I heard about their Catalog and immediately went, “Oh wow, can this now be used as part of Workflows?” Only to be told, “Yes! …and insights, and triggers…”
From engineers, product managers, to customer success, and sales, no one stops at, “It's good enough.”
Everyone stops at, “We’ve made it magical."
I’m very excited, and humbled to be part of the journey, and the future of it.
