Over the past 15 years of working on and running infrastructure and engineering teams, Iâve led countless critical incidents affecting millions of users.
Iâve even been in incidents across a variety of industries. Some even generated memesâDiablo IIIâs Error 37 comes to mind. Not fun. But many of the biggest incident challenges werenât in debugging the problem and getting a fix out. They were about coordinating a team, getting the right people in the room, and keeping customers in the loop.
Itâs been many years now since I was running around a data center to find a spare server, but solving the unique challenges that arise in emergencies still excites me.
A few months back it hadnât occurred to me that I might be joining incident.io as VP Engineering but when Pete and I started talking about the next stage of the company's journey, I knew I had to join. Being a very happy customer made it all the more easy to make the decision.
Complementary, highly collaborative, and incredibly enthusiastic about the people they managed and the work they were doing. What else could I ask for?
Two weeks after joining and I wanted to take some time to take stock, write about why I joined and reflect on how thatâs marrying up with expectations.
The main reasons that stood out when deciding to join were: the team, the problem space, the product, and the pace.
I met with a wide range of people before I joinedâengineers, product managers, designers, data leaders and more. The calibre of every single individual was extremely high and the teamsâ customer obsession combined with their product focus made it clear that this was a special place.
Everyone had intelligent questions, a story about smart approach theyâd taken to solve a problem, a thorny issue they had to overcome, and all that was delivered with a healthy dose of pragmatism and humor.
My own experience interacting with the team as a customer had already given me glimpses of this: I once submitted a bug through Slack, had it acknowledged within minutes, and twenty minutes later another message alerted me to the fact that it was fixed.
I was always curious about how they managed this, and now Iâm excited to be a part of the culture behind it!
A clearly articulated vision helps drive a team more autonomously through purpose and Iâm a believer that management and leadership have an important role to play in the execution of that, too. I was particularly impressed and delighted when I met the Engineering Managers, Chris, and Alicia, and realized they had a magic of their own.
Complementary, highly collaborative, and incredibly enthusiastic about the people they managed and the work they were doing. What else could I ask for?
Dealing with incidents is hard. I know that incident.io makes handling them much easier.
Last but not least, I had worked with a few members of the team at GoCardless who I respected immensely, something that provided me with additional confidence that I wasnât just seeing things through rose-tinted glasses.
Thereâs nothing quite like the adrenaline of digging through a thorny problem to ensure that service comes back up as soon as possible, especially in an âalways connectedâ world, where customer expectations are higher than ever.
I will never forget the time when I was woken up to sort out a cluster promotion failure.
I went from bed, to washing my face, to figuring out there was no failover server in the disaster recovery site in a handful of minutes. I remember thinking, âWait, I think thereâs no physical server for the failover and thatâs why it failed. But that makes no sense. Did I miss some communication? Some decision and piece of work? Who do I escalate this to?â
A day later we learned someone had physically moved said server a few weeks before without broader communication and when this database server failed over, it wasnât able to recover.
Dealing with incidents is hard. I know that incident.io makes handling them much easier.
Another piece of the problem space that I find so captivating is learning through incidents. In case you think thatâs hyperbole, below is a picture I just took of my bookshelf to prove my incident-nerd credentials.
I particularly recommend Command and Control by Eric Schlosser for anyone looking to learn more about the complexity of the world, from operators to strategic decisions and how they affect the likelihood and the management of incidents. Itâs also far enough from any SaaS story that it wonât feel personal, but with lots of lessons to take.
The incident.io founders and team have this belief that delivering something comparable to the status quo isnât enough, which I share. This space is ripe for innovation and as youâll see from the rest of this post, I believe we're uniquely positioned to do just that.
Iâve been a fan of the product from its very early days. When I had to introduce some structure to incident management at a previous startup, Pete, Chris, and Stephen had just started incident.io. I immediately saw the value, and soon we were their first paying customer.
It is quite incredible the number of high-quality features built and how much polish they added to the product in such a short amount of time.
At this point, the product had the absolute barest bones to be called an incident management tool though. You could declare incidents through Slack and ... that was about it!
If youâre wondering, that top navbar is now a sidebar including Incidents, Alerting, Status Pages, Insights, Catalog, Workflows and more.
It is quite incredible the number of high-quality features built and how much polish they added to the product in such a short amount of time. Every time I used the product, I came out of it feeling like incident.io was increasingly the obvious way to manage incidents in this modern, changing world.
The pace has previously been blogged about by one of our product engineers, Macey.
One would be forgiven for thinking that making the product high quality and feature rich, would make the team be slow, or overthink whatâs coming next.
The Changelog paints a different picture.
It shows me that even though the product is high quality and full of features, it also evolves rapidly, with lots of bugs fixed, enough improvements to make your head spin and a collection of magical product moments.
All of this happens at least daily. Not monthly, not quarterly or weekly, daily!
From engineers, product managers, to customer success, and sales, no one stops at, âIt's good enough.â
Last year, the team set themselves a goal of a Changelog post every week, which impressively, they exceeded! The things they shipped were great, but this level of commitment and consistency is also extremely impressive on itâs own. It signals a company dedicated to excellence and innovation in the space; and ultimately itâs a reflection of the trajectory and pace of delivery.
In my first week, and while I was meeting people, teams delivered better AI suggestions (AI now knows what retrospective incidents are!), dozens of improvements and bug fixes, and more. See for yourself!
Joining such a brilliant, cohesive, product-focused team was a no brainer. Iâve only been here a couple weeks but have already found the team truly inspiring.
Iâm still blown away by the pace and customer focus from everyone: not just the product managers and customer success folks youâd normally expect. âNot my jobâ arenât words Iâve heard. Every customer, new deal and product release matters.
Listening to customers, delighting them on a daily basis, and sprinkling âa-haâ moments in very serendipitous ways. I can think of the first time I heard about their Catalog and immediately went, âOh wow, can this now be used as part of Workflows?â Only to be told, âYes! âŠand insights, and triggersâŠâ
From engineers, product managers, to customer success, and sales, no one stops at, âIt's good enough.â
Everyone stops at, âWeâve made it magical."
Iâm very excited, and humbled to be part of the journey, and the future of it.
Going from 0 to shipping 2 features by the end of my first week
6 months in, Martha reflects on why she chose to join the engineering team of an early stage start-up, and what sheâs learnt since.
Introducing our AI-powered features to enable incident responders to save crucial time, learn from previous incidents, and become more resilient over time.
Ready for modern incident management? Book a call with one of our experts today.