The launch of On-call was an integral part of the incident.io mission to become the single place you turn when things go wrong, and recently we hosted a live virtual event to show how it all came together.
In this event, incident.io Co-founder and CTO Pete Hamilton sat down with incident.io Product Manager Megan McDonald, Product Engineer Rory Bain, and fellow Co-founder and CPO Chris Evans to demo the product, discuss the journey of the creation, and expand on what’s next.
Read on for the high-level overview, but if you want the full context, check out the video in its entirety.
The beginning
On-call was conceived from an experience shared by many, one described by Hamilton as the “3 AM chaos when you wake up and there’s a myriad of tools and processes and people, everything’s a little bit on fire.”
For the incident.io founders, this was a familiar pain prevalent in previous engineering roles – and one echoed by incident.io customers, as the number one feature request was consistently to build an on-call product.
“[Some are] probably thinking this is a bit confusing as to why we didn't [build it], and the answer is we just didn't feel ready, we wanted to really, really own and understand this response space,” said Hamilton.
Initially a small group of ten , the incident.io engineering team knew building an on-call tool of their desired scale was a question of when, not if. As the team–and the company–began to expand, building the product soon began–quietly.
Fast forward, and early reception of On-call has been very positive, and reassured the team that the time spent working on the level of attention to detail and care into the customer experience paid off, Hamilton said.
On-call: Past and future
As the On-call team technical lead, Product Engineer Rory Bain has spent his fair share of time on the product.
In the six months spent preparing for launch, Bain and the engineering team were focused on making sure there were no surprises. This meant working closely with customer beta testers, dogfooding the product constantly, and spending lots of time working on on-call foundations, like scheduling or escalation paths.
It wasn’t until the end that the team was able to build the things they were most excited about, like announcing shift changes to Slack. It was fun, Bain said, to build and to hear feedback about.
“It’s some of the simple stuff when you think about it, like, why has no one done this yet?” said Bain of the design details.
Also a factor in designing On-call was feedback from customers. For Product Manager Megan McDonald, the customer voice heavily influences product development.
On-call, along with every product at incident.io, McDonald explained, aims to follow the major pain points of customers and revolve around how the experience can be better, from getting paged to mitigating an issue to going into the post-incident world.
“It’s really letting the customers drive where they want the product to go,” said McDonald.
One thing McDonald knew was particularly important for customers? Reliability.
To ensure this, the engineer team dedicated time to running continuous tests in production, staging and development environments.
“We’re really proud of that…it’s something we’re going to continue investing in,” said Bain.
The decision to prioritize time spent on not only reliability but also observability and resilience was a no-brainer for McDonald.
“The reality is, if an on-call product is not reliable, if you don't think that this is going to wake you up when stuff is going wrong in the middle of the night, you're not going to buy it,” said McDonald.
This resonated with the engineering team, the majority of whom have either been on call or managed on-call programs previously.
On-call is just the latest addition to the tightly connected incident.io platform, working closely alongside Response, Status Pages, and Post-Incident Insights.
“We really want to make sure that once we've woken you up, you know what to do next, you know where to go. How do you take action, and then also make it really easy to mitigate those issues along the way,” said McDonald.
While there are things incident.io On-call already does to increase user experience – like the ability to bulk manage alerts – there are also additional things Bain and his team want to do in the future to make it even better.
What are McDonald and Bain most excited to ship in the near future?
“I’m really excited about what we can do for some of our on-call managers or engineering managers…we're quite excited to dig in and do some investment to make lives easier for people,” said McDonald.
For Bain, the answer is incorporating AI learnings into the product. One example of this was a recently shipped feature that automatically suggests follow ups after an incident was closed. “I think doing similar stuff with alerts would be super valuable,” said Bain.
Q & A
- New Relic isn’t on the list for alert sources. Is it planned?
- What On-call analytics are there (or are coming soon)?
- Can schedules be configured as mutually exclusive to automatically ensure someone isn’t on call for more than one of them at a time?
- How does the On-call feature work for the incident.io team if the incident.io platform is 100% down?
- What is the timeline for Terraform support to define escalation policies, alert routes, alerts, sources and all other bits related to on-call product?
Want to learn even more about how On-call can work for your team? Book a demo now.
