March 5th was a big day for incident.io as we released our on-call product to the world. Nine months of listening to our customers, coding, fixing, testing, and polishing came together for our biggest product launch to date.
Releasing On-call was a huge milestone and represented the next step in our journey as a company. The winds of change have touched everything, right down to our core principles, and I want to talk about how we adapted from our usual “rapid-fire release” culture to building something so big and complex in total silence!
The challenge
Launching On-call changed the way we work in many ways. For example, delivering on our alert availability guarantees forced us to find new ways to maintain pace along with even higher standards of reliability.
Managing the flood of demand to use On-call while still providing great customer support for all of our users made us level up our support and product response processes.
Adapting our culture
Making things magic for our customers is central to how we work. Last year, for example, we released a changelog every single week because, big or small, we don’t want anyone to miss out on a fix or feature that could make their lives better.
On-call was different. While we’d discussed it with customers in the past, and many people knew it was coming some day, we wanted to keep things under-wraps during the development phase so we could make the biggest possible splash at launch. We also have competitors who like to imitate us (we see it as a form of flattery), so why make it easy for them?! 😉
But building in private comes with trade-offs. Without our usual customer feedback loops, we had to look inward for answers.
We began dogfooding the product as early as possible, running it alongside our existing paging solution to get it in front of fellow incident.io employees across the business.
We put the product in front of the whole company as fast as we could, and did our best to remove the friction from the adoption and feedback processes. We introduced a dedicated segment in our weekly whole-company meeting to share product updates, make clear any requests for feedback, and to highlight what was coming up next.
In essence, the rest of incident.io became our proxy-customers.
We're blessed with a team that cares a lot about our product and consequently has very high standards. Keeping the whole company in the loop with gentle but constant reminders helped maximise our feedback surface area, kept ourselves accountable in front of some tough critics (our colleagues and founders), and gave us a way to celebrate the little wins as we released more features.
Design partners and the beta launch
We’re incredibly lucky to have a customer base of passionate individuals who care a lot about what we build, how we build it, and how it feels to use.
As a result, we had no trouble recruiting a handful of design partners, and although it was a small group, all of them were heavily invested in the project. Regular fortnightly catchups left us with several hours every week to share new features and get feedback on the product.
Our product engineers were heavily involved; as one of our product designers put it, "we want engineering to be more than just an execution factory.” The way we do that is by connecting the people making the low-level product decisions directly to our customers with feedback.
It’s a privilege to have users willing to spend significant amounts of time and effort to help us build the best product we can. From our customer Slack channels which are open to everyone in the business, to celebrating incident milestones, we invest heavily in making our customer experience as good as it can be.
It takes constant focus and can’t be done overnight, but the effort we invested into going above and beyond with our customers has given us a customer base that cares about what we're doing and how we do it. Ultimately that made it much easier to get the feedback we needed to build On-call.
De-risking by launching in stages
Along with scheduling, escalations, and paging, alerts is one of the core components of our On-call product. We launched our Alerts product two months before On-call, and it was a great dovetail between our existing incident response product and On-call.
With Alerts released to existing customers, they could connect events from alerting and observability platforms like Prometheus and Grafana and have them create incidents when required. While there was no native paging component yet, customers could integrate their existing paging solution to achieve the same result.
Ultimately, the release of Alerts allowed us to deliver some early value into customers’ hands, as well as de-risk the later On-call launch by getting a significant chunk of the technical work behind the product out in the wild.
Of course, this came with its own trade-offs. With Alerts now being actively used by customers, we had to shift part of our attention to product to support whilst still building the core product offering. We were no longer a team with purely a focus on building the next thing, we also had to help out existing customers. In some ways, this was useful as a warm up to launching and supporting the on-call product itself.
Shifting focus after the launch
With the extensive load-, user- and soak testing we’d performed in the lead-up to launch, along with the real-world testing of our design partners, we had high confidence in releasing the product to our entire customer base on launch day. It turns out that our confidence was well placed: there have been zero notable issues to date (knock on wood!).
What did change after launch was the shift in focus of the team. With so many customers now using the On-call product, we had to factor in requests for new alert sources, shipping new features (like Slack user group syncing), and responding to feedback about migration and onboarding.
While it might sound like a lot to contend with, this is our happy place. Working in public and (almost!) drowning in feedback is our sweet spot. Give us customers with opinions and feedback any day!
Reflections
The journey to launching our On-call product felt like a true test of our team's adaptability, resilience, and commitment to excellence. From the initial stages of internal testing to the meticulous planning of the launch, every step was important to get us where we are today.
We're super excited to continue refining and expanding our offerings, and with On-call now out in the wild, it’s great to be in a position to put customers back at the heart of everything we do.
Thank you for being a part of this journey with us. Seeing On-call in the hands of so many organizations already has made all of the hard work, late nights, and development iterations worthwhile.
