How Zendesk ditched 15 years of patchwork tooling, in 10 weeks

Tens of thousands of businesses depend on Zendesk to run critical customer-facing workflows. When Zendesk goes down, those companies can't serve their own customers, so for Zendesk's engineering team reliability is a business problem as much as an engineering one.

But reliability is only as good as the tooling behind it. Zendesk's had quietly become hard to maintain.

Fifteen years of patchwork tooling

As Zendesk scaled, its incident tooling grew piece by piece, never designed as a single system. Zendesk had run incident response on a homegrown tool for around 15 years, with PagerDuty handling on-call and alerting for 14 of them. Both had been solid in their time, but neither had kept pace with how Zendesk actually worked.

The homegrown system had sprawled over years of new products, new teams, acquisitions, and reorganizations, and the names of teams and services had drifted from what people actually called them. The team that owned a feature might still be listed under something it was called five years earlier, so resolving an incident relied on institutional knowledge, someone who remembered the history, rather than a standardized process anyone could follow.

"Our old system grew organically over 15 years of new products, features, teams, and acquisitions. It was a mess. During incidents we'd say 'we need the team that owns this feature,' and the name had drifted from what it's actually called now to what was configured five years ago," recalls Anna Roussanova, Engineering Manager at Zendesk.

PagerDuty had ballooned alongside it. At the time, the team counted 12 alert sources feeding into the setup, with no shared standards, and getting the right people into the room often took longer than anyone wanted. Response times had more to do with coordination than with the fix itself.

Why replace something that works?

A migration like this is scary. A team is replacing something that technically still works, on a system the whole business depends on. Businesses only take on that risk for something materially better. It's not an undertaking leaders like Tom Monaghan, VP, Engineering Productivity & Product Reliability take lightly. "Migrations of critical systems are generally very high risk. The probability of losing visibility into telemetry or alerts from your platform could result in a protracted customer outage or catastrophic failure. It's something we're very careful about."

Two things came together for Zendesk; the response times their drifting setup produced had become a reliability problem they couldn't ignore, and an upcoming renewal on their legacy on-call tool created an opportunity for action.

Once they decided to look, they evaluated the market against their incumbent, and it didn't take long for incident.io to stand out.

“I remember sitting in one of those initial meetings with incident.io, doing Slack DMs with my manager,” says Anna. 'I’m like, we're going to have to do this, aren't we? This is just too good.

What sold them was the platform itself: on-call and response in one place instead of jumping between disconnected tools. With alerts, escalation, and coordination unified, the right people reach an incident faster, and far less time is lost to the handoffs that had been dragging out triage and resolution. It also cost less than the separate tools it replaced.

For Tom, it also came down to focus. Zendesk could have kept building its own response tooling, but that wasn't the best place to spend the company's engineering effort. “We want to focus on what we're good at and leave vendors like incident.io to deliver the things they're very good at.”

The decision wasn't only about cleaning up what they already had. It was also about where Zendesk could go from there. Consolidating onto one platform was the immediate win, but Zendesk was also betting on a platform that would keep adding capabilities the old stack could never have supported.

Migrating 1,200 engineers in 10 weeks

The numbers were a bit daunting, even for an experienced team. 150 scrum teams, 1,200 engineers, and more than 5,000 monitors, in roughly 10 weeks. Anna led it with one other full-time engineer, plus a rotating cast of specialists who came in for specific alert sources. Three systems ran in parallel during the transition, so the migration had to be careful as well as fast.

What made the timeline workable for Anna and the team was planning before building.

“We had a great group of people who knew all the different pieces and could speak to them with authority. We put together an actual plan before the 10 weeks started, so once we got into the system we could just knock things out.”

It helped that most of Zendesk's alerting was already defined in code, so large chunks could be moved methodically rather than by hand. The team also treated the migration as a chance to clean up rather than copy the mess across. The roughly 5,000 Datadog monitors that had fanned out across about 500 PagerDuty services were consolidated to a handful of alert sources, and 253 escalation policies were cut to 153. incident.io's Catalog feature gave them a live, always-synced map of which team owned what, so ownership stopped drifting.

The way escalations work, if you pick a service, the system knows which team owns it, so it pages that team. “You don't have to think about it,” says Anna. And when new alert sources turned up mid-migration, adding them wasn't the bottleneck the team had feared.”It's honestly like five minutes to integrate, as long as you have the right permissions. Then it's just validating that the data comes across the way you expect.”

Rather than one big cutover, the team moved in batches. Early-adopter teams went live within about five weeks, starting with one of the most critical teams and its highest-priority alerts. That validated the setup before the riskier cutovers. For a while the team also ran the old and new systems side by side, watching the high-priority alerts land in both to confirm nothing was being missed before anything was switched off. By the official go-live date, much of engineering had already been running on incident.io for weeks.

Faster triage, and a lot less busywork

What mattered to Zendesk's leadership was the outcome, not the effort. The real test was go-live. “Day one of go-live, I felt a sense of paranoia,” recalls Tom. “With migrations this big you expect things to break, to be woken up in the middle of the night. We didn't have any of that. It turned out nothing was broken.”

The feedback that landed hardest came from inside engineering, where smooth migrations rarely get remarked on at all.

With alert routing consistent and the right people reaching incidents automatically, mean time to triage fell 32%, and mean time to resolve is trending down with it. In its first year on incident.io, Zendesk eliminated more than 800 hours of annual toil and saw over $500k in actualized savings.

Anna puts it simply, “With incident.io, the process of getting people into incidents is a lot less stressful.”

The biggest surprise wasn't something they set out to fix at all. Documentation used to eat engineers' time after every incident. Now AI drafts it automatically. We've clawed back hundreds and hundreds of hours,” says Tom, “and the fidelity is much higher, which means better learning.”

What isn't in question is how essential it's become.

From vendor to partner

Ask Anna and Tom what made the difference, in the migration and since, and both come back to the partnership. "incident.io was definitely a partner in this. There are so many things we've done in the year since going live that I don't think we'd have been able to do without that partnership," says Anna.

It started in the planning. Eryn, their incident.io CSM, came on site to help the team map out the migration, building the timeline and drafting the full sequence of go-live comms in a single working session, so nothing had to be improvised once cutovers began. “We showed her everything we were doing, so she understood our current workflow and our system, and could translate that into incident.io terms, recommending what would actually make sense for us. She was knowledgeable and reassuring, and she was right,” says Anna, “we did it.”

It has carried into the day-to-day since. A shared Slack channel means answers in 15 to 30 minutes rather than 24-to-48-hour ticket waits, and feature requests come back fast, sometimes as working prototypes. “Sometimes we're in an incident with actual customer impact and we need an answer. Being able to get that immediately is incredibly powerful.”

For Tom, that ongoing support is the real differentiator.

That combination, the right platform and a real partner, is what turned a migration most teams would dread into a foundation they can keep building on.