Being on-call at incident.io

Supporting our product and each other

At incident.io, we are building a product that our users rely on 24/7, all year round. This means it is crucial that it is always working, and that is where our on-call rotation comes in. As we explore in on-call at incident.io, we believe that everyone should be on-call because it tightens the feedback loop between shipping new features and maintaining what we have, leading to more pragmatic engineering decisions.

We use our own on-call software to run our on-call and incident response (with a fallback system in case things are really broken!), which makes being on-call a great opportunity to improve both our own processes and our product too.

Being on-call has an impact on home life, and we are committed to making it the best experience possible. Our approach is built on support, a thoughtful approach to compensation, and clear expectations.

How our on-call schedules work

Our on-call schedules run with one dedicated person holding the pager each week, from 5 PM on Friday to 5 PM the following Friday. We’ve found that putting the weekend at the start of a shift means it isn’t looming over you all week, and it makes for a pleasant Friday evening when a shift concludes.

There are a few things we do as a team to support whoever is on-call each week:

• You can always ask for cover (our product is really good for this!) if something comes up.
• If you have a disrupted night you should take the time back to rest up, and we’ll automatically suggest you request cover
• There’s always someone else to escalate to. We have a secondary rota you can escalate to if you’re stuck debugging something you’ve never seen before, and a founders rota if you need support with a tricky situation.
• If something is broken your phone will ring: you shouldn’t be watching Slack all the time!

New engineers are expected to join the rota after their probation period, and we actively work to accelerate this process. You’ll start off holding the pager during working hours with a buddy sat next to you to get plenty of practice runs before you’re taking the overnight shift.

How compensation works

Being on-call is a key part of being an incident.io engineer, and our overall on-call compensation already takes this into consideration. The on-call pay itself is a small additional amount to express our gratitude. This also helps make covering for others feel fairer and avoids any complex time-swapping: if I cover you for an hour, I’m getting paid for it so you don’t owe me an hour back!

Specifically, we pay £5/hour for on-call outside of normal working hours (9:30 AM - 5:30 PM UK). For a full week, this totals £640. Public and company holidays (like First Friday of the Month!) are paid double.

If someone has had a challenging night on the pager, we are happy for them to take the morning off. We don’t have a formula for this, you’re trusted to make a judgement and your team will encourage you to take the time off to get some rest!

What we think good looks like

Following on-call best practices, when our engineers are on-call, we expect them to be near their laptop, no more than 5 minutes away from being online and ready to respond. What does “ready to respond” mean, other than an internet connection? Again, we don’t have a long list of rules here: we’re all responsible adults who can figure this out. Exploring a gallery on the weekend, between stations on the way to work, at the gym, having a pint after work - all fine. So long as you’re able to make sensible decisions and communicate them quickly, you’re good.

I really can’t emphasize how much cover requests make on-call feel much better. It is very common to ask for cover for personal commitments like a brunch or a run, or even to swap entire weekends or shifts. We are all in this together, supporting each other to ensure our product runs smoothly while maintaining a healthy work-life balance.