Run incidents without leaving Slack
Tailored to your organization for smarter incident management
Confident, autonomous teams guided by automations
Learn from insights to improve your resilience
Build customer trust even during downtime
Connect everything in your organization
Streamlined incident management for complex companies
Run incidents without leaving Slack
Tailored to your organization for smarter incident management
Confident, autonomous teams guided by automations
Learn from insights to improve your resilience
Build customer trust even during downtime
Connect everything in your organization
Streamlined incident management for complex companies
A few years ago I’d just moved to London and started out at my first software job. I was having a great time building things and making new friends, and one evening a friend and I decided there was a new problem we wanted to solve: we really didn’t like the expenses software. We thought it was confusing and over-complex, and decided we could do better.
Fast-forward a few years and I pop a message in the incident.io Slack asking for examples of bad build-vs-buy decisions and 6 minutes later…
Why is that late-night decision still remembered years later? Let’s breakdown what I got so wrong.
Expenses are pretty simple right? Employees spend some money, send in a receipt, tag it with a category, and then get reimbursed every week or two. Surely that’s not too hard?
At first we felt pretty vindicated: I think it only took a few evenings to build something that worked reasonably well! But then the feature requests started to roll in. We added an email address you could forward receipts to. The finance team wanted to tweak the reporting so they could spend less time sending out the reimbursements.
Then they wanted to add an approvals step, where expenses needed sign-off from a manager. That meant we needed to track who everyone’s manager is, which would change quite frequently since we were growing fast.
At first I didn’t mind spending a few hours each week building out this little app. I was having fun hacking on something with my friends, and it always feels great to tell someone the feature they wanted is live now.
But the enjoyment wears off: the more I settled in to life in London the more I had other things I wanted to spend my evenings doing, and the more our little expense app got used the more people started finding annoying little bugs which weren’t nearly as fun to fix as building new things.
So what about doing some of this work at…work? That might feel like the obvious choice, but it comes at a cost: what do I not do, and how do I justify that? It turns out that “sorry, I was fixing the expenses system” isn’t the best reason for projects slipping behind schedule.
Building an internal system is very rarely going to be anywhere near as valuable as delivering the product your company is actually selling.
After a while the friend I’d built this with left the company, and then I did too. This left a slow-burning question behind: who can fix all the little bugs in that weird expenses system?
When you designed and built a system, fixing bugs is much easier than if you’ve never seen it before. You can share that knowledge through pairing, reviewing code, or talking someone through it, but only if you’re still around to do that. After I’d left all that knowledge had gone with me, which made the cost of maintenance even higher.
When I asked around, this wasn’t the only example of building something that really should have just been bought. Most of the other examples were in choosing to run an open-source database or monitoring tool internally, rather than buying the SaaS option. This has the same hidden and unpredictable maintenance burden of something you build in-house, only you never get to leverage knowing the system deeply because it’s unlikely that you can hire the person who wrote it.
There’s a few key reasons to build something in-house:
As engineers, we have a tendency to under-estimate how hard things are to build - just ask any project manager watching deadlines fly past, or read the comments under a product launch post. We also often under-estimate our own value: the average engineer in the SF Bay Area is paid $120 per hour according to levels.fyi (in London it’s $70). Even a few hours a month at that rate buys quite a lot of SaaS!
It’s much easier to step back and think about whether something is really worth building before spending lots of time on it. It’ll certainly feel better than having an awful expense system haunting you for years.
Enter your details to receive our monthly newsletter, filled with incident related insights to help you in your day-to-day!
Better learning from incidents: A guide to incident post-mortem documents
Post-mortem documents are a great way to facilitate learning after incidents are resolved.
Luis Gonzalez
How we’ve made Status Pages better over the last three months
A few months ago we announced Status Pages -- the most delightful way to keep customers up-to-date about ongoing incidents. Since then, we've launched several features to add an extra bit of delight. Read on to learn more.
Asiya Gorelik
The balancing act of reliability and availability
To prevent issues like downtime, you have to focus on the reliability and availability of your product. But there's a balance to be struck here.
incident.io