Behind the scenes
Building an AI incident responder
A note from the team
We’re building an agent that will investigate the incident with you, show you what’s wrong and why it’s wrong. Eventually, we’ll show you how to fix it or offer to fix it on your behalf. It will feel like we’re right there with you, helping you resolve the incident just like your best & most experienced colleague would.
To get there, we’re on the sharp edge of what’s possible with AI. It’s pushed us to become ‘AI Engineers’, build internal tools to tame the chaos of non-deterministic systems, and rethink our product process for AI systems which are much harder to evaluate. We’ve built resilience, learned to navigate the uncertainty and stay motivated through tough, ambitious R&D.
These are the people and stories behind AI at incident.io.
Weaving AI into the fabric of incident.io
How incident.io became an AI-native company — building reliable AI for reliability itself, and transforming how teams manage and resolve incidents.
The timeline to fully automated incident response
AI is rapidly transforming incident response, automating manual tasks, and helping engineers tackle incidents faster and more effectively. We're building the future of incident management, starting with tools like Scribe for real-time summaries and Investigations to pinpoint root causes instantly. Here's a deep dive into our vision.
Avoiding the ironies of automation
Incidents happen when the normal playbook fails—so why would we let AI run them solo? Inspired by Bainbridge’s Ironies of automation, this post unpacks how AI can go wrong in high-stakes situations, and shares the principles guiding our approach to building tools that make humans sharper, not sidelined.
You can’t vibe code a prompt
Why blindly trusting AI to optimize your prompts can backfire, and human intuition is still essential when building intelligent agents.
Controlling costs when building with AI
Building with AI is one of the easiest ways to create a huge infrastructure bill. Teams need visibility and awareness of what they're spending, along with guardrails to catch mistakes. This is how we control spend at incident.io.
Optimizing LLM prompts for low latency
When the time taken to execute a prompt becomes an issue, these strategies can optimize response latency without impacting prompt behaviour.
Break chatbot speed limits with speculative tool calls
Learn how we engineered a 50% reduction in chatbot response times by speculatively executing tool calls, making us one of the fastest chatbots around.
Why we built our own AI tooling suite
A technical deep-dive into why we built our own AI tooling suite, from evals to model leaderboards, and the key benefits we gained as a result.
Tricks to fix stubborn prompts
If your prompt isn't doing what you expect and you've tried all the classic suggestions, here are tricks you can use to make even stubborn prompts listen.
Come build the future of Incident Response
Ever curious about why incident.io is the most exciting place to build your career in 2025? Let us share our incredible product momentum, ambitious roadmap, and unique engineering culture that's attracting the talent eager to redefine incident response.
Join our team – we’re hiring!
Join the ambitious team helping companies move fast when they break things.
See open roles