Behind the scenes

Building an AI incident responder

A note from the team

We’re building an agent that will investigate the incident with you, show you what’s wrong and why it’s wrong. Eventually, we’ll show you how to fix it or offer to fix it on your behalf. It will feel like we’re right there with you, helping you resolve the incident just like your best & most experienced colleague would.

To get there, we’re on the sharp edge of what’s possible with AI. It’s pushed us to become ‘AI Engineers’, build internal tools to tame the chaos of non-deterministic systems, and rethink our product process for AI systems which are much harder to evaluate. We’ve built resilience, learned to navigate the uncertainty and stay motivated through tough, ambitious R&D.

These are the people and stories behind AI at incident.io.


The timeline to fully automated incident response

The timeline to fully automated incident response

AI is rapidly transforming incident response, automating manual tasks, and helping engineers tackle incidents faster and more effectively. We're building the future of incident management, starting with tools like Scribe for real-time summaries and Investigations to pinpoint root causes instantly. Here's a deep dive into our vision.

Ed DeanEd Dean
April 9, 2025
Avoiding the ironies of automation

Avoiding the ironies of automation

Incidents happen when the normal playbook fails—so why would we let AI run them solo? Inspired by Bainbridge’s Ironies of automation, this post unpacks how AI can go wrong in high-stakes situations, and shares the principles guiding our approach to building tools that make humans sharper, not sidelined.

Chris EvansChris Evans
March 26, 2025
You can’t vibe code a prompt

You can’t vibe code a prompt

Why blindly trusting AI to optimize your prompts can backfire, and human intuition is still essential when building intelligent agents.

Milly LeadleyMilly Leadley
March 25, 2025
Controlling costs when building with AI

Controlling costs when building with AI

Building with AI is one of the easiest ways to create a huge infrastructure bill. Teams need visibility and awareness of what they're spending, along with guardrails to catch mistakes. This is how we control spend at incident.io.

Ed StephinsonEd Stephinson
March 25, 2025
Optimizing LLM prompts for low latency

Optimizing LLM prompts for low latency

When the time taken to execute a prompt becomes an issue, these strategies can optimize response latency without impacting prompt behaviour.

Lawrence JonesLawrence Jones
March 25, 2025
Break chatbot speed limits with speculative tool calls

Break chatbot speed limits with speculative tool calls

Learn how we engineered a 50% reduction in chatbot response times by speculatively executing tool calls, making us one of the fastest chatbots around.

Lisa Karlin CurtisLisa Karlin Curtis
March 25, 2025
Why we built our own AI tooling suite

Why we built our own AI tooling suite

A technical deep-dive into why we built our own AI tooling suite, from evals to model leaderboards, and the key benefits we gained as a result.

Lisa Karlin CurtisLisa Karlin Curtis
March 25, 2025
Tricks to fix stubborn prompts

Tricks to fix stubborn prompts

If your prompt isn't doing what you expect and you've tried all the classic suggestions, here are tricks you can use to make even stubborn prompts listen.

Milly LeadleyMilly Leadley
March 25, 2025
Come build the future of Incident Response

Come build the future of Incident Response

Ever curious about why incident.io is the most exciting place to build your career in 2025? Let us share our incredible product momentum, ambitious roadmap, and unique engineering culture that's attracting the talent eager to redefine incident response.

Norberto LopesNorberto Lopes
March 20, 2024

Team in SF

Join our team – we’re hiring!

Join the ambitious team helping companies move fast when they break things.

See open roles