Behind the scenes

Building an AI incident responder

A note from the team

We’re building an agent that will investigate the incident with you, show you what’s wrong and why it’s wrong. Eventually, we’ll show you how to fix it or offer to fix it on your behalf. It will feel like we’re right there with you, helping you resolve the incident just like your best & most experienced colleague would.

To get there, we’re on the sharp edge of what’s possible with AI. It’s pushed us to become ‘AI Engineers’, build internal tools to tame the chaos of non-deterministic systems, and rethink our product process for AI systems which are much harder to evaluate. We’ve built resilience, learned to navigate the uncertainty and stay motivated through tough, ambitious R&D.

These are the people and stories behind AI at incident.io.

  • Ed Stephinson
  • Milly Leadley
  • Lisa Karlin Curtis
  • Lawrence Jones
  • Rory Malcolm
  • Ed Dean
  • Elizabeth Wicks
  • Tom Petty

Avoiding the ironies of automation

Avoiding the ironies of automation

Incidents happen when the normal playbook fails—so why would we let AI run them solo? Inspired by Bainbridge’s Ironies of automation, this post unpacks how AI can go wrong in high-stakes situations, and shares the principles guiding our approach to building tools that make humans sharper, not sidelined.

Chris EvansChris Evans
March 26, 2025
You can’t vibe code a prompt

You can’t vibe code a prompt

Why blinding trusting AI to optimize your prompts can backfire, and human intuition is still essential when building intelligent agents.

Milly LeadleyMilly Leadley
March 25, 2025
Controlling costs when building with AI

Controlling costs when building with AI

Building with AI is one of the easiest ways to create a huge infrastructure bill. Teams need visibility and awareness of what they're spending, along with guardrails to catch mistakes. This is how we control spend at incident.io.

Ed StephinsonEd Stephinson
March 25, 2025
Optimizing LLM prompts for low latency

Optimizing LLM prompts for low latency

When the time taken to execute a prompt becomes an issue, these strategies can optimize response latency without impacting prompt behaviour.

Lawrence JonesLawrence Jones
March 25, 2025
Break chatbot speed limits with speculative tool calls

Break chatbot speed limits with speculative tool calls

Learn how we engineered a 50% reduction in chatbot response times by speculatively executing tool calls, making us one of the fastest chatbots around.

Lisa Karlin CurtisLisa Karlin Curtis
March 25, 2025
Why we built our own AI tooling suite

Why we built our own AI tooling suite

A technical deep-dive into why we built our own AI tooling suite, from evals to model leaderboards, and the key benefits we gained as a result.

Lisa Karlin CurtisLisa Karlin Curtis
March 25, 2025
Tricks to fix stubborn prompts

Tricks to fix stubborn prompts

If your prompt isn't doing what you expect and you've tried all the classic suggestions, here are tricks you can use to make even stubborn prompts listen.

Milly LeadleyMilly Leadley
March 25, 2025
Come build the future of Incident Response

Come build the future of Incident Response

Ever curious about why incident.io is the most exciting place to build your career in 2025? Let us share our incredible product momentum, ambitious roadmap, and unique engineering culture that's attracting the talent eager to redefine incident response.

Norberto LopesNorberto Lopes
March 20, 2024