Building with AI

Over the past year, we’ve been hard at work building a new agent that will investigate incidents with you, show you what’s wrong and why it’s wrong. Eventually, it’ll show you how to fix incidents or offer to fix them on your behalf. It will feel like we’re right there with you, helping you resolve the incident just like your best and most experienced colleague would.

To get there, we’re on the sharp edge of what’s possible with AI. It’s pushed us to become ‘AI Engineers,’ build internal tools to tame the chaos of non-deterministic systems, and rethink our product process for AI systems which are much harder to evaluate. We’ve built resilience, learned to navigate the uncertainty and stay motivated through tough, ambitious R&D.

Last week, we launched a new page: incident.io/building-with-ai. The page is full of technical blog posts that the team has written with the lessons we’ve learned on that journey.

Here’re some examples of what you can expect to see:

• You can’t vibe code a prompt: Why blindly trusting AI to optimize your prompts can backfire, and human intuition is still essential when building intelligent agents.
• Controlling costs when building with AI: Building with AI is one of the easiest ways to create a huge infrastructure bill. Teams need visibility and awareness of what they're spending, along with guardrails to catch mistakes.
• Optimizing LLM prompts for low latency: When the time taken to execute a prompt becomes an issue, these strategies can optimize response latency without impacting prompt behaviour.
• Break chatbot speed limits with speculative tool calls: Learn how we engineered a 50% reduction in chatbot response times by speculatively executing tool calls, making us one of the fastest chatbots around.
• Why we built our own AI tooling suite: A technical deep-dive into why we built our own AI tooling suite, from evals to model leaderboards, and the key benefits we gained as a result.
• Tricks to fix stubborn prompts: If your prompt isn't doing what you expect and you've tried all the classic suggestions, here are tricks you can use to make even stubborn prompts listen.

We’re planning to continue building in public, and sharing the lessons we’re learning on this page along the way. We’d love to hear from you with any stories, or lessons you’ve learned, building AI products too.

What else we've shipped