Is AI just another third-party dependency? Yes, right up until it isn't

November 16, 2026 — 8 min read

Ask an engineering team how they manage their cloud provider, their database or their payments API, and you'll get a familiar answer: SLAs, SLOs, dashboards, alerts, a failover plan, a runbook. Ask the same team about the AI model behind their product, and the answer tends to get a lot fuzzier. For plenty of teams, the first sign something's wrong is someone asking "is the AI broken for anyone else?" in Slack, or a quick peek at the model provider's status page.

That made me wonder: is AI just another third-party dependency?

I put this in front of two people who live with this every day. Michael Tweed is a Principal Software Engineer at Skyscanner, where AI tooling is part of how engineers write, review, ship, and run software. Sahaj Sawhney is a Product Engineer on the AI products here at incident.io. They work on opposite sides of the problem, and their answers were remarkably similar: mostly yes.

Both were quick to add that "mostly" is doing a lot of work in that sentence.

The case for "yes"

Let's start with where the analogy holds, because it holds further than you might expect.

You can, and should, treat it like any other dependency, even though it's hard. That means sensible observability baked in and reasonable alerting thresholds. The goal is to solve the hard parts so that you can treat model APIs just like any other APIs.

The same discipline applies to vendors, SLAs and SLOs, and Skyscanner's setup reflects that. A platform team owns the availability side, including the endpoints serving models and an LLM gateway that sits in front of providers. They're responsible for the usual questions: is it up, what's the latency, what happens when something falls over, and how do we fail over? The aim is to keep it as normal as possible, like any other dependency.

There's also a good reliability argument for this framing. Our VP of Engineering, Norberto, wrote about how dependency ceilings work: if your service depends on a database, a queue, and compute, your maximum theoretical availability is the product of all three. Three components at 99.99% each, and you're capped at roughly 99.97%. A model provider is one more link in that chain, and it deserves the same scrutiny as the rest. Norberto's rule of thumb is that any dependency without a demonstrable history of hitting your target needs an alternative.

So far, so familiar. Here's where it gets interesting.

"Normal" is harder to define

The first crack in the analogy shows up when you try to say what "healthy" looks like.

With a normal API, you alert on error rates and latency, and that covers most of what matters. With a model, it's quite hard to pin down what normal or good looks like from the vendor's side. You don't just want to know that a call returned a 200. You want to know whether the answer was any good. A model can be fully available and noticeably worse than it was yesterday, and nothing on a standard uptime dashboard will tell you.

That's why teams are building quality into their monitoring. At incident.io, we trace usage across the platform the way we'd trace any other request, and we use LLMs as judges to grade interactions and responses. Those grades become metrics we can alert on. Simple signals help too, like a thumbs up or thumbs down on a response. Skyscanner runs evals for the same reason.

Then there's the question of the vendor changing things under you. In theory, a provider shouldn't change a model, and its behavior should be repeatable. The obvious follow-up is how much you can trust that. It comes down to how much you invest in your own monitoring versus how much you rely on the vendor. And when a model upgrade goes wrong, the blast radius is bigger than a slow response. If AI has been writing code, writing documents, and pushing changes during that window, you may need to ask whether what it produced needs another look.

None of this is unfamiliar territory for anyone who's managed a vendor relationship. It's just that you have to define "working" far more carefully than you would for a database.

Where the analogy really breaks: debugging

This is where the line is clearest: you treat the APIs the same way, but you respond to issues differently.

Debugging an AI system takes a degree of experience that most teams haven't built yet. Some of the issues are strange, and a lot of it comes down to upskilling a team and building a robust debugging stack: tracing, tooling, and the kind of observability that lets you actually work out what went wrong when it does.

Take the first hour of an AI tooling issue at Skyscanner. Engineers flag it quickly, and it's usually a manual report that comes first, like "my prompt's taking too long". These days there's observability to help separate a provider issue from one person's machine, but the hard part is the same as it ever was: working out at what point in the stack things are failing. With more tools, more providers and more places for things to go wrong, that gets harder, not easier.

There's a subtler difference too: the context that agents write for other agents. If a model generates documentation or memory, and a different model reads it later, the same inputs might produce different outputs. Is that okay? Is something lost? With a normal dependency, that question never comes up, because the same input gives you the same output.

My guess is that some of this fades over time. Every new category of infrastructure starts out feeling strange before the practices catch up.

The contrarian bit: maybe you don't need a model there

This is my favorite part.

Models are non-deterministic by design. That's what makes them useful, and it's also what makes them awkward as dependencies. Most of the dependencies we rely on are valuable because they behave the same way every time. If you're trying to push a model into being 100% deterministic all of the time, then you probably don't need a model there.

One engineer we spoke to is guilty of this. They were building something that used a model to run some analysis, and by the end they'd written so much Python around it to force the output into a particular shape that the model was contributing a tiny part of the result. Their question to themselves: "Did I really need the model at all?"

I love this because it flips the usual resilience conversation. Instead of asking how to make the model more reliable, you ask how much of the job really needs a model. The parts that need to behave identically every time can often be plain, deterministic code, which you already know how to monitor and fail over. The same idea shows up on the product side: when part of an agentic system is down, the best case is that it still gets to the answer, perhaps more slowly, by falling back on more traditional deterministic workflows.

Treat the model as a dependency for the parts that genuinely need judgment, and keep everything else boring.

So, is it just another dependency?

Mostly yes, with an asterisk. Treat it like a dependency for SLAs, SLOs and observability, because that's where decades of reliability practice already apply. Then accept that quality is harder to measure, that vendors can change things under you, that debugging is its own skill, and that the best way to lower your exposure is sometimes to ask whether you needed a model in the first place.

And my bet is that the asterisk gets smaller over time.

If you want to see how we're applying these ideas in our own products, take a look at Investigations or book a demo.

Picture of Komal Ahuja
Komal Ahuja
Product
View more

See related articles

View all

So good, you’ll break things on purpose

Ready for modern incident management? Book a call with one of our experts today.

Signup image

We’d love to talk to you about

  • All-in-one incident management
  • Our unmatched speed of deployment
  • Why we’re loved by users and easily adopted
  • How we work for the whole organization