# Don't add a read replica until you've read this

*July 21, 2026*

As the size and complexity of their relational database workload grows, every company eventually goes through the process of off-loading work on a read replica. It comes with lots of benefits, but at a cost of increased complexity. This article is about how we dealt with that, a lot of learnings, and some useful techniques.

incident.io is an incident management product relied on by thousands of customers to be the thing that supports them through anything from a minor blip to a full outage. Any disruption to that service has a significant impact on those users, and that’s always top of mind for our engineering team. Everything we build is designed to be performant, reliable, and gracefully degrading.

When large parts of the internet goes down, as they did for the [AWS outage on 20 October 2025](https://health.aws.amazon.com/health/status?eventID=arn:aws:health:us-east-1::event/MULTIPLE_SERVICES/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE_BA540_514A652BE1A), we see a meaningful increase in alert volume as engineers across the globe are getting woken up. During events like this, we simply can’t fall over from the increased load. This means we always have to run with significant spare capacity. And so we’re always looking for opportunities to reduce the load on our DB, either through performance improvements or code redesign. While working on projects to control the resource utilization on our primary database, we clearly identified that we, like a lot of online services, run an overall read heavy workload.

We already had a read replica set up and some queries were already utilizing it, but it was something that we were doing on a case by case basis. We knew we could do more. We set ourselves the goal to move _everything_ over to the read replica. For every query that could be move moved to the read replica, that’s additional capacity for our primary, and additional protection for those events of massive traffic that we design for.

## The benefits of read replicas

But before that, let’s do a quick recap of why you’d want to introduce read replicas into your stack. They do bring complexity, having two databases and two database connection pools in your app logic is more to think about than just having the one. You also need to manage two things, where both often quickly become critical to running your service. Double the graphs and metrics and warnings to worry about. Not to mention, you’re now paying for two databases.

They bring a lot of benefits though. They’re the natural step to take to give people access to run operational queries on the production system, without risking those queries disrupting the production workload. A query run on the primary can cause overload, delays, contention, locks, and much more. On the read replica the blast radius is much smaller, although there are notable exceptions to this like `hot_standby_feedback`. But maybe the most interesting thing is that it opens the door to horizontal scaling. Relational databases generally don’t scale horizontally, just vertically. Writing to two primary databases is slower than to one, and for most of us spending more to get less performance is not a very interesting prospect. But read replicas are different, since they don’t need to coordinate writes, you can just have more of them and load balance read queries across them. That’s pretty cool.

## Read-after-write consistency

Even before approaching this larger re-think we had already established some useful primitives. Our backend language is Go, but the basic techniques translate to some degree to any language.

The number one thing you need to deal with as you’re migrating work over to a read replica is [read-after-write consistency](https://jepsen.io/consistency/models/read-your-writes). Or in other words, avoiding stale reads. The gist of it is, if you write something to the primary and then immediately after read the thing back but from the replica, there’s no guarantee that you get the same thing back. This gets worse the faster you read after writing. Now if you’re hand-crafting some beautiful artisanal code in your walled garden project, you can probably attempt explicitly picking the primary or read replica for each individual query through a code flow. But for practical reasons we often end up just limiting the read replica to the queries and code paths where we feel that nothing can go wrong.

That’s not what we’re looking for, we want to move everything over. That means we need automated detection of mutating queries, and to automatically fall over to the primary after a write has been detected to ensure that you are able to read your own writes. The basic mechanism we introduced for this uses the [Go context](https://pkg.go.dev/context) to carry a special flag that controls whether we can use the read replica. It defaults to true.

```jsx
type taintedKey struct{}

// A write taints the context: reads on a tainted context must
// go to the primary, since the replica may not have caught up.
func Taint(ctx context.Context) context.Context {
      return context.WithValue(ctx, taintedKey{}, true)
}

func CanUseReplica(ctx context.Context) bool {
      tainted, _ := ctx.Value(taintedKey{}).(bool)
      return !tainted
}

```

The second concept we introduced was a simple function for detecting whether a given query was “safe” to go to the read replica. Initially we designed this to be conservative, we were ok with some things being misinterpreted and going to the primary, as long as we don’t get accidental and weird bugs where we send the wrong query to the wrong place. Some heuristics got us a long way, like whenever we start a transaction, we send it to the primary and mark the `ctx` so that any subsequent queries also go to the primary, with the basic assumption that anything in a transaction is probably something you want on the primary (this turns out to not be quite true, more on this later).

```jsx
// Transactions probably mean writes: run on primary,
// and taint the ctx so everything after follows it there.
func (db *DB) Transaction(ctx context.Context, fn func(context.Context) error) error {
      ctx = Taint(ctx)
      return db.primary.Transaction(ctx, fn)
}

```

The second heuristic was to strip comments and trim whitespace, then check if the query starts with `SELECT`. Not very elegant, but it got the job done. There’s a fancier version further down!

```jsx
func IsReadOnly(query string) bool {
      q := strings.TrimSpace(stripComments(query))
      return strings.HasPrefix(strings.ToUpper(q), "SELECT")
}

```

On top of this we built a basic layer on top of our database handles that created both primary and replica transaction pools and transparently switched between them, using the flag we had previously set. This means it was now generally safe for us to just opt in any code path to use the read replica, trusting this mechanism to direct the queries to the right place and maintain read-after-write consistency.

```jsx
func (db *DB) route(ctx context.Context, query string) *sql.DB {
      if !IsReadOnly(query) {
              return db.primary // and taint the ctx here
      }
      if !CanUseReplica(ctx) {
              return db.primary // we wrote earlier, stay consistent
      }
      return db.replica
}

```

Additionally we put a [circuit breaker](https://martinfowler.com/bliki/CircuitBreaker.html) in here. If we’re not able to open connections to the read replica we give up and send all queries to the primary. Note that although we’re happy to have that fail over behavior for now, with enough total load across your databases, your primary might not be able to handle this.

## Event handlers

incident.io is designed from the ground up on event publication and subscription, the entire system is built around publishing messages and having a fleet of workers processing them. This gives us all kinds of interesting super powers, including the ability to buffer up messages to process them later during periods of overload.

The workers is exactly where we had been adding some read replica offloading, more specifically in a set of workers that execute low priority workloads that do read heavy work. This is also where we started our work, primarily by moving more and more slices of our workers over to the read replica. Our initial results were really positive, we were almost being too successful and we quickly had to upgrade our read replica because it was taking over so much work from the primary. After having moved some of our heaviest read query workloads, congratulating ourselves for our great work, we noticed something odd. Not a super clear pattern, but the occasional `NotFoundError` coming out of our database adapter in the subscribers on the workers that we had just moved.

And that’s when it struck us. We were ensuring read-after-write consistency within the context of a go `ctx`, but what about across the boundary of publishing and processing a message?

Let’s say we have a request come in, to create a post-mortem document. But the post-mortem document itself can take a while to create and involves hitting a separate internal service over the network, we don’t do all of that work in line with the client waiting. So we write the row to the database, respond immediately to the client, and then publish a message to a worker to deal with actually creating the document. Incredibly, what we were seeing was the time between publishing a message to the message queue and the worker pulling that message to process it be shorter than the replication lag to the replica. This wasn’t because the replica was falling behind, it was doing just fine, it was just that _occasionally_ the event was processed incredibly quickly. Overall it was a rare occurrence, something between 0.1 and 0.5% of messages, and they were automatically retried, but obviously not something we wanted to keep getting alerted on. Interestingly we had technically had this issue for a few code paths for a while, but it wasn’t until we wholesale moved large parts of our codebase over to the read replica that this problem showed itself.

We discussed some options and quickly homed in on a well-documented tool in the PostgreSQL world.

## Log sequence numbers to the rescue

The log sequence number, or LSN, is a special pointer in PostgreSQL that represents a specific position in the [Write-Ahead Log](https://www.postgresql.org/docs/current/wal-intro.html). This means that you can use it to check whether the read replica has caught up with a specific write in the primary database. You get the LSN with a built in PostgreSQL function:

```jsx
SELECT pg_current_wal_lsn()::text

```

The basic technique we want to apply:

1. Grab the LSN from the primary after having done your write
2. Pass it to the code running on the worker
3. Compare the read replica LSN with the LSN from the primary, if the read replica is at or after that LSN, you know that you are past the point where your write happened

In order to enforce read-after-write consistency across our workers, we started stamping each published message with the LSN from the primary at the time of publishing. Grabbing the LSN is very cheap, it’s just an in-memory operation on the PostgreSQL side.

Having rolled that out, every message now has this stamp on it. So then we added a check on the subscriber side that compared the LSN on the message with the LSN of the read replica. The LSN comparison can be done inside of a SQL query using the [native data type](https://www.postgresql.org/docs/current/datatype-pg-lsn.html):

```jsx
SELECT pg_last_wal_replay_lsn() >= $1::pg_lsn

```

If the replica was caught up, we are fine to process it. If it isn’t, we just kick the message back to the queue.

That last part actually turns out to be a great back pressure mechanism in overload situations. If the read replica is failing to keep up with the primary, for whatever reason, instead of directing all queries to the primary and risking overloading it, we just keep nacking the messages until the read replica is feeling better again, with exponential backoff to avoid overload. It turns a potential outage situation into a graceful degradation instead, all the work is processed as expected, just with a delay.

If you’re doing this, you want to couple it with some dashboards and alerts. You don’t want to get caught out on the replica lagging behind. We often see people measure read replica lag in time, but that’s not a useful measurement. A read replica can catch up 5 minutes of delay in seconds because the rate of change has slowed down, or it can struggle to close the distance on 30s of delay because the rate of change is going up. Basically, the replay rate can change. A more useful measurement is the number of bytes behind. It still doesn’t quite convey whether you have a problem or not, but it’s more consistent than measuring it as time.

No more random `NotFoundError`, great success!

## Stamping LSN on users

After having tackled the events and subscribers, we looked around for other places where we would expect to bump into the same consistency problem, where we need to be careful to read our own writes, and we identified the API as another surface area to deal with. This included our public API where we support creating resources, returning an ID, and then reading the resource back. Done quickly you’d risk a 404. Additionally, our dashboard relies on our internal private API and we frequently use the same pattern there: create an empty resource and return the ID, queue the work to actually create the resource, frontend polls the API using the ID, resource is eventually created and displayed in the UI.

There are a few different [classic blog posts]( https://brandur.org/postgres-reads) that talk about using LSN stamping on API requests, so this is not exactly new territory. We want to apply the same principles as we did for our events, but we can be a bit more elegant about it for the API. Thinking about the problem we’re solving, it’s basically focused on the case of a `POST` followed by a `GET`. Or more generally, a mutating request followed by a read. Since we’re pretty consistent about HTTP verbs in our API, we can actually limit LSN stamping to mutating requests: `POST`, `PUT`, `PATCH`, `DELETE`. Additionally we can limit our scope to a given actor, whether it’s a user, an API key, or something like our mobile app, as the domain of consistency. If I create a document I want to see it immediately, but it’s ok for my coworker to have a few milliseconds delay to see the same document.

We added a middleware that runs after the request handler in our [Goa web layer](https://goa.design/), checks the method of the request, and optionally stamps the actor **with the LSN. We store this directly in PostgreSQL, using the native data type.

```jsx
UPDATE users SET read_after_write_lsn = GREATEST(read_after_write_lsn, pg_current_wal_lsn()) WHERE id = ?

```

This has the benefit of us already loading that row anyway during auth at the start of every request, so we could include this column there. This is also why we chose not to move this work to a key value store like Redis. Redis would add a network request, looking up the LSN, to every single incoming request, and we still need to get the latest LSN from replica too.

We shipped the middleware and with our actor tables quickly filling up with LSN stamps, each representing the last action each one has taken, we tackled the other half of this. We added another middleware that runs after auth and takes the LSN stamp from the actor and compares it with the read replica. Unlike subscribers, we can’t just nack the message and send it back to the queue to be processed later. People tend to not want to have to retry all their requests, and we didn’t want to push this on every client to our system, of either having to retry on a certain status code, or carry LSN stamps on headers. So instead of nacking, we fall back to the primary. We may need to rethink that at some point, when we can’t afford to go to primary anymore, but after rolling this out the actual impact on the primary is minimal. Only about 0.01% of requests to our API fail the LSN check.

With this we could move the last part of our codebase over to the read replica, reversing the trend of increased CPU utilization on primary week over week and getting it back down to a healthy level with all that extra capacity that we aim for.

## Optimizations

After having rolled this out we went after two optimizations that we had identified while we were working on this project. Although fairly simple, we didn’t implement them before rolling out because we 1. wanted to avoid premature optimizations, and 2. doing it after means we get a pretty graph and clear validation of the efficacy of the optimization. Everyone loves a pretty graph.

The first one was to limit the number of nacks on the event subscribers. As mentioned before, we saw between 0.1 and 0.5% of processed events getting sent back to the queue due to replica lag. Each nack causes a delay of at least 10 seconds in processing the message, since that’s our default retry delay. That’s very much acceptable, but we had a theory: that when we had replica lag the lag is almost always very small.

So whenever we check the replica and it’s behind, we added a 100 ms sleep before trying again. The impact was striking, almost eliminating nacks. Over time we saw the nack rate drop to ~0.03%. For what was basically a couple of lines of code. In real terms we’re not talking about a lot of messages, but it was a worthwhile improvement. PostgreSQL 19 actually comes with this [functionality built in](https://rednafi.com/system/wait-for-lsn/), with the new `WAIT FOR LSN`.

Secondly, when investigating why certain queries, that were actually perfectly safe to run on the read replica, were getting directed to the primary we realized our naive approach to detecting mutating queries was maybe just a little bit too naive. We were missing out on tons of juicy queries that were absolutely eligible for the read replica. Taking inspiration from the [https://github.com/pgplex/pgparser](https://github.com/pgplex/pgparser) library, we created a small tokenizer to replace our heuristics. Armed with the tokenizer, we could now confidently identify any queries that were safe to run on the read replica. When we rolled it out we saw a massive drop in CPU utilization on the primary, and a corresponding increase on the replica side.

## Database connection pool routing strategies

So now we’re in our new better world where everything is opted into the read replica by default, we have read-after-write consistency, and thanks to LSN stamping, that applies across events and subscribers, and the API as well.

But we had two remaining concerns: firstly that our code was still explicitly opting into the read replica everywhere, and secondly that internal concerns were leaking: anyone who needed specific database pool behaviors had to understand exactly how all of this worked. We didn’t want to have to maintain this for the team permanently, or introduce friction for everyone. So we went after the last major part of the project: inverting the default and creating a “DSL” for choosing database connection pool routing strategies. Everything goes to the read replica, whether you’re aware or not, and the mechanisms we’ve implemented ensures your code just works.

The easy part was tweaking our database handles with internal pool routing, we changed it to be enabled by default. The bigger part was giving the engineering team the tools to tweak the behavior where they needed to. We identified four strategies:

* `ReadAfterWrite` - this is our default behavior. Track mutations and route to primary after.
* `StaleRead` - where you have reads after writes, but you’re fine with them going to the read replica.
* `Primary` - pin the context to the primary and send all queries there. We use this where we can’t tolerate any kind of lag, primarily around our on-call product.
* `Replica` - pin the context to the replica and send all queries there. This is kind of a nuclear option, it will force queries to the replica even when the replica can’t execute them. Useful where you’d rather fail loudly and fix your code.

We accept these four strategies on every different level. The database handle takes strategies, and so does the subscriber definitions and the web layer API endpoint definitions. You can also override strategies per context.

This gives us the inverted default, everything goes to read replica, while still providing the engineering team with the tools they need to keep shipping.

## Outcome

Once every part of our codebase was “opted in” to the new read replica behavior, we inverted the default. Everything now goes to the read replica, and the people writing code don’t have to worry about it. It just works. Getting there is not hard, but it did come with a lot of learnings.

So what about the numbers? After the project wrapped up more than 60% of all read queries go to the replica. Of the ones that don’t, the majority were explicitly pinned to the primary. Only a small number of reads are actually routed to primary to preserve read-after-write consistency.

**We cut CPU utilization on the primary in half.** Looking back over the last few months CPU utilization had been growing significantly week over week as our customer base has been expanding. This project reversed the trend and we’ve had several weeks of CPU utilization going down week over week. A healthy primary with lots of spare capacity means that we can keep handling disaster situations where half the internet goes down and everyone gets paged.

![C](https://cdn.sanity.io/images/oqy5aexb/production/58bb0c9272b3bf2a8759f44f38f89ff0d0cbb8e5-1866x590.png)

**Replica lag causes back pressure instead of overload.** In a catastrophe situation where our replica is failing to keep up, or going down, all events are safely buffered in our queue service and only picked up when the replica is ready to get back to work. This means that problems with the replica do not spread to other parts of our stack.

**We’ve set the stage for horizontal scaling.** The door is now wide open for us to add additional read replicas, giving us a clear path to keep scaling our product for the future.

**Engineering can keep shipping.** Our transparent, opt-in by default, database pool routing strategies and read-after-write consistency mechanisms ensure that the read replica does not get in your way. You could work here for months without even realizing it exists.

We hope this writeup can help demystify read replicas and how to get the most out of them, applying fairly straightforward techniques to ensure read-after-write consistency. We really enjoyed working on this project and we hope you’ve enjoyed reading about it! If you’re interested in this kind of thing, come join us, we’re hiring!