Why we rebuilt the engine behind Estuary

For as long as data integration has existed, you've had to choose. You could move data fast or you could move it correctly, and every data engineer I know learned to live somewhere in the middle.

We spent the last year removing the choice by rebuilding the Estuary runtime from scratch.

It’s the same platform and the same connectors, with a very different engine underneath.

I want to walk through what we changed and why it was worth a year of engineering time instead of a year of features.

Why correctness is not optional anymore

Nobody bought a data pipeline because it was correct. Teams bought it because it was easier than building their own, and for a long time that was the right call.

When a human reads a dashboard, small inconsistencies are survivable. Someone notices the number looks off, refreshes, then asks a colleague. There's almost always a person in the loop acting as the error-correction layer.

That layer has been slowly fading away. More and more of what reads your warehouse isn't a person anymore. It's an operational system, or an agent, making a decision without pausing to sanity-check the row it just read. This means that a partial transaction stops being just a confusing chart and rather it becomes a decision made on data that was never correct in the first place.

The trade streaming systems make

Distributed systems scale by giving something up.

You get ordering, but only within a partition. The system guarantees exactly-once, but it ends at a partition boundary. You get transactional writes, paired with no transactional reads. These are all “reasonable” engineering trades, but they're also why so many teams run reconciliation jobs on the side, and why "eventually consistent" shows up in so many docs.

When we specified the new runtime, we wrote the requirements to forbid those trades. Four of them, specifically:

1. Reads stage on local disk, so they run far ahead of processing without running the machine out of memory.
2. One coordinated checkpoint across every worker on a task, moving as a single unit.
3. Idempotent recovery: after any crash, restart from that checkpoint with no duplicates and no gaps.
4. A reader-side atomic view of a transaction, even when it spans many logs.

Every architectural decision after that followed from refusing to weaken those four.

What actually changed

In the rebuilt runtime, the whole transaction loop moved from Go to Rust, coordination moved to a single per-task leader, and staging moved from memory to local disk.

Rust handed us a lot out of the box. No garbage collector, so no GC pauses in the middle of a transaction and memory we control and that control is most of why staging could move to disk cleanly: we manage the buffers, hand out the on-disk blocks zero-copy, and back-pressure turns into a disk-space policy instead of an out-of-memory crash.

Among the many, two consequences are worth emphasizing.

The first is throughput. On a single shard (a unit of compute), the new engine moves data up to 8x faster than the old one and the new architecture allows for essentially infinite horizontal scalability. Backfills are the critical example, as enterprises cannot afford days or weeks just to get their existing data ready for their AI agents.

The second is correctness. You can begin a transaction in your source database, update rows across several tables and finally commit. The new runtime delivers that to your destination as one commit containing the whole transaction, however many workers handled it in between. At no point can a downstream reader see half of it. The mechanism is a set of causal hints: an acknowledgement names the sibling logs from the same source transaction, and the runtime holds visibility until every one of them has been read through, then promotes the whole frontier at once.

Underneath both of those is a decoupling that is probably more interesting than the speed number. Because grouping is decided when data is read rather than when it's written, parallelism is no longer tied to how your storage happens to be laid out.

It runs itself

Collection partitions that stay throttled under sustained load now split themselves, up to eight ways, completely autonomously. Manual memory and buffer tuning is gone, because disk-backed staging replaced the in-memory ring buffers entirely. Materializations report how far behind they are, in bytes and in wall-clock time, so "is the backfill done yet" is something you look at instead of something you infer. Restarts hand off in-flight work rather than discarding it.

For you, and for our support team, that's a real cut in the number of things that can go wrong at 2am.

Where we’re going

New pipelines already run on the new engine by default. Existing ones are moving over gradually, one task at a time, with zero downtime.

The new runtime lays down the groundwork. When the cost of coordination stops scaling with your storage layout, a lot of the things we'd shelved become possible. What we build on top of it over the next year is going to be the real story, especially if you’re interested in the intersection of AI agents and data infrastructure. This week is the foundation.

Meet the new Estuary: https://estuary.dev/runtime