What Is Estuary? A Platform for CDC, Streaming & Batch Data

In short: Estuary is a right-time data platform. It sits between source systems and the systems that consume their data, supporting CDC, ETL/ELT, streaming, and scheduled data movement within the same platform. Estuary captures data once from databases, SaaS apps, and event streams into durable, schema-validated collections in your own cloud storage, then delivers that data to as many destinations as you want, each at the latency it actually needs. One system covers change data capture, real-time data delivery, and batch data integration.

Data teams inherit a trade-off where batch pipelines are simple to run and too slow for anything that has to react. On the other hand, streaming pipelines are fast, and famously fragile, expensive, and demanding to operate. Most organizations resolve this by running both, separately. That means two vendors, two failure modes, two on-call rotations, and two systems that give different answers about the same data.

The underlying issue is that data arrives on the pipeline's schedule instead of the business's. An hourly sync means your warehouse can be 59 minutes behind reality, and everyone downstream quietly learns to simply work around that.

That was tolerable when the consumer of the data was a human reading a dashboard on Monday morning, but it is much less tolerable now that the consumer is often an AI agent taking action. Stale data used to produce a stale report and now, it produces wrong decisions.

The core idea: capture once, deliver many times

In a modern data stack, Estuary sits between the systems that generate and store data, including applications, databases, and data warehouses. Its primary role is to move data between these systems using ELT, ETL, and CDC, either continuously or on a schedule, such as hourly.

Estuary is built around a single principle: capture data once, store it durably, and deliver it as many times as you need, at whatever cadence each destination actually requires.

The four parts of an Estuary data pipeline

Why collections are important

The collection is the piece that makes everything else work. Because the data is stored durably rather than passed through:

• The source database is read once, no matter how many destinations you eventually add.
• If you stand up a new warehouse next year, you backfill it from the collection's history instead of taking another snapshot of production.
• If a destination goes down for six hours, it catches up from the collection when it comes back.

Right-time data movement: real-time and batch from the same capture

On top of that we put latency control where it belongs, which is on the individual pipeline. We call it right-time data movement.

One materialization from a collection can deliver continuously at sub-second latency for an operational system. Another materialization from the same collection can load on a nightly schedule for the finance team, because nobody needs the monthly close to update every 400 milliseconds. Same source, same capture, same collection, different delivery cadence per destination. You set the freshness target and we handle the mechanics.

What teams use Estuary for

Estuary is more than just data movement and that reflects well in how our customers utilize the platform. Here are some of the main use cases:

• Analytics: Keep warehouses and lakes current for dashboards, reporting, forecasting, and dbt models. Teams can stream changes continuously where freshness matters and schedule less time-sensitive workloads such as finance reporting from the same captured data.
• Operational use cases: Move changes between production systems, including databases, search indexes, customer-facing applications, and event-driven services. This supports use cases such as inventory visibility, fraud detection, order tracking, and customer 360 without routing everything through the warehouse first.
• AI: Keep vector databases, feature stores, and model pipelines synchronized with changing operational data. Teams use this for their RAG over current documents and tickets, real-time personalization, model monitoring, and agents that need fresh account, product, or inventory context before taking action.

Platform cost is an architecture question

Real-time data is usually sold as a premium tier. The common pricing model in this category bills on monthly active rows, which charges you more precisely as your data becomes more valuable, and penalizes you for syncing frequently. Under that model, the honest way to control your bill is to accept staler data. I have watched teams make that trade. It is a bad trade to have to make.

Estuary bills on data movement instead. Cost tracks the volume that actually moved, not how often you asked for it. Three things follow from that:

1. Since change data capture moves changed rows rather than whole tables, the bill reflects your rate of change.
2. Capturing once and fanning out means the fifth destination does not cost you a fifth extraction from the source.
3. Moving a pipeline from nightly to continuous is a configuration change rather than a budget conversation.

Current numbers are on our pricing page, including a free tier large enough to run a real proof of concept against a production database.

What Estuary does: CDC, streaming, batch, and SaaS in one platform

Change data capture (CDC) from databases

For databases, Estuary reads the transaction log directly. The write-ahead log in PostgreSQL, the binary log in MySQL, native CDC in SQL Server, change streams in MongoDB, with Oracle also supported. Reading the log rather than polling tables means no query load on the source, and it means deletes are captured correctly, which polling-based tools tend to miss. A capture typically starts with a backfill of current state and then continues with ongoing changes.

Batch and SaaS sources

Plenty of systems have no change log to read. For REST APIs, object storage, files, and SaaS applications like Salesforce, HubSpot, and NetSuite, Estuary pulls on a schedule or incrementally. Those sources land in the same collections and feed the same downstream materializations, so a batch source and a CDC source look identical to everything after the capture.

Streaming and Kafka compatibility

Collections can be read as Kafka topics through Dekaf, so existing Kafka consumers work against Estuary without you running brokers. Kafka is also available as a source and a destination.

Here's the summary of how those three paths converge:

All four land in the same collections and everything downstream of the capture is identical.

Transformations with derivations

Derivations let you filter, join, aggregate, enrich, and reshape data in the pipeline using streaming SQL or TypeScript, with Python available in private and bring-your-own-cloud deployments. Estuary does not require you to transform in flight. Many teams land raw data and model it in the warehouse with dbt, and that is a perfectly good architecture.

Schema validation and schema evolution

Every collection has a schema, and documents are validated against it. Connector discovery infers schemas from the source, and compatible changes such as new fields generally flow through without anyone intervening.

Connectors

More than 200 native connectors covering databases, SaaS applications, event streams, warehouses, lakes, and operational systems. See the full list of connectors here.

Deployment

Estuary runs as a public cloud service, as a dedicated private deployment, or as a bring-your-own-cloud data plane in a cloud account and VPC you own. Estuary manages the public option for you. Private deployments provide isolated data-plane infrastructure in your chosen AWS, GCP, or Azure region, while BYOC gives you ownership of the underlying cloud resources with Estuary managing their deployment, updates, and scaling.

In every deployment model, the durable data behind collections is stored in object storage in your own cloud account. With private and BYOC deployments, the compute that captures, processes, and delivers that data also stays inside the private data plane. The SaaS control plane remains separate and manages pipeline specifications and orchestration without access to the data moving through the private data plane.

Data is encrypted in motion and at rest, connector credentials are encrypted, and internal communication uses mutual TLS. Role-based access control, field redaction, IP allowlisting, IAM authentication, PrivateLink, and SSH tunnels provide additional controls over who can access data and how systems connect. The platform is SOC 2 Type II certified and supports HIPAA, GDPR, CCPA, and CPRA compliance.

How you work with it

You can build pipelines visually in the web UI, or manage them as version-controlled specifications with the flowctl CLI and run them through CI/CD like any other code. We also publish Agent Skills, so you can build and debug pipelines by asking an AI assistant.

It works with Claude Code, Cursor, Copilot, and 30 or so other tools through the open source agent skills.

Under the hood: Gazette and the Rust runtime

Gazette: data as files, not as broker disk

The foundation of all of this is Gazette, a streaming broker Johnny built at Arbor and open-sourced after the acquisition. It has been running production workloads for about a decade.

What makes Gazette unusual is that it stores data as ordinary files in cloud storage rather than on broker-local disk. A new consumer can therefore backfill petabytes by reading files from object storage, without proxying those reads through the brokers that are serving live traffic, and without risk to anything already running.

The runtime: read-time grouping and whole transactions

The Estuary runtime that sits on top of Gazette is written in Rust for maximum performance.

Parallelism is decoupled from storage layout. Partitions are assigned to consumers at read time rather than fixed at write time, so storage layout and compute scale independently and you never have to repartition a collection to add throughput. A shuffle leader coordinates distributed transactions across the shards of a single task, which is what lets one materialization scale horizontally instead of forcing you to split your work across several.

The throughput numbers are per shard. A single worker sustains on the order of 200 GB per hour. That is the unit that matters, because shards scale out horizontally without a ceiling: you are not tuning a single big worker to go faster, you are adding shards to a task that already coordinates them.

Destinations see whole transactions. The runtime manages transaction boundaries that stretch from milliseconds to hours and from kilobytes to hundreds of gigabytes, using destination backpressure to size transactions and reducing source changes into their cumulative effect within each one. Two things follow. Destinations see complete transactions rather than fragments, so a system reading your warehouse is not acting on half of a multi-row change. And a destination only has to absorb one write per key per transaction even when that value reflects thousands of upstream documents, which is how a busy CDC stream can be safely landed in something that accepts updates every few seconds.

It tunes itself. When a collection's journal shows sustained append throttling, the runtime splits it automatically, up to eight ways, so hot partitions do not require hand-tuning. Memory and buffer tuning is gone, because staging moved from memory to local disk. None of this requires configuration.

Where to start

If any of this sounds like a problem you have, the fastest way to evaluate it is to run it.

1. Point a capture at a real database. The free tier covers a real pipeline against a real source, and setup for most sources is a matter of entering credentials and choosing what to sync.
2. Materialize it somewhere you already look. Pick a destination you'd notice being wrong.
3. Then change the cadence. Turn one materialization continuous and leave another on a schedule. That is the part that's hard to appreciate from a diagram.

The documentation is thorough, and Estuary 101 is a recorded walkthrough if you'd rather watch someone do it first.

We are still working on the same problem we started with in 2019, which is making it unremarkable for a company's data to be in the right place at the right time. Tell us where it is not working for you.