The Estuary runtime
Discover the underlying tech behind Estuary's best-in-class data movement.
- 200GBper hour, per worker
- Scaleat any time
The engine underneath every pipeline
You describe the pipeline. The runtime handles the rest.
Most streaming systems fix parallelism at write time. Partitions get decided when data lands, and each one maps to exactly one consumer. That works until you need more capacity, at which point a layout chosen months ago becomes the ceiling.
Estuary's runtime decides grouping at read time instead. Storage layout and compute scale independently, so a task adds shards without repartitioning anything upstream.
Durable log at the center
Captures write into collections. Collections are a durable ordered log in object storage. Materializations read from that log.
What happens between the log and the workers reading it is the secret sauce.
Compute scales on its own
Add shards without touching the storage layout. Capacity is not a decision you make once, at the start, forever.
Reads and writes are independent
A dead destination doesn't stop the capture. The materialization resumes where it left off.
Backfill and streaming are one path
Same log, same order. No batch job to reconcile, no window where the two disagree.
Up to 8x faster backfills
Even faster on destinations that parallelize
Maintains transactional consistency
Clear observability on backfill progress
Up to 200 GB per hour per worker
Rewriting the runtime
Sometimes the best-laid plans are iterative. We've tuned up the Estuary runtime to bring you the fastest data possible at any scale.
Read the runtime whitepaper
Learn about the tech behind the runtime and the story behind the rewrites. Estuary:
Scales data movement to handle high throughputs
Ensures end-to-end transactional consistency
Decouples storage layout from shard grouping





























