The Next Iteration of Estuary’s Runtime

Get up to speed faster with improved backfill efficiency.

Estuary has made a major overhaul of the platform runtime, led by Estuary’s CTO and the driving force behind Gazette, Johnny Graettinger. Intended to improve Estuary’s parallelization capabilities, the updates support faster, more efficient data pipelines, as well as breaking ground for highly-anticipated feature requests.

Here’s some of what’s new:

• Faster initial loads
• Supports higher data volumes
• Updated metrics
• Expanded sync schedules

And all of this higher capacity maintains transactional integrity, so your pipelines can support end-to-end transactional consistency even at top speed.

Whether you’re interested in the underlying architecture or just the end effects, we have new resources to cover it all.

Up to 8x Faster Initial Loads

We’re always happy to receive positive customer feedback, whether it’s about opening up new opportunities by supporting both batch and real-time data or keeping data costs under control. But one con we kept hearing was that it took a while for the initial data load to complete.

Coltene, for example, noted that Estuary’s initial load was 2x slower than a leading competitor they trialed, even though Estuary became an order of magnitude faster once the platform got caught up.

Data engineers can’t wait around on a backfill when trialing out a proof of concept or performing a migration. So, Estuary got to work.

Estuary’s runtime enhancements were specifically designed to address this weakness, with:

• 2-8x faster speeds when running tasks on a single thread
• Additional improvements on top of this for destinations that support multi-threaded behavior
• End-to-end transactional consistency maintained even at scale
• Ability to handle higher volumes, up to 200 GB/hour per worker
• With the upcoming ability to add workers, this will allow essentially unlimited scalability

These improvements cancel out the former cons and then some, making it easier than ever to migrate your architecture and get your data moving. Estuary won’t be your data bottleneck.

Curious about the numbers? Keep an eye out for an upcoming benchmark study that will explore these performance changes in detail.

The Architecture That Makes This Possible

To pull off such a feat, we implemented some tricky architectural logistics.

Estuary is already decoupled by design, with separate control and data planes as well as separate capture and materialization tasks that allow you to flexibly configure one-to-one pipelines, many-to-many, or anything in between. The new architecture takes this concept a step further, down to the shard level.

Scaling out computational processes to support large workloads always boils down to a basic solution: parallelize the work. Tasks get broken down into smaller pieces, separate shards work their way through their own piece of the task, and then they come back together for the final result. The challenge is ensuring correctness when different shards may process different inputs out of their original order.

Many real-time data movement systems such as Kafka, solve this by mapping parallel partitions, which are fixed at write time, to exactly one consumer. While this ensures operations are stateful, the structure is rigid. It doesn’t support changes once you get started, like adding new partitions if the load grows.

Estuary instead decouples storage layout from grouping, allowing the reader to decide grouping rather than the writer. This lets compute scale independently while maintaining low-latency, stateful streams.

How do we manage this read-time shuffle and maintain transactional consistency? Find the full story on our architectural improvements in our new whitepaper.

What Else It Unlocks

A faster, more scalable, and more efficient platform was reason enough to make these changes. But while we were restructuring our runtime anyway, we also decided to make sure the stage was set to support additional new features.

Here’s what we’ve been able to accomplish or are currently working on thanks to the runtime redesign:

Improved Metrics and Observability

In the past, it hasn’t always been clear when a backfill has caught up, or it’s required careful checking on the bytes read in and out of various components of your pipeline.

The runtime improvements have now allowed us to improve our tracking capabilities as well. Materializations on the new runtime will display bytes behind and wall-clock time behind so you can see exactly how far ahead your captures are. This provides a precise view of your pending data backlog without guesswork or approximations.

This feature is out now. To see these new fields in the materialization details page, related tasks must be running on the new runtime.

Expanded Sync Schedule Capabilities

Sync schedule settings were originally implemented on the connector-level. With the new runtime updates, we’ll be pulling sync schedules into the runtime instead.

What does this mean? Essentially that we’re expanding sync schedule capabilities, and those improvements will be available for all materialization connectors rather than individual ones. And “all” means all. Sync schedule configuration will be available for every materialization, not just data warehouses.

You will also be able to set multiple scheduling intervals for your syncs, so you can choose latency for off-hours, business as usual, and extra-time-sensitive intervals. If you need to push a one-off sync, you’ll have a “Sync Now” button that won’t disrupt your regular schedule.

This feature is coming soon to a dashboard near you.

FAQs

How can I try the new runtime?

You probably already are, at least in part! The updated runtime is meant to be a smooth backend transition that simply unlocks faster backfill speeds and extra features for you.

All newly-created captures and associated materializations use the new runtime by default. Existing tasks are being rolled over gradually.

When are the runtime updates rolling out?

Right now! We’ve been running a gradual rollout to ensure the runtime updates get stress-tested and hardened every step of the way. Capture tasks are being migrated first with materializations and derivations to follow.

Additional improvements, such as finetuning individual materialization connectors for multi-sharded behavior, and extras, like new sync schedule features, will be coming in the following weeks.

How can I tell if my existing tasks are on the new runtime?

Check your task specification. The shards stanza may indicate which version of the runtime the task is using:

json
  "shards": {
    "flags": {
      "enable-runtime-v2": "true"
    }
  }

If enable-runtime-v2 is set to true, the task is on the new runtime; if false, it’s on the old runtime.

If you don’t see the shards stanza in your specification or it doesn't include the runtime flag, you can reach out to the Estuary team to check the status of your tasks.

Learn more about the migration in our docs.

Start sooner and stay up to speed with your data, whatever your volume.