New List View Improves Collection Observability

Estuary’s bindings view has expanded in capture and materialization task overview pages. The new view lists all of the connector’s associated bindings and their corresponding data collections, their status, data volume, and freshness, with new filtering and sorting capabilities. This gives you an at-a-glance window into the health of your bindings.

The exact view differs slightly for captures and materializations. The capture list includes:

• Source streams
• Docs written
• Data volume written
• Time last data was received

Materialization lists include:

• Docs read
• Data volume read
• Bytes behind*
• Time behind*

And both task types include:

• Collection names
• Status
• Filtering by collection name and status
• Sorting by collection data volume or freshness
• Ability to download table as a CSV

Check out the new view in the dashboard: simply select one of your captures or materializations and scroll down to the Collections section on the task overview page.

* Tracking bytes behind and time behind is a new materialization feature based on Estuary’s new runtime. Materializations must be on the latest runtime to report these values.

Why It Matters

The new view provides a more detailed breakdown on exactly where data is being written and read rather than lumping volume and freshness together for the overall connector task. Along with the ability to quickly search for a binding’s status or sort by time behind, this lets you troubleshoot specific bindings with greater accuracy.

Observability is always an important topic for any data architecture and Estuary wants to ensure you have options without inundating you with noise. This update provides a handy dashboard view of our collection metrics. You can also access these metrics directly to slice and dice data your own way and import it into tools you already use:

• See our OpenMetrics API for imports into Prometheus, Datadog, and other observability platforms
• Or directly access logs and stats via the CLI with flowctl

Whether you lean more programmatic or UI-side, you now have insights on data volume and freshness broken down by collection.

FAQs

What are all the status options a collection might display?

Collections can display one of several statuses in the list view:

• Enabled: the binding is enabled and data has been observed successfully moving through it
• No data: the binding is enabled but no data has been observed within the selected time frame; this may be expected for bindings with infrequent data updates or may indicate a blockage in your data flow
• Disabled: the binding is not currently enabled; to enable, you can edit the connector task to include the binding in the capture/materialization

How do I troubleshoot stale data in a collection?

A binding may indicate “No data” for a number of reasons:

• There haven’t been any updates to the table or resource in the source so there’s nothing new to send downstream. You can try checking the source or updating the timeframe.
• A SaaS capture binding may have a high interval and so may sync infrequently by default. Considering any tradeoffs, like rate limiting or API charges, you may set a more frequent sync interval.
• Materialization bindings may specify a backfill priority. If another binding has a higher priority and is still backfilling, the materialization may not have reached the “No data” binding yet. You can update binding priority via the specification.

How can I export collection metrics to other tools?

You have options for both one-off and continuing exports:

• Export a CSV of a specific task’s binding list with associated collections and stats via the collection list download button.
• Use the OpenMetrics API to integrate with observability platforms
• Pull current logs and stats via flowctl

Sort, slice, and dice connector bindings for an all-in-one view of data collection statuses.