Tune Alert Notifications and Thresholds by Environment

Silence noise where it’s not needed and ensure urgent issues get through.

Estuary has overhauled alerts and notifications so you have more control over:

• What notifications you want to receive
• Who should receive them
• Which environments or prefixes you want those settings to apply to
• When alerts should fire with finetuned thresholds

This lets you separate out configurations by prefix so, for example, you can route urgent production issues to an on-call email, send development task failures to a team mailing list for further investigation, and ignore idle test tasks.

These options are now available in the dashboard with additional configuration features available through Estuary’s CLI tool, flowctl.

Try It Out

To start configuring your notifications:

1. Navigate to the Admin Settings page in your Estuary dashboard.
2. Click Configure Notifications under the Organization Notifications section. The alert subscription modal will open.
3. Choose your prefix: the base tenant name (such as acmeCo/) will apply notification settings to the whole tenant. Typing in a subprefix (such as acmeCo/prod-env/) will apply your settings to tasks under that specific prefix. This is how you can separate default notification settings by environment, or even down to the task level.
4. Set desired global settings. These are default values that will apply to the chosen prefix. Currently, you can choose a default setting for Data Movement Stalled alerts.
5. Define one or more recipients to receive notifications. Each recipient consists of an email (this can be a mailing list or even a Slack-forwarding address) and the alert types they should subscribe to.
6. Save the alert subscription.

Additional threshold settings are available for task failures, chronically failing tasks, and idle tasks via flowctl.

For further details on available alert types, equivalent flowctl instructions, and threshold configuration, check out the docs.

Why It Matters

Monitoring and observability are important for any data stack to ensure data movement blockages are discovered immediately and not a week later in the middle of pulling analytics for the board meeting. It’s equally important to not drown in noise: too many alerts can be as ineffective as too few.

Since everyone’s architecture is configured differently and has different needs, the best way to ensure alerts work for everyone is to provide fine-grained configuration.

Estuary’s alert settings allow for this flexibility. You can now, for example:

• Assign an on-call email to receive notifications for production tasks (under a yourOrg/prod/ prefix) while assigning a less-urgent mailing list to development tasks (under a yourOrg/dev/ prefix)
• Set a default interval for alerts when data movement stalls rather than choosing the interval on individual tasks; this can also be set on a per-subprefix basis
• Choose for the same recipient to receive different alert types based on subprefix, such as receiving all alerts for production tasks while opting not to receive “Task Idle” or “Data Movement Stalled” alerts for test tasks
• Use flowctl to increase the task failure threshold for alerts when tasks frequently run into false-alarm failures

We’re committed to continuous improvement. Have an alert type you’d want to see represented? Wish there were additional alert thresholds you could configure? Drop a note in our Slack on what you want to see from us next.

FAQs

Estuary’s alerts reference tasks. What are they?

Tasks in Estuary are your pipeline’s active processes. Captures, materializations, and derivations are all tasks and can all fire alerts if something goes wrong.

How do I add tasks to specific environments so I can set up scoped alerts?

Alert scoping (along with permissions and grants) is based on Estuary’s prefix system. When creating a task, you can define the task name, including any additional path prefixes, such as a Postgres capture task called acmeCo/prod/orders-db/source-postgres/.

You can structure your paths however you like, such as by production/development environment or region, including nesting. For example, acmeCo/eu/prod/customers/source-postgres/ nests both a regional indicator (eu/) and an environment (prod/).

What happens if I’m subscribed to notifications through the base tenant and a subprefix?

Let’s set the scene. You configure notification defaults for your tenant acmeCo/ and set yourself as a recipient. You then configure different settings for the subprefix acmeCo/test/. The more specific settings for acmeCo/test/ will override the tenant defaults on matching tasks.

Focus on what really matters with configurable alert notifications.