Improved Defaults and Customization for Field Selection

Creating the Perfect Destination Schema Shouldn’t Be Hard

Estuary’s latest platform update overhauls field selection to provide smarter defaults and fine-grained customization. Taken in tandem, these tools allow you to easily prune complex document schemas while pinning down the fields that really matter.

New and updated features include:

• Priority-Based Defaults: A new priority algorithm chooses default fields to materialize, weighing a field’s selection and rejection reasons.
• Depth-Based Selection Modes: Indicate broad customization preferences by including fields based on field depth in the source schema.
• Intuitive Fine-Grain Control: Require or exclude fields at the individual field level for complete control.
• What You See Is What You Get: Quickly review field outcomes to validate your destination schema before populating data.

These features streamline the field selection process while still allowing you to have the final say on your schema.

A New Look

The field selection table now has a streamlined design with clear require/exclude options and outcomes. Easily see what’s being materialized and quickly make changes with modes, an Exclude All button, and individual field actions.

Estuary will automatically select certain fields for you to get you started. These selections are based on a priority model and a default field depth of one.

Conflicts—when fields match selection and rejection criteria—are clearly labeled and provide tooltip information on reasoning.

Getting Started

To check out the new field selection flow:

1. Start by editing or creating a new materialization.
2. In the Source Collections section, select one of the collections associated with your materialization.
3. In the collection’s Config tab, scroll down to the Field Selection table.

Here, you can view your collection fields, require or exclude them, and quickly review each field’s outcome—whether it will be materialized or not.

Want to affect a wider swath of fields at once? Exclude All to start fresh or choose from one of the depth-based modes:

• Depth Zero only selects top-level fields
• Depth One or Two selects nested fields up to the appropriate depth
• Unlimited Depth selects all fields by default

📚 Check out a full guide on customizing your materializations with field selection.

FAQs

Does this update affect my existing materializations?

No, your existing materializations will retain all their current settings. New field selection functionality will only be relevant once you start editing a materialization or create a brand new one.

What reasons does Estuary use to select or reject a field by default?

Estuary considers a number of criteria for each field that count for or against that field’s inclusion in the materialization. These reasons are ranked by priority and compared against each other. Reasons can include a field being a key, being user-defined, or can be based on user or connector requirements. Estuary surfaces relevant criteria to the user if there are any conflicts.

Can I change my materialized fields if the defaults don’t work for my use case?

Yes, Estuary provides depth-based modes to quickly change selection criteria across a range of fields as well as require and exclude options for individual fields. More drastic changes, starting from the source system, may require a backfill or dataflow reset.

With intelligent defaults that just work for most occasions as well as detailed customization options, Estuary gives you reliable, intuitive destination schemas.