How CDC Works in Estuary: Source to Destination Explained

Change data capture (CDC) reads every insert, update and delete from your database's log and streams it to your destinations. Here's how Estuary does it, from source to destination.

We follow one order placed by Will through an Estuary pipeline: from the moment it's written to a Postgres database, through captures, collections and materializations, to the moment it lands in a warehouse, search index or AI model.

You'll learn:
- How log-based CDC reads the write-ahead log (WAL) as a second reader, with no extra load on your production database
- How the initial snapshot and live WAL streaming run in parallel, so existing rows and new changes both arrive
- Why polling-based methods miss deletes, and why log-based CDC doesn't
- How schema validation works, and how Estuary adapts to schema changes and pauses only for breaking ones
- What collections are: a data lake and a real-time stream at the same time
- How collection keys keep a full history of every change
- How derivations create new collections with stateless or stateful transformations
- How materializations deliver data to each destination at its own speed, from real-time to scheduled batches
- How read, reduce, write cycles and reduction strategies work, with last-write-wins as the default
- How exactly-once delivery keeps data correct, even if something crashes mid-write

This is Episode 1 of our series on how Estuary works under the hood.

Chapters
0:00 What happens to data between source and destination?
1:00 What is a capture?
1:49 How log-based CDC reads the write-ahead log (WAL)
2:12 Initial backfill: snapshot and WAL in parallel
2:44 Why polling misses deletes
3:09 JSON documents and schema validation
3:31 How Estuary handles schema changes
4:07 What is a collection?
4:52 Collection keys and change history
5:25 Derivations: stateless and stateful transformations
6:23 What is a materialization?
7:23 Choosing sync speed per destination
7:43 Automatic batching to protect destinations
8:06 Read, reduce, write and reduction strategies
8:36 Exactly-once delivery
9:00 Key takeaways

Links
Try Estuary free: https://dashboard.estuary.dev/register
Estuary docs: https://docs.estuary.dev
Schema evolution guide: https://docs.estuary.dev/guides/schema-evolution/

#ChangeDataCapture #CDC #DataEngineering