How to Migrate a Production CDC Pipeline With Minimal Downtime and No Data Gaps

A CDC migration replaces the tool that reads changes from a production database, the destination those changes land in, or both, while the database keeps taking writes. To migrate a CDC pipeline without missing changes, give the new pipeline a complete baseline of its own plus every change after it, instead of trying to hand it the old pipeline's position. Start it with its own backfill and its own replication position, run it beside the old pipeline into separate tables, and compare the two at a boundary both have fully applied. Then switch consumers over in planned groups, and keep the old pipeline recoverable until the new one has run through a full business cycle.

Run that way, a production CDC migration avoids data gaps and limits consumer disruption to the switch itself. Source writes continue through the backfill and the parallel run, apart from one planned pause for validation. Both outcomes have conditions: the source has to retain enough log to cover the new pipeline's backfill, validation has to check deletes as well as rows, and the rollback path has to still work on the day you need it. The examples use a PostgreSQL to Snowflake pipeline.

CDC migration checklist

This checklist follows a parallel migration to a new CDC tool, using separate destination tables.

1. Inventory the tables and their keys, the source's reader limits and log retention, every consumer of the old tables (including streams and tasks), and each tool's delete and type conventions.
2. Give the new pipeline its own replication slot or MySQL server ID, with enough log retention to cover its backfill.
3. Start the new pipeline with its own backfill into a separate schema.
4. Wait for the backfill to finish and the lag to meet consumers' freshness requirements.
5. Pause writes, let open transactions finish, confirm both pipelines have applied every table's changes, and snapshot both outputs. Resume writes and compare the snapshots, deletes included.
6. Prepare and test consumer queries for the new tables, write down the rollback trigger, and name the person who decides on rollback.
7. Switch consumers over one group at a time.
8. Keep the old pipeline recoverable through a full workload cycle, and monitor both.
9. After the rollback window closes and no consumers use the old tables, stop the old pipeline, drop its replication slot and any publication or capture instance the new pipeline doesn't use, and archive or drop its tables.

What makes a production CDC migration risky?

A CDC pipeline keeps a position in the source's transaction log, such as a PostgreSQL replication slot's log sequence number (LSN), a MySQL binlog file and offset, or a SQL Server LSN. That position records how far the capture has read. A slot's confirmed position means the pipeline has taken a change, not that Snowflake has applied it. A migration has to replace the capture position, and the main risks come from doing that badly:

• Gaps: Changes committed after the old pipeline stops and before the new one starts reading never reach the destination.
• Duplicates: Changes that both pipelines deliver land twice when the destination appends rows instead of merging them on the key.
• Lost replication state: A slot dropped too early, or a binlog purged while a pipeline was paused, leaves nothing to resume from, and recovery usually means a new full backfill.
• Missed deletes: Checks that compare row counts and inserts pass while rows deleted at the source are still live in the destination.

A major upgrade or failover of the source database is a separate procedure, because the stored log position may not exist on the upgraded or promoted server.

How do you cut over a production CDC pipeline safely?

Keep the old pipeline serving consumers until the new one is proven, then move consumers in steps you can reverse. The table compares CDC cutover strategies and when each one fits.

A parallel run gets close to a zero downtime CDC migration on the read side, because consumers never wait on a backfill. Keep the repoint to a metadata change, such as pointing a view that consumers read at the new tables with CREATE OR REPLACE VIEW ... COPY GRANTS. Snowflake applies each replace atomically, but replacing several views isn't atomic as a group, and a stream on a replaced view goes stale. Plan the switch per consumer group: which views, jobs and streams move together, and in what order. Don't swap table names with ALTER TABLE ... SWAP WITH while both pipelines are running, because each writes by table name and would start writing into the other's table.

How do you backfill historical data while CDC is still running?

A pipeline records a position in the log, reads the existing rows, then streams changes from that position and reconciles the overlap, so that a row changed during the read is neither lost nor applied out of order. Incremental approaches, such as Debezium's incremental snapshots, read tables in watermark-fenced chunks, which avoids table locks and lets an interrupted backfill resume. On a large PostgreSQL database, plan for the WAL retained during a multi-day backfill.

Most tools can't pick up where another tool stopped. Each tool keeps its position in its own format and its own place: Debezium in its offset store (a Kafka offsets topic when it runs on Kafka Connect), Fivetran in internal state that its API doesn't expose for database connectors. A PostgreSQL replication slot also serves one consumer at a time, so while the old pipeline keeps running, the new one needs a slot of its own. That leaves three ways to start it:

1. Its own backfill, then stream. The new pipeline reads every row itself and can be checked against the source directly. It costs a full read of the source and enough log retention to cover it. This is the default.
2. Start from now and take over the old pipeline's tables. The new pipeline skips the backfill and inherits the rows the old one loaded. Start the new capture before the old pipeline stops, so the captures overlap, and have the new pipeline apply changes only after the old one has drained and stopped writing. Replaying the overlap re-applies changes the tables already hold, so rows can return to older values and deleted rows can reappear until the replay catches up. Use an apply that skips changes the destination already reflects, or pause consumers through the replay. The new pipeline also has to match the old tables' keys, types, and delete conventions.
3. A filtered backfill from a known point, such as a PostgreSQL backfill filtered on the row-version transaction ID (xmin). It reads only rows changed since a cutoff, though it still scans each table. Treat it as a recovery path: the cutoff has to come before every change the old pipeline hadn't yet applied to the destination, not just before the moment it stopped, and rows deleted in the gap need a separate check.

Whichever start you choose, apply changes to the destination as a merge on the primary key. Most CDC tools guarantee at-least-once delivery, so a change can arrive twice after a restart. A keyed merge makes the repeat harmless only when changes to each key are applied in order, or when the apply skips a change older than the version already stored. A source transaction that spans several tables can also reach the destination tables at different times, so decide whether consumers need related changes to appear together.

Should you run old and new CDC pipelines in parallel?

Yes, in most production migrations, for as long as validation and the rollback window need. Dual-run CDC pipelines let you compare two complete outputs and reverse the cutover by repointing. Skip the parallel run when the source can't carry a second reader, when a second full copy costs too much at the destination, or when the old pipeline is already broken.

What a second reader costs the source

The source pays the steady-state cost of log-based capture a second time:

• PostgreSQL: Each pipeline needs its own replication slot, and each slot retains WAL from its own position, even while nothing is connected to it. max_replication_slots and max_wal_senders both default to 10 and change only with a restart. Slot names are unique across the cluster.
• MySQL: Each binlog reader needs its own server ID. Binlog retention is global, so it has to be long enough for the slower pipeline's lag and for the new pipeline's backfill.
• SQL Server: Both pipelines read the same change tables, and one cleanup job applies one retention period to all of them. A table can have at most two capture instances, so have the new pipeline reuse the existing instance where it's compatible, leaving the second for a schema transition.
• MongoDB: Each change stream resumes from its own token, but both depend on the oplog still holding that history, so size the oplog window for the slower reader.

Separate or shared destination tables

Write the new pipeline into its own schema. Separate tables keep the two tools' conventions apart and make validation a comparison of two complete datasets. In shared tables, those conventions collide: Fivetran's default soft-delete mode flags deletes with _fivetran_deleted, Debezium's JDBC sink deletes rows only when delete. enabled is set, and the new tool adds its own metadata columns. If tables must be shared, disconnect the old tool rather than pausing it. A paused Fivetran connection runs any requested re-sync when it's re-enabled.

Schema changes during the parallel run

Hold source schema changes until the rollback window closes if you can. If one has to land, confirm that both pipelines handled it the same way before comparing outputs, and agree in advance how the new tool will handle schema changes after the migration.

How do you validate a CDC migration before cutover?

Validate in layers, at a boundary both pipelines have fully applied. Two pipelines are rarely at the same point while changes are flowing, so a comparison taken mid-flight shows differences that aren't errors, and a timestamp filter can't show a row as it stood before a later update. The default boundary is a planned write pause: stop writes, let open transactions finish, confirm that every table on both sides has applied the source's last change, and snapshot both sides with a Time Travel point or a clone. Then resume writes and compare the snapshots.

Validating while writes continue needs a process that tracks rows changing during the comparison, retries them, reconciles deletes, and reports every row it couldn't validate before cutover. AWS DMS validation, for example, counts rows that keep changing as suspended rather than validated.

Matching row counts are the first check, not the validation; Microsoft's migration guidance prefers checksums and hash functions for deeper checks.

Use the same hash function on both sides, over only the business columns both share: leave out each tool's metadata columns and filter soft-deleted rows first. In Snowflake, HASH_AGG hashes a whole table regardless of row order, and duplicate rows change the result. Cast columns to matching types first: in Snowflake's HASH, 10 and '10' hash differently, and equal numeric values can hash differently across types. Avoid XOR-based checksums, which cancel out pairs of duplicate rows. Across two engines, compute the same canonical row hash on both sides, as Google's Data Validation Tool does.

Cutover point, rollback window, and retiring the old pipeline

The cutover point is the moment consumers stop reading the old pipeline's output and start reading the new one's. Retiring the old pipeline comes later, and the rollback window runs between the two.

What defines the cutover point?

Cut over when all of these are true:

1. Every table in scope has a complete, verified baseline, and the new pipeline is capturing ongoing changes.
2. Validation has passed at a defined boundary on every layer, deletes included.
3. The new pipeline's measured lag meets the consumers' freshness requirement.
4. Consumer queries have been updated for the new tables: delete filters, metadata columns, and any type or naming differences.
5. There's a written rollback trigger and a named person who decides between rolling back and fixing forward.

How long should the rollback window stay open?

Keep the rollback window open for at least one full workload cycle, including month-end jobs, batch loads and peak traffic. Google notes that a fallback "might be required days or weeks after the switchover." Rolling back needs the old tables, the consumers' old queries, and an old pipeline that can still catch up. A stopped old pipeline puts that last one at risk: its PostgreSQL slot keeps retaining WAL, its MySQL binlog position can expire, and a Fivetran pipeline whose slot was dropped needs a full historical re-sync. With separate tables, keeping the old pipeline running until the window closes makes rollback a repoint.

When can you retire the old pipeline?

Retire the old pipeline once the window has closed, nobody reads the old tables, and a schema change has gone through the new pipeline or the plan for one has been tested. Drop its replication slot, or the source keeps retaining log for it, and drop its publication, capture instance or supplemental logging only if the new pipeline doesn't use them. Archive or drop the old destination tables, which deleting a CDC connection usually leaves in place.

What can go wrong

Monitoring after cutover

Microsoft's migration guidance recommends close monitoring for the first 24 to 48 hours after cutover. On the new pipeline, watch freshness (how far the destination's applied position trails the source, with a heartbeat on tables that go quiet), source retention (slot lag or its equivalent, against the limit), failures and alerts, and drift, with a daily repeat of the completeness and delete checks against the source. Monitor the old pipeline as well, because a stalled one keeps its PostgreSQL slot holding WAL. For a worked example with Prometheus, see monitoring a pipeline with Estuary's observability tools.

How the pattern changes by starting point

• Debezium: Its PostgreSQL slot defaults to debezium, and with the pgoutput plugin its publication defaults to dbz_publication. Its automatic slot drop is meant for testing, so the slot comes out by hand.
• Fivetran: Its sync state can't be exported. Its destination tables carry _fivetran_synced, plus _fivetran_deleted in the default soft-delete mode, and its replication slot is one you created yourself.
• Custom CDC: Triggers or timestamp polling often have no log position to replace, and a timestamp poll never sees a hard delete.
• A new destination only: When the pipeline can load a new destination from changes it has already captured, the source isn't read again, and the migration comes down to running a second destination, validating it and repointing consumers.

Where Estuary fits

• Estuary's PostgreSQL connector backfills each keyed table in key-ordered chunks while reading the WAL. Per table, Only Changes mode skips the backfill; use it only with a verified existing baseline and a gap-free handoff.
• Each Estuary capture needs its own replication slot, so in a parallel run, give it a slot name different from the old pipeline's.
• On SQL Server, leave the connector's Automatic Change Table Cleanup off while another tool reads CDC changes from the same source tables, because it deletes change rows from every capture instance on those tables once Estuary has stored them. Rely on the cleanup job's retention instead, set long enough for the slower reader; the default is three days.
• When both the capture and the materialization run on Estuary's new runtime, the capture records which writes belong to the same source transaction, and the materialization applies each group as one destination commit. The new runtime is still rolling out, so confirm both tasks' runtime before relying on this. Don't count on grouping through a derivation, such as the _fivetran_deleted mapping below, because derivations move to the new runtime later.
• Captured changes are stored as collections, and each materialization loads them into a destination independently, so adding a destination backfills it from the collections rather than the source, as far back as the collections' storage retains. Estuary's managed bucket deletes data 20 days after collection, so a production pipeline that needs longer history should store its collections in its own bucket.
• For standard (non-delta) bindings, the Snowflake materialization merges on the collection key and soft-deletes by default, marking deleted rows with "_meta/op" = 'd', which validation queries filter out. If consumer queries filter on Fivetran's _fivetran_deleted, a derivation can map _meta/op = 'd' to _fivetran_deleted = TRUE and every other event to FALSE.
• To take over tables another tool loaded, the allow_existing_tables_for_new_bindings feature flag lets a new binding merge into an existing table, and notBefore skips collection documents Estuary published before a set time. Set notBefore early enough that no change the old pipeline missed is skipped. Changes the old pipeline already applied may then be replayed, so the precautions for taking over the old pipeline's tables, earlier in this guide, apply. notBefore is only a filter: lowering it later doesn't re-read documents it already skipped. Confirm the flag's caveats with Estuary support.
• Each materialization reports how much of its source collections it has yet to read and how far behind it is. Metrics are available through the OpenMetrics API, and task-failure alerts fire after repeated failures.
• Estuary runs as a managed service, in a private deployment, or in your own cloud through BYOC.

Shippit replaced Fivetran with Estuary for its SaaS data pipelines, and its monthly bill fell by 45%. For a move from a specific tool, the Debezium and Fivetran migration guides walk through the steps.

Conclusion

A production CDC migration avoids data gaps when the new pipeline builds its own baseline and passes validation, deletes included, before consumers move. Switching one consumer group at a time limits disruption, and rollback remains available while the old pipeline stays recoverable through a full business cycle.

Planning a CDC migration? Talk to Estuary about your current source, destination, and cutover requirements.