MongoDB Change Data Capture (CDC): How It Works, Methods & Setup

What is MongoDB CDC?

MongoDB change data capture (CDC) is the process of capturing the inserts, updates, and deletes that happen in a MongoDB database and making those changes available to other systems. Most modern CDC does this continuously, in near real time, though some methods capture changes on a schedule. Common destinations are analytical systems like PostgreSQL, Snowflake, BigQuery, and Databricks, search systems like Elasticsearch, and event platforms like Kafka.

MongoDB is a document database. It's a strong fit for application workloads, but it isn't built for large-scale analytical queries, cross-system joins, or serving a data warehouse. CDC is how you keep those other systems current without running expensive scans against your operational database. Instead of periodically exporting whole collections, CDC moves only what changed, so a warehouse or search index can stay within seconds of the source while the primary keeps serving your application.

The rest of this guide is MongoDB-specific: how MongoDB exposes changes through the oplog and Change Streams, the methods and tools you can use, a setup checklist, the problems that show up in production, and where a managed platform like Estuary fits.

How MongoDB CDC works

Most MongoDB CDC runs on two native building blocks: the oplog and Change Streams.

1. The oplog: Every MongoDB replica set keeps an operation log, the oplog, in the local.oplog.rs collection. MongoDB's docs describe it as "a special capped collection that keeps a rolling record of all operations that modify the data stored in your databases." Secondaries replay it to stay in sync with the primary. That same log is what CDC reads.
2. Change Streams: Rather than tail the oplog directly, which is low-level and easy to get wrong, most tools use Change Streams, the supported API MongoDB built on top of the oplog. They're available in MongoDB 3.6 and later and require a replica set or sharded cluster (they don't work on a standalone server). Change Streams notify consumers only after a change has persisted to a majority of data-bearing members, which prevents a delivered change from later being rolled back during failover. Each change arrives as an event with an operationType of insert, update, replace, or delete, plus invalidate when a collection is dropped or renamed.

Put together, a MongoDB CDC pipeline works like this:

1. MongoDB writes every data change to the oplog: Inserts, updates, replaces, and deletes all land there.
2. A CDC connector subscribes to changes through a Change Stream (or, in lower-level setups, by reading the oplog directly) and translates each event into a record for the destination.
3. An initial snapshot, or backfill, handles existing data: An initial snapshot, or backfill, loads existing documents. A reliable connector establishes its Change Stream position before or while scanning the collection and captures changes that occur during the backfill. After the scan finishes, it continues streaming new changes without leaving a gap between the snapshot and ongoing CDC.
4. Resume tokens keep the stream recoverable: Each change event carries a resume token on its _id, and MongoDB also emits periodic high-watermark tokens during quiet periods. A connector stores the latest token and uses resumeAfter or startAfter when reconnecting. This restores its Change Stream position as long as the required history remains in the oplog; downstream delivery still needs appropriate deduplication or transactional guarantees.
5. Oplog retention sets the recovery window: A resume token only works if the operation it points to is still in the oplog. The oplog window is the time between its oldest and newest entries. If a connector is down longer than that window, the changes it needs have been truncated and it has to run a fresh backfill. This window is the single most important operational number in MongoDB CDC.

Two details to mention upfront, as they shape how downstream data looks:

• Updates carry only the changed fields by default (updateDescription). To get the whole document after an update, a connector requests fullDocument: "updateLookup", which is a follow-up read and can reflect a later state if the document changed again in between.
• Delete events do not include the full deleted document. Their documentKey contains the document’s _id and, for a sharded collection, may also contain its shard-key fields. To capture the document’s previous values, enable pre-images.

Minimal MongoDB Change Stream example

javascript
const collection = client.db("sales").collection("orders");

const changeStream = collection.watch([], {
  fullDocument: "updateLookup"
});

for await (const event of changeStream) {
  console.log(event.operationType, event.documentKey, event.fullDocument);
}

In a production CDC consumer, persist event._id as the resume token only after the destination has successfully processed the corresponding event.

MongoDB CDC methods compared

There's more than one way to capture changes from MongoDB. Change Streams are the primary modern mechanism and the preferred way to capture ongoing changes to collections. The others still have narrow, legitimate uses.

• Change Streams: The native, supported CDC API, built on the oplog. Captures inserts, updates, and deletes in near real time, resumes cleanly with resume tokens, and is the right default for production. Requires a replica set or sharded cluster and enough oplog retention.
• Oplog tailing: Reading local.oplog.rs directly. It captures the same changes at low latency, but you take on resume handling, error handling, and format parsing yourself. Change Streams exist specifically to replace this, so tailing is now a lower-level, custom-build choice.
• Timestamp-based polling: Repeatedly querying for documents whose updated_at (or similar) field is newer than the last run. Simple to set up, but it can't see deletes, can miss intermediate updates between polls, and its latency is bounded by the poll interval. Fine for simple incremental copies, not for a faithful replica.
• Snapshot / batch sync: Periodically copying a whole collection and comparing it to the previous copy. It works for small or low-frequency data, but it's expensive on large collections and doesn't reliably capture deletes.
• Application-level change capture: Having the application publish an event whenever it writes. You get full control over the event shape, but anything that changes the database outside that application path (a migration, a manual fix, another service) is invisible.

The table below summarizes the trade-offs:

MongoDB CDC setup checklist

Before you connect a CDC tool to MongoDB, work through these. Managed services and tools handle some of them for you, so verify rather than assume.

1. Confirm your deployment supports Change Streams. They require a replica set or sharded cluster; a standalone mongod won't work, so convert it to a single-node replica set at minimum. On a sharded cluster the change stream is opened against mongos (the query router) and merges the per-shard streams.
2. Confirm permissions. The capture user needs read access to the databases and collections you're capturing, plus the changeStream privilege. A common minimal role grants find and changeStream on the target resources.
3. Ensure network access. Allowlist your CDC tool's IP addresses, or connect over an SSH tunnel or private networking if the database isn't publicly reachable.
4. Plan the initial snapshot (backfill). Decide how existing data is loaded before streaming starts, and how large those collections are, since the backfill is often the slowest part of a first sync.
5. Check the oplog retention window. Make sure it's long enough that a connector can be down for maintenance without losing its place. A retention window of at least 24 hours is a reasonable floor, and more is safer for large or bursty workloads. On Atlas the oplog is managed with a configurable minimum retention; self-hosted, you size it yourself.
6. Decide how to handle deletes. Change Streams report a delete with only the document _id. Decide whether the destination should hard-delete the row or mark it deleted, and whether you need pre-images to see the deleted document's contents.
7. Decide how to handle schema changes and nested documents. MongoDB documents are flexible and can nest deeply. Decide how new or changed fields and nested structures map into the destination.
8. Choose the destination and write mode. Pick the target system and whether it applies changes as a running upsert (a live mirror) or keeps an append-only history of every change.
9. Test pause and resume. Stop the pipeline, let some changes accumulate, and Confirm that no changes are missing after recovery. If delivery is at-least-once, verify that destination upserts keyed by _id safely handle reprocessed events. This is the behavior that matters most in production, so test it before you rely on it.

For broader guidance on coordinating snapshots, recovery, schema evolution, and delivery guarantees, review these CDC implementation best practices.

Common MongoDB CDC challenges

CDC on MongoDB is well-trodden, but a handful of issues account for most production problems:

• Oplog retention and rollover: If a connector falls behind or is paused longer than the oplog window, the operations it needs are gone and it has to re-run a full backfill. Size the oplog for your worst-case downtime, and monitor how close consumers run to the edge of the window.
• Deletes: A delete event carries only the _id. Downstream, that's usually enough to remove or mark the row, but if you need the values that the document held, you must enable pre-images. Note that timestamp-polling approaches can't see deletes at all.
• Schema changes and nested documents: MongoDB doesn't enforce a schema, so fields can appear, disappear, or change type over time, and documents can nest arrays and sub-documents. A relational destination needs a plan for flattening nested structures and for handling fields that don't exist on every document.
• Large documents: MongoDB documents can be up to 16 MB. Large documents cost more to decode and move, and if you enable pre- and post-images, each change stores extra copies, which adds storage and processing.
• Sharded clusters: Change streams on a sharded cluster merge events from every shard through mongos, and each shard has its own oplog window, so size retention per shard. On MongoDB 5.3 and later, change streams no longer emit events for orphaned documents during chunk migration, so plan for the added coordination.
• Destination consistency: Most CDC delivery is at-least-once, which means a destination can occasionally see a change more than once. If exactly-once isn't guaranteed end-to-end, the destination needs idempotent writes (an upsert keyed on _id) so replays don't corrupt the data.

MongoDB CDC tools comparison

Several tools can move MongoDB changes to a destination. They differ in whether they're managed or self-run, how much infrastructure they need, and how they deliver data. Several CDC tools and platforms support MongoDB, but their deployment models, capture methods, recovery behavior, and delivery latency differ.

• Estuary: A managed real-time platform. Its MongoDB connector captures through Change Streams, with batch modes for collections that can't use them, and materializes to many destinations. Low setup, and no streaming infrastructure to run (covered in detail below).
• Debezium: The widely used open-source CDC engine. Its MongoDB connector reads Change Streams and publishes to Kafka, so you run Kafka and Kafka Connect. Flexible and free, but with real operational overhead.
• MongoDB Kafka Connector: MongoDB's own Kafka Connect source connector, also built on Change Streams. A good fit if you're already committed to Kafka.
• Airbyte: Open-source and managed options. Its MongoDB source uses Change Streams for CDC, then syncs to destinations, often in batches.
• Fivetran: A managed connector that captures MongoDB changes with log-based (change-stream) reads. This delivers to the destination in batches, which adds latency even though capture is change-based.
• Striim: An enterprise real-time integration platform with MongoDB CDC support. Capable, but with a heavier setup and licensing model.
• AWS DMS: Supports MongoDB as a source, reading Change Streams by default (with an option to read the oplog directly). It offers document and table migration modes and needs a replica set and adequate oplog retention. AWS-centric, and geared more to database migration than to analytics destinations.
• Google Datastream: Now generally available for MongoDB sources, using Change Streams to replicate inserts, updates, and deletes into BigQuery and Cloud Storage.

A short comparison:

If you are comparing managed and self-hosted options, use this framework to evaluate CDC solutions across capture method, backfills, delivery guarantees, operational burden, and cost.

How Estuary supports MongoDB CDC

Estuary is a real-time data integration platform with 200+ connectors. Its MongoDB capture connector captures changes through MongoDB Change Streams and delivers them to warehouses, search systems, and other destinations in real time.

What the MongoDB connector does:

• Change Streams by default, with batch modes for other cases: The connector runs each collection in one of three modes. Change Stream Incremental is the preferred mode and the only one that captures deletes. Batch Snapshot re-scans a collection on a schedule (useful for deployments that can't use change streams), and Batch Incremental scans forward on a strictly increasing field for append-only collections.
• Live backfill: During the initial snapshot, the connector reads the change stream while it scans existing documents, so it captures changes that happen during the backfill instead of losing them. After the snapshot it continues on the change stream indefinitely.
• Broad MongoDB coverage: Self-hosted MongoDB and MongoDB Atlas, plus Amazon DocumentDB and Azure Cosmos DB, which have their own connector variants.
• Secure connectivity: SSH tunneling and IP allowlisting for databases that aren't publicly reachable.
• Delete capture and document images: Change Stream Incremental mode captures deletes. For updates, Estuary requests a full post-image using fullDocument: "updateLookup". It also captures available pre-images for update, replace, and delete events when changeStreamPreAndPostImages is enabled on the source collection.

Steady-state Change Stream capture is typically lightweight because it reads the oplog rather than repeatedly scanning collections. Initial backfills and fullDocument: "updateLookup" add source reads, so monitor source load during backfills and high-update workloads.

Try it: for a worked example, the Real-Time CDC with MongoDB tutorial walks through capturing from a MongoDB Atlas collection with the connector. In short: create the capture by pointing Estuary at your MongoDB connection string and selecting collections, then create a materialization to your destination (PostgreSQL, Snowflake, BigQuery, and others). Changes flow continuously once both are published.

MongoDB CDC performance considerations

A few factors determine how a MongoDB CDC pipeline performs:

• Document size and shape. Larger documents and deeply nested structures cost more to decode (from BSON) and move.
• Change frequency. High write rates mean more oplog activity and more events to process.
• Connector throughput. How fast the tool decodes BSON and batches writes to the destination.
• Network latency between the source, the connector, and the destination.
• Destination write capacity. The target has to keep up with the incoming change rate.
• Backfill volume. The initial snapshot of a large collection is usually the slowest phase.

As one data point, Estuary's engineering team documented a MongoDB connector optimization that raised throughput on standard 20 KB documents from 34 MB/s to 57 MB/s, roughly 200 GB per hour, through faster BSON decoding and improved batch prefetching. The methodology is in the MongoDB connector optimization deep-dive.