Change Data Capture for PostgreSQL with Estuary
In this quick tutorial, Dani demonstrates how to effortlessly set up a PostgreSQL Change Data Capture (CDC) pipeline using Estuary in less than a minute. Watch as he works with a live sales table, showing you just how simple it is to connect your Postgres database and start replicating data in real time. You'll also see how Estuary handles schema capturing and backfills your existing data, making real-time data integration both fast and efficient.
#PostgresCDC #changedatacapture
Try Estuary today for real-time data pipelines and seamless integration with your databases!
- Sign up for a free account: https://dashboard.estuary.dev/register
- Join our Slack community: https://estuary-dev.slack.com/join/shared_invite/zt-86nal6yr-VPbv~YfZE9Q~6Zl~gmZdFQ#/shared-invite/email
- Make sure to check out our Postgres CDC guide: https://estuary.dev/the-complete-change-data-capture-guide-for-postgresql/
Key things covered:
0:00 – Introduction: Setting up PostgreSQL CDC Pipeline
0:09 – Sales Table Example and Real-Time Updates
0:21 – Creating a Postgres Capture in Estuary
0:33 – Verifying the Sales Table and Schema
0:44 – Backfilling Data and Real-Time Replication
More videos
Amazon RDS to Snowflake in Real Time (Postgres CDC)
Move data from Amazon RDS (PostgreSQL) to Snowflake in real time using change data capture, with no batch jobs and no custom code. In this short demo, Estuary captures live changes from an RDS Postgres database and materializes them into Snowflake in minutes. How do you sync RDS Postgres to Snowflake in real time? Estuary captures inserts, updates, and deletes from your Postgres database, lands them in a collection, and materializes that collection into Snowflake with low latency. Insert a row in Postgres and watch it appear in Snowflake seconds later, no pipeline to build or maintain. ⏱️ Chapters 0:00 Real-time data integration with Estuary 0:11 Setting up Amazon RDS PostgreSQL as a source 0:27 Configuring the Estuary source connector 0:51 Testing and publishing the source 1:09 Creating a collection for real-time data 1:36 Materializing data to Snowflake 2:01 Testing real-time data ingestion 🔑 What you'll learn - How real-time CDC differs from batch ETL for loading Snowflake - Connecting an Amazon RDS PostgreSQL source in Estuary - Creating a collection and materializing it to Snowflake - Verifying the live sync by inserting a record in Postgres 🔗 Try Estuary free: https://dashboard.estuary.dev/register 📚 Docs: https://docs.estuary.dev 📝 Tutorial: https://estuary.dev/blog/tutorial/ 🛠️ Tools: Amazon RDS, PostgreSQL, Snowflake, Estuary #Snowflake #Postgres #CDC #ChangeDataCapture #DataEngineering #RealTimeData #AmazonRDS #DataPipeline #ETL #Estuary
PostgreSQL CDC Setup Tutorial: Real-Time Data Capture with Estuary (Step-by-Step)
Learn how to set up PostgreSQL Change Data Capture (CDC) with Estuary in under 10 minutes. This tutorial walks through enabling logical replication on Google Cloud SQL, creating the replication user, and connecting your Postgres database to Estuary for real-time streaming to Snowflake, BigQuery, Databricks, and more. ⚠️ Heads up: This video was recorded on an earlier version of Estuary. The core concepts (logical replication, replication user, Cloud SQL flags) still apply, but the UI and a few setup steps have been updated since. For the latest instructions, follow the docs: https://docs.estuary.dev/reference/Connectors/capture-connectors/PostgreSQL/ 👉 Try Estuary Free: https://dashboard.estuary.dev/register 👉 Book a Demo: https://estuary.dev/demo 👉 Full Docs: https://docs.estuary.dev/reference/Connectors/capture-connectors/PostgreSQL/ ━━━━━━━━━━━━━━━━━━━━━━━ WHAT YOU'LL LEARN ━━━━━━━━━━━━━━━━━━━━━━━ ✅ How to enable logical replication on Cloud SQL PostgreSQL ✅ How to create a dedicated replication user with correct privileges ✅ How to configure the Estuary PostgreSQL capture connector ✅ How to troubleshoot connection string and IP allowlist errors ━━━━━━━━━━━━━━━━━━━━━━━ CHAPTERS ━━━━━━━━━━━━━━━━━━━━━━━ 0:00 Intro: PostgreSQL CDC with Estuary 0:52 Enable Logical Replication on Cloud SQL PostgreSQL 2:45 Create Estuary Capture for PostgreSQL 3:00 PostgreSQL Endpoint Configuration (Host, Port, User) 3:55 Troubleshooting: No Collections Found in Source Database 5:14 Troubleshooting: PostgreSQL Connection String Errors ━━━━━━━━━━━━━━━━━━━━━━━ WHY POSTGRESQL CDC WITH ESTUARY ━━━━━━━━━━━━━━━━━━━━━━━ Estuary is a real-time data platform built for Change Data Capture. Unlike batch ETL tools, Estuary streams row-level changes from PostgreSQL's write-ahead log (WAL) to destinations like Snowflake, BigQuery, Databricks, Redshift, and ClickHouse with sub-second latency. Works with PostgreSQL on: - Google Cloud SQL - Amazon RDS / Aurora - Azure Database for PostgreSQL - Supabase, Neon, Heroku - Self-hosted PostgreSQL ━━━━━━━━━━━━━━━━━━━━━━━ CONNECT WITH ESTUARY ━━━━━━━━━━━━━━━━━━━━━━━ 🌐 Website: https://estuary.dev/ 💬 Slack Community: https://estuary-dev.slack.com/ 📚 Docs: https://docs.estuary.dev/ #PostgreSQL #CDC #ChangeDataCapture #DataEngineering #DataPipeline #RealTimeData #Estuary #CloudSQL #LogicalReplication #ETL
Introducing Estuary Agent Skills: Build Data Pipelines with AI
Estuary Agent Skills give your AI coding assistant the Estuary-specific context it needs to build and operate real-time data pipelines. Ask in plain English, capture my Postgres database into Snowflake, load these collections into BigQuery, or why is this connector failing, and your assistant knows the workflow. How do you build a data pipeline with an AI assistant? Instead of bouncing between docs, CLI commands, connector setup, and warehouse settings, Estuary Agent Skills follow the open SKILL.md pattern, so your assistant knows which flowctl commands to run, what specs to generate, and which checks to perform before publishing. You stay in control: you review the specs, approve the commands, and decide what credentials exist and what gets published. The skills are open source, so you can inspect everything. ⏱️ Chapters 0:00 The glue-work problem 0:29 Introducing Agent Skills 0:38 Pipelines in plain English 0:54 What's included 1:20 Demo: Postgres to Snowflake 1:44 You stay in control 2:34 Works with your AI tools 2:52 Get started 🔑 In the first release - Source skills: Postgres, MySQL, MongoDB, SQL Server - Materialization skills: Snowflake, BigQuery, Redshift, Databricks, Postgres - Operational skills: task health, catalog status, publication history, and more 🔗 Get the skills on GitHub: https://github.com/estuary/agent-skills 📖 Setup guide: https://docs.estuary.dev/guides/agent-skills/ 💡 Learn more: https://estuary.dev/capabilities/agent-skills/ 🚀 Start free: https://dashboard.estuary.dev/register #DataEngineering #AgentSkills #AI #Estuary #ChangeDataCapture #Snowflake #Postgres #ClaudeCode #Cursor #RealTimeData
PostgreSQL CDC: What, Why, and How
Set up PostgreSQL change data capture (CDC) without worrying about your WAL or replication slot loss. This step-by-step Postgres CDC guide covers WAL configuration, replication user setup, publication creation, and building your first real-time pipeline to Snowflake, Iceberg, or ClickHouse, no Kafka or Debezium required. We walk through why CDC beats batch replication for Postgres, how the WAL and replication slots actually work (and what goes wrong), and a full setup demo with Estuary, a real-time CDC platform that manages WAL advancement and decouples your source from your destination. LINKS - Set up your first data pipeline for free at Estuary: https://dashboard.estuary.dev/register/?utm_source=youtube&utm_medium=social&utm_campaign=postgres - Discover more PostgreSQL configuration options in Estuary’s docs: https://docs.estuary.dev/reference/Connectors/capture-connectors/PostgreSQL/ - Explore available destinations to continue your pipeline: https://docs.estuary.dev/reference/Connectors/materialization-connectors/ - Join Estuary on Slack: https://go.estuary.dev/slack - Contact us: https://estuary.dev/contact-us FAQ 1. What is Postgres CDC? CDC, or change data capture, is a method to record all updates made to a database. In conjunction with PostgreSQL, CDC often uses the Postgres write-ahead log for data replication. We start discussing Postgres CDC at 3:36 2. When should you use CDC? Is it just for real-time data? CDC can be beneficial for many different use cases since it has a low impact on the replicated Postgres database and handles replication more efficiently than batch queries. Since CDC is also very low latency, it’s a top choice for real-time use cases, though batch workflows will still benefit from an efficiency/low-impact standpoint. We go over CDC use cases at 2:29 3. Is CDC complicated to implement? CDC can be complex when creating and managing a DIY solution. You need to be careful that the WAL doesn’t grow beyond expected bounds. A managed solution can make setup and maintenance much simpler; just make sure that the integration platform follows CDC best practices, like keeping the source and destination decoupled. A brief implementation guide with Estuary starts at 5:55 Media resources used in this video are from Canva and the YouTube Studio Audio Library. CHAPTERS 0:00 Intro 0:45 On PostgreSQL 2:05 Change Data Capture 3:36 Postgres + CDC 5:55 Demo: Postgres setup 7:36 Demo: Basic Estuary setup 8:52 Demo: Additional settings 10:35 Wrap up
Real-time Data Products with Estuary - Ad Performance
In this detailed demo, Dani shows you how to build a real-time data product using Estuary. Learn how to capture and process data from a PostgreSQL database using Change Data Capture (CDC) and stream it into Snowflake for real-time ad performance calculations. This tutorial covers how to join and transform data using Estuary’s derivations and materialize the final output into Snowflake. Follow along as we work with ad clicks and impressions to perform transformations in real-time! 00:00 - Introduction: Real-Time Data Product with Postgres and Snowflake 01:30 - Setting Up Postgres CDC with Estuary 04:45 - Defining Derivations for Real-Time Transformations 09:45 - Materializing Data into Snowflake 12:03 - Real-Time Ad Performance Calculation in Snowflake 12:46 - Conclusion: Wrap-up and Further Resources
Estuary | The Right Time Data Platform
Welcome to Estuary, the Right Time Data Platform built for modern data teams. With Estuary, you can move and transform data between hundreds of systems at sub second latency or in batch, depending on your business needs. • Capture data from source systems using pre built, no code connectors. • Automatically infer schemas and manage both real time and historical events in collections. • Materialize your data to any destination with ease and flexibility. • Choose your deployment model: fully SaaS, Bring Your Own Cloud, or private deployment with enterprise level security. Start streaming an ocean of data and get going today: 🌊 https://dashboard.estuary.dev/register/?utm_source=youtube&utm_medium=social&utm_campaign=overview_video Learn more: 🌐 On our site: https://www.estuary.dev/?utm_source=youtube&utm_medium=social&utm_campaign=flow_overview 📚 In our docs: https://docs.estuary.dev/?utm_source=youtube&utm_medium=social&utm_campaign=flow_overview Connect with us: 💬 On Slack: https://go.estuary.dev/slack 🧑💻 In GitHub: https://github.com/estuary ℹ️ On LinkedIn: https://www.linkedin.com/company/estuary-tech/ #righttimedata #datapipelines #streamingdata #realtimeanalytics #CDC #dataengineering #Estuary
How to Stream Data to MotherDuck with Estuary (Step-by-Step)
Learn how to load your data into MotherDuck—cloud-based DuckDB—with Estuary. We’ll cover a little about what makes DuckDB unique before diving into a step-by-step demo. Whether you're building a real-time pipeline from NetSuite, Snowflake, BigQuery, PostgreSQL, or MongoDB to MotherDuck, Estuary lets you do it in minutes — no code required. Find more: - Register at Estuary: https://dashboard.estuary.dev/register - Sign up with MotherDuck: https://app.motherduck.com/?auth_flow=signup - Read Estuary’s docs on MotherDuck: https://docs.estuary.dev/reference/Connectors/materialization-connectors/motherduck/ - Follow along with MotherDuck’s tutorial on working with Estuary: https://motherduck.com/blog/streaming-data-to-motherduck/ Media resources used in this video are from Pexels and the YouTube Studio Audio Library. 0:00 Intro 1:01 DuckDB's features 1:45 MotherDuck 2:16 Connector demo with Estuary 2:57 Setting up AWS resources 3:46 Setting up MotherDuck 4:23 Publishing the connector 4:53 Outro #motherduck #datapipeline
PostgreSQL to Iceberg - Streaming Lakehouse Foundations
Stream Real-Time Data from Postgres to Iceberg with Change Data Capture and Estuary. In this step-by-step tutorial, we demonstrate how to set up and stream real-time data from a PostgreSQL database into Iceberg tables using change data capture (CDC) with Estuary. Learn how to capture, ingest, and materialize data using Estuary's seamless integration. This demo uses a sales database to showcase how changes in a PostgreSQL table are tracked and replicated into an Iceberg table stored in AWS S3. Check out Estuary's Iceberg integration: https://estuary.dev/destination/s3-iceberg/ Join Estuary's community Slack: https://estuary-dev.slack.com/join/shared_invite/zt-86nal6yr-VPbv~YfZE9Q~6Zl~gmZdFQ#/shared-invite/email 00:00 - Introduction: Streaming Data from Postgres to Iceberg 00:18 - Postgres Sales Database Overview 01:08 - Starting Change Data Capture (CDC) with Estuary 02:09 - Materializing Data into Apache Iceberg 04:17 - Backfilling Data into Iceberg 05:21 - Querying Iceberg Tables with Python 06:10 - Conclusion: Demo Recap
Seamless Data Integration, Unlimited Potential
Discover the simplest way to connect and move your data.Get hands-on for free, or schedule a demo to see the possibilities for your team.