Postgres to ClickHouse with CDC: Estuary 101 Webinar and Live Demo
How do you stream Postgres data into ClickHouse with CDC? In this Estuary 101 webinar, Zulf Qureshi (Head of Sales Engineering, Estuary) explains right-time data pipelines and shows a live end-to-end demo: a Postgres CDC capture, collections in cloud storage, and a ClickHouse materialization, with a new row arriving in real time.
What you’ll learn:
1. What “right-time” data means: capture changes in real time, then load in real time or on a schedule to control cost
2. How Estuary’s architecture works: captures, collections and materializations
3. Why collections matter: no single point of failure, replay without re-reading the source, and capture once, load to many destinations
4. Deployment options: public, private and bring your own cloud (BYOC)
5. Schema evolution, backfills, soft deletes, hard deletes and delta updates
6. Monitoring with alerts, logs and lineage
Try Estuary free: https://dashboard.estuary.dev/register/
Chapters:
0:00 Introduction and agenda
1:07 The data integration problem
2:35 Why Estuary: one platform for batch, CDC and streaming
4:50 Right-time data: real-time capture, scheduled loads
7:16 Estuary architecture: captures, collections, materializations
10:11 Why collections matter: no single point of failure
12:25 Replay without re-reading the source
14:42 Capture once, load to many destinations
16:33 Deployment options: public, private, BYOC
20:20 200+ connectors
21:34 Live demo: Postgres CDC capture
34:37 Collections and data lineage
36:05 ClickHouse materialization: soft deletes, hard deletes, delta updates
41:01 Live insert from Postgres to ClickHouse
42:33 Contact and resources
43:49 Q&A: Can you batch CDC data?
#ClickHouse #Postgres #CDC #DataEngineering #Estuary
More videos
How CDC Works in Estuary: Source to Destination Explained
Change data capture (CDC) reads every insert, update and delete from your database's log and streams it to your destinations. Here's how Estuary does it, from source to destination. We follow one order placed by Will through an Estuary pipeline: from the moment it's written to a Postgres database, through captures, collections and materializations, to the moment it lands in a warehouse, search index or AI model. You'll learn: - How log-based CDC reads the write-ahead log (WAL) as a second reader, with no extra load on your production database - How the initial snapshot and live WAL streaming run in parallel, so existing rows and new changes both arrive - Why polling-based methods miss deletes, and why log-based CDC doesn't - How schema validation works, and how Estuary adapts to schema changes and pauses only for breaking ones - What collections are: a data lake and a real-time stream at the same time - How collection keys keep a full history of every change - How derivations create new collections with stateless or stateful transformations - How materializations deliver data to each destination at its own speed, from real-time to scheduled batches - How read, reduce, write cycles and reduction strategies work, with last-write-wins as the default - How exactly-once delivery keeps data correct, even if something crashes mid-write This is Episode 1 of our series on how Estuary works under the hood. Chapters 0:00 What happens to data between source and destination? 1:00 What is a capture? 1:49 How log-based CDC reads the write-ahead log (WAL) 2:12 Initial backfill: snapshot and WAL in parallel 2:44 Why polling misses deletes 3:09 JSON documents and schema validation 3:31 How Estuary handles schema changes 4:07 What is a collection? 4:52 Collection keys and change history 5:25 Derivations: stateless and stateful transformations 6:23 What is a materialization? 7:23 Choosing sync speed per destination 7:43 Automatic batching to protect destinations 8:06 Read, reduce, write and reduction strategies 8:36 Exactly-once delivery 9:00 Key takeaways Links Try Estuary free: https://dashboard.estuary.dev/register Estuary docs: https://docs.estuary.dev Schema evolution guide: https://docs.estuary.dev/guides/schema-evolution/ #ChangeDataCapture #CDC #DataEngineering
SQL Server CDC to Snowflake + ClickHouse in Real Time (Fan-Out Demo)
See how to capture SQL Server change data once and fan it out to Snowflake, ClickHouse, and other destinations without creating duplicate source pipelines. In this DBTA roundtable webinar, Estuary demonstrates a capture-once, fan-out architecture using change data capture (CDC). The demo starts with an existing SQL Server to Snowflake pipeline for scheduled BI workloads, then adds ClickHouse as a real-time destination using the same captured collections—without re-reading the SQL Server source. You’ll see how Estuary handles durable collections, batch vs. real-time sync schedules, backfills, schema mapping, merge vs. delta updates, and exactly-once delivery. ⏱️ Chapters 0:00 Introduction 2:08 Capture once, fan out architecture 2:53 Why multiple destinations add complexity 3:44 How the fan-out pattern works 6:40 SQL Server to Snowflake demo 8:37 Sources, collections & destinations 10:11 Inspecting CDC records 13:59 Snowflake pipeline 15:39 Adding ClickHouse for real-time workloads 18:16 ClickHouse ingestion options 18:53 Configuring the ClickHouse connector 20:38 Real-time vs. batch sync 25:14 Publishing & initial backfill 28:59 Real-time data in ClickHouse 30:27 Fan-out architecture complete 31:09 More destinations & migration use cases 32:29 Q&A: transforms, latency & deduplication 🔑 What you'll learn * How to capture CDC data once and serve multiple destinations * How Snowflake and ClickHouse can consume the same data at different frequencies * How to add destinations without re-reading the source database * How Estuary handles backfills, CDC updates, and exactly-once delivery 🔗 Try Estuary free: https://dashboard.estuary.dev/register 📚 Docs: https://docs.estuary.dev #ChangeDataCapture #ClickHouse #Snowflake #DataEngineering #RealTimeData
PostgreSQL CDC: What, Why, and How
Set up PostgreSQL change data capture (CDC) without worrying about your WAL or replication slot loss. This step-by-step Postgres CDC guide covers WAL configuration, replication user setup, publication creation, and building your first real-time pipeline to Snowflake, Iceberg, or ClickHouse, no Kafka or Debezium required. We walk through why CDC beats batch replication for Postgres, how the WAL and replication slots actually work (and what goes wrong), and a full setup demo with Estuary, a real-time CDC platform that manages WAL advancement and decouples your source from your destination. LINKS - Set up your first data pipeline for free at Estuary: https://dashboard.estuary.dev/register/?utm_source=youtube&utm_medium=social&utm_campaign=postgres - Discover more PostgreSQL configuration options in Estuary’s docs: https://docs.estuary.dev/reference/Connectors/capture-connectors/PostgreSQL/ - Explore available destinations to continue your pipeline: https://docs.estuary.dev/reference/Connectors/materialization-connectors/ - Join Estuary on Slack: https://go.estuary.dev/slack - Contact us: https://estuary.dev/contact-us FAQ 1. What is Postgres CDC? CDC, or change data capture, is a method to record all updates made to a database. In conjunction with PostgreSQL, CDC often uses the Postgres write-ahead log for data replication. We start discussing Postgres CDC at 3:36 2. When should you use CDC? Is it just for real-time data? CDC can be beneficial for many different use cases since it has a low impact on the replicated Postgres database and handles replication more efficiently than batch queries. Since CDC is also very low latency, it’s a top choice for real-time use cases, though batch workflows will still benefit from an efficiency/low-impact standpoint. We go over CDC use cases at 2:29 3. Is CDC complicated to implement? CDC can be complex when creating and managing a DIY solution. You need to be careful that the WAL doesn’t grow beyond expected bounds. A managed solution can make setup and maintenance much simpler; just make sure that the integration platform follows CDC best practices, like keeping the source and destination decoupled. A brief implementation guide with Estuary starts at 5:55 Media resources used in this video are from Canva and the YouTube Studio Audio Library. CHAPTERS 0:00 Intro 0:45 On PostgreSQL 2:05 Change Data Capture 3:36 Postgres + CDC 5:55 Demo: Postgres setup 7:36 Demo: Basic Estuary setup 8:52 Demo: Additional settings 10:35 Wrap up
Amazon RDS to Snowflake in Real Time (Postgres CDC)
Move data from Amazon RDS (PostgreSQL) to Snowflake in real time using change data capture, with no batch jobs and no custom code. In this short demo, Estuary captures live changes from an RDS Postgres database and materializes them into Snowflake in minutes. How do you sync RDS Postgres to Snowflake in real time? Estuary captures inserts, updates, and deletes from your Postgres database, lands them in a collection, and materializes that collection into Snowflake with low latency. Insert a row in Postgres and watch it appear in Snowflake seconds later, no pipeline to build or maintain. ⏱️ Chapters 0:00 Real-time data integration with Estuary 0:11 Setting up Amazon RDS PostgreSQL as a source 0:27 Configuring the Estuary source connector 0:51 Testing and publishing the source 1:09 Creating a collection for real-time data 1:36 Materializing data to Snowflake 2:01 Testing real-time data ingestion 🔑 What you'll learn - How real-time CDC differs from batch ETL for loading Snowflake - Connecting an Amazon RDS PostgreSQL source in Estuary - Creating a collection and materializing it to Snowflake - Verifying the live sync by inserting a record in Postgres 🔗 Try Estuary free: https://dashboard.estuary.dev/register 📚 Docs: https://docs.estuary.dev 📝 Tutorial: https://estuary.dev/blog/tutorial/ 🛠️ Tools: Amazon RDS, PostgreSQL, Snowflake, Estuary #Snowflake #Postgres #CDC #ChangeDataCapture #DataEngineering #RealTimeData #AmazonRDS #DataPipeline #ETL #Estuary
SQL Server CDC to ClickHouse: Real-Time Replication, Step by Step
Learn how to stream SQL Server data to ClickHouse in real time using Change Data Capture (CDC) with Estuary. This step-by-step tutorial shows how to replicate SQL Server inserts, updates, and deletes into ClickHouse for real-time analytics, without running analytical queries on your production SQL Server tables. How does SQL Server to ClickHouse CDC work? SQL Server's native CDC feature captures database changes from its transaction log and stores them in CDC change tables. Estuary reads these changes, streams them into collections, and continuously materializes the data into ClickHouse. This lets you keep ClickHouse updated with SQL Server data without building and managing a separate Kafka-based pipeline. What you'll learn: - How SQL Server Change Data Capture (CDC) works - How to enable CDC on SQL Server databases and tables - How to configure the SQL Server CDC connector in Estuary - How to stream SQL Server changes into ClickHouse - How ClickHouse handles CDC updates, deletes, and row versions - How to query the latest data using ClickHouse's `FINAL` keyword - How to verify real-time replication with live inserts, updates, and deletes Step-by-step chapters: 00:00 SQL Server to ClickHouse: Real-time analytics overview 00:10 How SQL Server Change Data Capture works 00:35 Enable CDC on SQL Server 01:00 Configure the SQL Server CDC connector in Estuary 01:15 Public, private, and BYOC data planes 01:50 Connection details and network configuration 02:27 Discover SQL Server tables and publish the capture 03:05 Preview captured CDC data in Estuary 03:36 How ClickHouse handles updates and deletes 04:06 Configure ClickHouse materialization in Estuary 05:26 Live demo: SQL Server inserts, updates, and deletes in ClickHouse 06:08 Recap Documentation and resources: SQL Server CDC connector documentation: https://docs.estuary.dev/reference/Connectors/capture-connectors/SQLServer/ ClickHouse materialization documentation: https://docs.estuary.dev/reference/Connectors/materialization-connectors/ClickHouse/ How to enable CDC in SQL Server: https://estuary.dev/blog/enable-sql-server-change-data-capture/ Try Estuary: https://dashboard.estuary.dev/register Estuary helps teams build real-time data pipelines using CDC, streaming, and batch data integration. #SQLServer #ClickHouse #ChangeDataCapture
Real-time Change Data Capture for Postgres with Estuary
Learn how to set up PostgreSQL Change Data Capture (CDC) for real-time data replication using Estuary. This step-by-step tutorial explains how Postgres CDC works with the write-ahead log (WAL), logical replication, publications, and replication slots, then walks through creating a PostgreSQL CDC capture in Estuary. How does PostgreSQL CDC work? PostgreSQL records database changes in its write-ahead log (WAL). Log-based CDC uses logical replication to capture committed inserts, updates, and deletes without repeatedly polling the source tables. A publication defines which tables are included, while a replication slot tracks the consumer's progress through the WAL. Estuary can backfill existing PostgreSQL data and then continuously capture changes into reusable collections that can feed destinations such as Snowflake, ClickHouse, Databricks, and more. What you'll learn: - How PostgreSQL CDC works using WAL and logical replication - How publications and replication slots work - PostgreSQL CDC vs. batch data extraction - How to configure a PostgreSQL CDC connector in Estuary - Supported Postgres platforms, including Amazon RDS, Aurora, Cloud SQL, Azure, AlloyDB, Neon, and Supabase - Public, private, and BYOC deployment options - PostgreSQL authentication and advanced connector settings - How to connect to private PostgreSQL databases - How to discover tables, configure collections, and publish a CDC capture Video chapters: 00:00 How log-based PostgreSQL CDC works 00:30 PostgreSQL CDC vs. batch connectors 01:08 Data planes and deployment options 01:42 PostgreSQL connection settings and history mode 02:09 Authentication, publications, replication slots, and watermarks 02:36 Connecting to private PostgreSQL databases 03:06 Discover tables, publish capture, and start streaming Documentation and resources: PostgreSQL connector docs: https://docs.estuary.dev/reference/Connectors/capture-connectors/PostgreSQL/ PostgreSQL batch connector: https://docs.estuary.dev/reference/Connectors/capture-connectors/PostgreSQL/postgres-batch/ Deployment options: https://docs.estuary.dev/getting-started/deployment-options/ Connecting to private networks (SSH tunneling): https://docs.estuary.dev/guides/connect-network/ PrivateLink: https://docs.estuary.dev/private-byoc/privatelink/ Try Estuary free: https://dashboard.estuary.dev/register Estuary is a real-time data integration platform that helps teams capture database changes using CDC and deliver data to multiple downstream systems. #PostgreSQL #ChangeDataCapture #PostgresCDC
Estuary | The Right Time Data Platform
Welcome to Estuary, the Right Time Data Platform built for modern data teams. With Estuary, you can move and transform data between hundreds of systems at sub second latency or in batch, depending on your business needs. • Capture data from source systems using pre built, no code connectors. • Automatically infer schemas and manage both real time and historical events in collections. • Materialize your data to any destination with ease and flexibility. • Choose your deployment model: fully SaaS, Bring Your Own Cloud, or private deployment with enterprise level security. Start streaming an ocean of data and get going today: 🌊 https://dashboard.estuary.dev/register/?utm_source=youtube&utm_medium=social&utm_campaign=overview_video Learn more: 🌐 On our site: https://www.estuary.dev/?utm_source=youtube&utm_medium=social&utm_campaign=flow_overview 📚 In our docs: https://docs.estuary.dev/?utm_source=youtube&utm_medium=social&utm_campaign=flow_overview Connect with us: 💬 On Slack: https://go.estuary.dev/slack 🧑💻 In GitHub: https://github.com/estuary ℹ️ On LinkedIn: https://www.linkedin.com/company/estuary-tech/ #righttimedata #datapipelines #streamingdata #realtimeanalytics #CDC #dataengineering #Estuary
Introducing Estuary Agent Skills: Build Data Pipelines with AI
Estuary Agent Skills give your AI coding assistant the Estuary-specific context it needs to build and operate real-time data pipelines. Ask in plain English, capture my Postgres database into Snowflake, load these collections into BigQuery, or why is this connector failing, and your assistant knows the workflow. How do you build a data pipeline with an AI assistant? Instead of bouncing between docs, CLI commands, connector setup, and warehouse settings, Estuary Agent Skills follow the open SKILL.md pattern, so your assistant knows which flowctl commands to run, what specs to generate, and which checks to perform before publishing. You stay in control: you review the specs, approve the commands, and decide what credentials exist and what gets published. The skills are open source, so you can inspect everything. ⏱️ Chapters 0:00 The glue-work problem 0:29 Introducing Agent Skills 0:38 Pipelines in plain English 0:54 What's included 1:20 Demo: Postgres to Snowflake 1:44 You stay in control 2:34 Works with your AI tools 2:52 Get started 🔑 In the first release - Source skills: Postgres, MySQL, MongoDB, SQL Server - Materialization skills: Snowflake, BigQuery, Redshift, Databricks, Postgres - Operational skills: task health, catalog status, publication history, and more 🔗 Get the skills on GitHub: https://github.com/estuary/agent-skills 📖 Setup guide: https://docs.estuary.dev/guides/agent-skills/ 💡 Learn more: https://estuary.dev/capabilities/agent-skills/ 🚀 Start free: https://dashboard.estuary.dev/register #DataEngineering #AgentSkills #AI #Estuary #ChangeDataCapture #Snowflake #Postgres #ClaudeCode #Cursor #RealTimeData
How to Stream Data into MotherDuck (DuckDB) with Estuary: Step by Step
How do you load data into MotherDuck? Estuary’s MotherDuck connector streams data into cloud-based DuckDB, with no code required. This video covers what makes DuckDB unique, what MotherDuck adds, then a step-by-step setup demo. Whether you’re building a real-time pipeline from PostgreSQL, MongoDB, NetSuite, Snowflake, or BigQuery to MotherDuck, Estuary lets you set it up in minutes. What you’ll learn: - Why DuckDB is fast: in-process, columnar-vectorized query execution - What MotherDuck adds: serverless DuckDB in the cloud, team collaboration, and secure secret storage - Moving data from OLTP to OLAP without slowing down production - Setting up the MotherDuck connector: S3 staging bucket, AWS access keys, MotherDuck access token, and soft or hard deletes Find more: - Register at Estuary: https://dashboard.estuary.dev/register - Sign up with MotherDuck: https://app.motherduck.com/?auth_flow=signup - Estuary’s MotherDuck connector docs: https://docs.estuary.dev/reference/Connectors/materialization-connectors/motherduck/ - MotherDuck’s tutorial on streaming data with Estuary: https://motherduck.com/blog/streaming-data-to-motherduck/ 0:00 Intro: why DuckDB? 1:01 DuckDB’s features 1:45 What is MotherDuck? 2:16 MotherDuck connector demo with Estuary 2:57 Set up AWS resources (S3 bucket and access keys) 3:46 Set up MotherDuck (access token) 4:23 Publish the connector 4:53 Outro Media resources used in this video are from Pexels and the YouTube Studio Audio Library. #MotherDuck #DuckDB #DataPipeline #DataEngineering #Estuary
Change Data Capture for PostgreSQL with Estuary
In this quick tutorial, Dani demonstrates how to effortlessly set up a PostgreSQL Change Data Capture (CDC) pipeline using Estuary in less than a minute. Watch as he works with a live sales table, showing you just how simple it is to connect your Postgres database and start replicating data in real time. You'll also see how Estuary handles schema capturing and backfills your existing data, making real-time data integration both fast and efficient. #PostgresCDC #changedatacapture Try Estuary today for real-time data pipelines and seamless integration with your databases! - Sign up for a free account: https://dashboard.estuary.dev/register - Join our Slack community: https://estuary-dev.slack.com/join/shared_invite/zt-86nal6yr-VPbv~YfZE9Q~6Zl~gmZdFQ#/shared-invite/email - Make sure to check out our Postgres CDC guide: https://estuary.dev/the-complete-change-data-capture-guide-for-postgresql/ Key things covered: 0:00 – Introduction: Setting up PostgreSQL CDC Pipeline 0:09 – Sales Table Example and Real-Time Updates 0:21 – Creating a Postgres Capture in Estuary 0:33 – Verifying the Sales Table and Schema 0:44 – Backfilling Data and Real-Time Replication
How to Replicate Oracle to Snowflake Using CDC (Step-by-Step)
Learn how to replicate Oracle to Snowflake in real time using Change Data Capture (CDC) with Estuary. This step-by-step tutorial shows how to capture Oracle inserts, updates, and deletes using LogMiner and stream them into Snowflake without building a custom data replication pipeline. How does Oracle to Snowflake CDC work? Oracle records database changes in redo logs. Estuary uses Oracle LogMiner to read those changes and tracks its progress using Oracle System Change Numbers (SCNs). Estuary first backfills existing Oracle data into collections, then continuously captures new inserts, updates, and deletes. These changes are delivered to Snowflake using Estuary's Snowflake materialization connector. In this demo, we enable delta updates and configure Snowpipe Streaming to preserve individual change events in Snowflake. What you'll learn: - How Oracle CDC works using redo logs, LogMiner, and SCNs - How to configure Oracle supplemental logging and CDC prerequisites - How to create an Oracle CDC capture in Estuary - How to discover Oracle tables and perform an initial backfill - How to connect Oracle collections to Snowflake - How standard updates differ from delta updates - How to configure Snowpipe Streaming for low-latency data ingestion - How to verify Oracle inserts, updates, and deletes in Snowflake Video chapters: 00:00 How Oracle CDC works: LogMiner, redo logs, and SCNs 00:30 Oracle CDC prerequisites 00:55 Configure Oracle CDC capture in Estuary 01:24 Public, private, and BYOC deployment options 01:58 Oracle connection details and advanced settings 02:41 Discover Oracle tables and collections 03:09 Publish capture and backfill existing Oracle data 03:23 Preview Oracle CDC events in Estuary 03:59 Configure Snowflake materialization 05:21 Standard vs. delta updates and Snowpipe Streaming 05:55 Map Oracle collections to Snowflake tables 06:44 Live demo: Oracle inserts, updates, and deletes 07:50 Recap Documentation and resources: Oracle CDC connector documentation: https://docs.estuary.dev/reference/Connectors/capture-connectors/OracleDB/ Snowflake materialization documentation: https://docs.estuary.dev/reference/Connectors/materialization-connectors/Snowflake/ Oracle to Snowflake migration guide: https://estuary.dev/blog/oracle-to-snowflake/ Try Estuary: https://dashboard.estuary.dev/register Estuary is a real-time data integration platform supporting change data capture, streaming, and batch pipelines. #OracleToSnowflake #OracleCDC #Snowflake
![[WEBINAR] Estuary Agent Skills: From Setup to Success thumbnail](/static/0bfcc292be14e15168775e4eda941993/71d4d/maxresdefault.webp)
[WEBINAR] Estuary Agent Skills: From Setup to Success
In this webinar, Dani Palma (Developer Relations, Estuary) walks through the new Estuary Agent Skills product launch, from installation to building and debugging an Estuary pipeline. Agent Skills lets your AI assistant create and monitor production-grade Estuary pipelines in the tool your team already uses. You describe the pipeline using plain language prompts. The agent builds it. You review, approve, and run it. Agent Skills are compatible with Claude Code, Cursor, Codex, and 45+ other agents that support the open SKILL.md standard. We currently offer 37 skills that cover captures, materializations, pipeline operations, derivations, and schema management. Interesting in trying out Agent Skills? Here's how to get started: Agent Skills docs: https://docs.estuary.dev/guides/agent-skills/ Agent Skills overview: https://www.youtube.com/watch?v=lc71ILyd89g Agent Skills GitHub repo: https://github.com/estuary/agent-skills
Dekaf: How to Use Kafka Minus the Kafka
Have you ever wanted Kafka’s real-time pub/sub benefits without implementing and maintaining a whole Kafka ecosystem yourself? Learn how with Dekaf. We’ll cover some Kafka basics to help explain how Estuary’s Kafka API compatibility layer fits seamlessly into a modern data architecture. - Register for a free Estuary account: https://dashboard.estuary.dev/register/?utm_source=youtube&utm_medium=social&utm_campaign=dekaf_video - Learn more about Dekaf: https://docs.estuary.dev/reference/Connectors/dekaf/ - Find the example kcat command: https://docs.estuary.dev/guides/dekaf_reading_collections_from_kafka/#2-set-up-your-kafka-client - Join us on Slack: https://go.estuary.dev/slack Media resources used in this video are from Pexels, Canva, and the YouTube Studio Audio Library. 0:00 Introduction 0:30 What is Kafka? 1:04 The Kafka Ecosystem 2:39 Integrating with Kafka Consumers... 3:26 ...using Dekaf 4:12 Dekaf Setup 6:16 kcat Test 6:45 Final Thoughts
How to Connect NetSuite to Estuary for Real-Time Data Integration
Learn how to connect NetSuite to Estuary and stream your ERP data in real-time to destinations like Snowflake, BigQuery, Databricks, Redshift, and more — no custom ETL code required. This tutorial walks through the full NetSuite setup (SuiteAnalytics, integration, role, user, and token-based authentication) and the Estuary capture configuration end-to-end. 👉 Try Estuary Free: https://dashboard.estuary.dev/register 👉 Book a Demo: https://estuary.dev/demo ━━━━━━━━━━━━━━━━━━━━━━━ WHAT YOU'LL LEARN ━━━━━━━━━━━━━━━━━━━━━━━ ✅ How to enable SuiteAnalytics in NetSuite ✅ How to create a NetSuite integration with token-based authentication ✅ How to set up a role and assign permissions for Estuary access ✅ How to generate NetSuite access tokens (consumer key, secret, token ID, token secret) ✅ How to configure the Estuary NetSuite capture connector ✅ How to find your NetSuite account ID and internal role ID ━━━━━━━━━━━━━━━━━━━━━━━ CHAPTERS ━━━━━━━━━━━━━━━━━━━━━━━ 0:00 Intro: NetSuite to Estuary Integration 0:18 NetSuite Setup: SuiteAnalytics, Account ID, Integration 2:08 Create Role and Assign User Permissions 2:37 Generate NetSuite Access Tokens 3:25 Enable Internal IDs in NetSuite 4:02 Create the NetSuite Capture in Estuary 5:16 Endpoint Configuration and Authentication 5:43 Test, Save, and Publish the Capture ━━━━━━━━━━━━━━━━━━━━━━━ WHY NETSUITE WITH ESTUARY ━━━━━━━━━━━━━━━━━━━━━━━ NetSuite is one of the most widely used ERPs for finance, operations, and inventory data, but getting that data into a modern warehouse is notoriously painful. Estuary is a real-time data platform that streams NetSuite data into analytics destinations with sub-second latency — without requiring custom SuiteScript, scheduled CSV exports, or homegrown ETL pipelines. Common NetSuite pipelines built on Estuary: - NetSuite to Snowflake - NetSuite to BigQuery - NetSuite to Databricks - NetSuite to Redshift - NetSuite to PostgreSQL ━━━━━━━━━━━━━━━━━━━━━━━ CONNECT WITH ESTUARY ━━━━━━━━━━━━━━━━━━━━━━━ 🌐 Website: https://estuary.dev/ 💬 Slack Community: https://estuary-dev.slack.com/ 📚 Docs: https://docs.estuary.dev #NetSuite #DataIntegration #ERP #DataPipeline #Estuary #SuiteAnalytics #NetSuiteToSnowflake #DataEngineering #RealTimeData #ETL
Real-time Data Products with Estuary - Ad Performance
In this detailed demo, Dani shows you how to build a real-time data product using Estuary. Learn how to capture and process data from a PostgreSQL database using Change Data Capture (CDC) and stream it into Snowflake for real-time ad performance calculations. This tutorial covers how to join and transform data using Estuary’s derivations and materialize the final output into Snowflake. Follow along as we work with ad clicks and impressions to perform transformations in real-time! 00:00 - Introduction: Real-Time Data Product with Postgres and Snowflake 01:30 - Setting Up Postgres CDC with Estuary 04:45 - Defining Derivations for Real-Time Transformations 09:45 - Materializing Data into Snowflake 12:03 - Real-Time Ad Performance Calculation in Snowflake 12:46 - Conclusion: Wrap-up and Further Resources
PostgreSQL to Iceberg - Streaming Lakehouse Foundations
Stream Real-Time Data from Postgres to Iceberg with Change Data Capture and Estuary. In this step-by-step tutorial, we demonstrate how to set up and stream real-time data from a PostgreSQL database into Iceberg tables using change data capture (CDC) with Estuary. Learn how to capture, ingest, and materialize data using Estuary's seamless integration. This demo uses a sales database to showcase how changes in a PostgreSQL table are tracked and replicated into an Iceberg table stored in AWS S3. Check out Estuary's Iceberg integration: https://estuary.dev/destination/s3-iceberg/ Join Estuary's community Slack: https://estuary-dev.slack.com/join/shared_invite/zt-86nal6yr-VPbv~YfZE9Q~6Zl~gmZdFQ#/shared-invite/email 00:00 - Introduction: Streaming Data from Postgres to Iceberg 00:18 - Postgres Sales Database Overview 01:08 - Starting Change Data Capture (CDC) with Estuary 02:09 - Materializing Data into Apache Iceberg 04:17 - Backfilling Data into Iceberg 05:21 - Querying Iceberg Tables with Python 06:10 - Conclusion: Demo Recap
MongoDB to Snowflake in real-time (no Debezium)
In this video, Jeff from Estuary walks you through how to move data from MongoDB to Snowflake using Estuary, a real-time ETL platform. Learn the key benefits of using Estuary, including low-latency Change Data Capture (CDC) and automatic unpacking of nested documents. You'll also see a step-by-step guide to setting up a MongoDB Atlas database and creating a real-time data pipeline with Estuary. Key features covered: - Real-time data replication from MongoDB to Snowflake - Low-latency data movement and automatic flattening of nested documents - Backfilling data and setting up materializations to Snowflake in just a few clicks #MongodbtoSnowflake #changedatacapture If you have any questions, feel free to join our community Slack. Start building real-time data pipelines with Estuary today! Sign up for a free account: https://dashboard.estuary.dev/register Join our Slack community: https://estuary-dev.slack.com/join/shared_invite/zt-86nal6yr-VPbv~YfZE9Q~6Zl~gmZdFQ#/shared-invite/email Blog: https://estuary.dev/mongodb-to-snowflake/ 0:00 – Introduction: Moving Data to Snowflake with Estuary 0:12 – Key Benefits of Using Estuary: Real-Time Data Integration 1:18 – Automatic Flattening of Nested Data 2:08 – Testing Connection to the MongoDB Source 2:25 – Saving and Publishing the Real-Time Pipeline 2:54 – Sending Data to Snowflake and Other Destinations 3:12 – Real-Time Backfill and Data Materialization to Snowflake
Estuary 101: How To Build Right-Time Data Pipelines
Join hosts Dani and Zulf for a fast-paced walkthrough of how to design and ship right-time data pipelines with Estuary. In this session, you’ll get: - Context: What “right-time” really means, where Estuary fits among batch vs. streaming and managed vs. self-hosted options, and why unified ingestion reduces cost and complexity. - Live End-to-End Demo: Connect CDC sources, apply declarative transformations, and materialize data simultaneously into a warehouse, analytical engines, and object storage—plus a look at observability, error recovery, and real-world scenarios like schema drift and backfills. - Live Q&A: Ask about your specific stack, pipeline designs, and how to scale Estuary for enterprise workloads. Perfect for data and analytics engineers, architects, and platform owners who want fresher data with fewer moving parts.
How to Stream Data into Snowflake
Ingest data into a Snowflake warehouse using real-time Snowpipe Streaming or using batch COPY INTO commands. Estuary makes Snowflake integration simple with pre-built no-code connectors. Following along? Find the copy/pasteable commands in Estuary’s Snowflake docs: https://docs.estuary.dev/reference/Connectors/materialization-connectors/Snowflake/ - Set up your first data pipeline for free at Estuary: https://dashboard.estuary.dev/register/?utm_source=youtube&utm_medium=social&utm_campaign=snowflake_ingestion - Learn more about Estuary’s Snowflake capabilities: https://estuary.dev/solutions/technology/real-time-snowflake-streaming/ - Read the complete guide to Snowpipe Streaming: https://estuary.dev/blog/snowpipe-streaming-fast-snowflake-ingestion/ - Discover how Snowflake fared in Estuary’s Data Warehouse Benchmark: https://estuary.dev/data-warehouse-benchmark-report/ - Download Snowflake Ingestion Playbook: https://estuary.dev/snowflake-ingestion-whitepaper/ FAQ 1. What is the fastest way to load data into Snowflake? Snowpipe Streaming with row-based ingestion. In Estuary, you can enable it per table using Delta Updates. 2. Why use key pair authentication for Snowflake? It provides strong security, short-lived tokens, and is Snowflake’s recommended approach for service integrations like Estuary. 3. Can I mix real-time and batch ingestion in the same pipeline? Yes. With Estuary’s Snowflake connector, you can run some tables in batch (COPY INTO or Snowpipe) and others in real time with Snowpipe Streaming. Media resources used in this video are from Pexels, Canva, and the YouTube Studio Audio Library. 0:00 Introduction 1:05 Snowflake concerns 1:51 Ingestion options 3:23 Beginning the demo 3:47 Create Snowflake resources 4:28 User auth setup 5:17 Estuary connector config 6:44 Customization options 8:07 Wrapping up
What’s Next for Data Warehouses? Lessons from Our Benchmark and Emerging Trends
Dani and Ben talks about key findings on performance ceilings, cost traps, and failure modes, and explore the major trends reshaping data warehouse architecture, including: - Separation of Compute & Storage – How Snowflake Gen2, Databricks serverless, and open table formats like Iceberg are changing the game. - Lakehouse Reality Check: What’s working for teams adopting Iceberg, schema evolution patterns, and lake-native pipelines. - Flexibility Over Centralization: Moving beyond “one warehouse to rule them all.
Oracle CDC Setup Tutorial: Stream Oracle Data in Real Time with Estuary
Learn how to set up **Oracle Change Data Capture (CDC)** with Estuary and continuously capture changes from an Oracle Database using **Oracle LogMiner**. In this tutorial, we use a Docker-based Oracle instance to walk through the CDC setup from start to finish. You’ll see how to configure archive logging, set up the required permissions and supplemental logging, configure archive-log retention, and connect Oracle to Estuary. Once your Oracle data is captured, you can use it for real-time replication and data pipelines to destinations such as **Snowflake, BigQuery, ClickHouse, PostgreSQL, Databricks, Amazon Redshift, Kafka, and more**. ### What you'll learn - How Oracle Change Data Capture works with LogMiner - How Oracle redo logs and archive logs are used for CDC - How to enable archive logging and supplemental logging - How to configure permissions for the Estuary Oracle user - How archive-log retention affects CDC recovery - How Oracle container databases (CDB/PDB) affect setup - How to connect Oracle to Estuary - How to discover tables and start capturing changes This setup can be used as the foundation for pipelines such as **Oracle to Snowflake CDC, Oracle to BigQuery replication, Oracle to ClickHouse, Oracle to PostgreSQL, and other Oracle data integration workflows**. ### Follow along Example Oracle CDC project: https://github.com/estuary/examples/tree/main/oracle-capture Try Estuary for free: https://dashboard.estuary.dev/register/?utm_source=youtube&utm_medium=social&utm_campaign=oracle_capture Oracle connector documentation, including setup instructions for non-container databases: https://docs.estuary.dev/reference/Connectors/capture-connectors/OracleDB/ Explore Oracle integrations: https://estuary.dev/source/oracle/ Have questions? Join the Estuary Slack community: https://go.estuary.dev/slack ### Chapters 0:00 Introduction 0:40 Oracle & CDC 2:30 Demo: Project overview 5:42 Demo: Run container 7:22 Demo: Estuary setup 8:50 Wrapping up Media resources used in this video are from Pexels, Canva, and the YouTube Studio Audio Library. #OracleCDC #OracleDatabase #ChangeDataCapture #DataEngineering #Estuary
Stream CRM Data from HubSpot to MotherDuck
Need blazing-fast analytics to keep up with your customers? Try sending your HubSpot data to MotherDuck: Estuary makes the process simple and straightforward. Register for free at: https://dashboard.estuary.dev/register/?utm_source=youtube&utm_medium=social&utm_campaign=hubspot_motherduck Ready to dive in deeper? Try these resources: 📄 Hubspot capture connector docs: https://docs.estuary.dev/reference/Connectors/capture-connectors/HubSpot-real-time/ 🐤 MotherDuck materialization docs: https://docs.estuary.dev/reference/Connectors/materialization-connectors/motherduck/ 💬 Our Slack community: https://go.estuary.dev/slack Music sourced from the YouTube Studio Audio Library 0:00 Introduction 0:20 Starting the pipeline 1:01 (Optional) Create HubSpot access token 1:39 Finish capture 2:15 MotherDuck materialization 2:51 Create staging bucket 4:40 MotherDuck credentials 5:20 Complete pipeline & wrap up
Save Webhook Data to Databricks in Real Time
Learn how to stream incoming webhook data to Databricks without setting up and maintaining your own server for webhook captures. Follow along with our 3-minute demo and try out Estuary for free → https://dashboard.estuary.dev/register Learn more from our: - Website: https://estuary.dev/ - Webhook capture docs: https://docs.estuary.dev/reference/Connectors/capture-connectors/http-ingest/ - Databricks materialization docs: https://docs.estuary.dev/reference/Connectors/materialization-connectors/databricks/ - Blog article on webhook setup: https://estuary.dev/blog/webhook-setup/ 0:00 Intro 0:19 Set up webhook capture 1:19 Configure webhook in Square 2:08 Create Databricks materialization 2:43 Outro
Unify Your Data in Microsoft Fabric with Estuary
Want to get your data into Microsoft Fabric—fast and without writing code? Discover what unified data can do with a Microsoft Fabric integration. We’ll cover what makes this relatively new data platform unique and how you can enhance its capabilities further using Estuary. A step-by-step demo walks through Fabric warehouse connector setup in Estuary so you can get your data flowing. Interested in more? - Register for a free Estuary account: https://dashboard.estuary.dev/register - Learn more about Microsoft Fabric: https://estuary.dev/blog/what-is-microsoft-fabric/ - Find source connectors to go with your Fabric destination: https://docs.estuary.dev/reference/Connectors/capture-connectors/ - Join us on Slack: https://go.estuary.dev/slack Media resources used in this video are from Pexels and the YouTube Studio Audio Library. 0:00 Introduction 0:26 Microsoft Fabric 1:33 Covering gaps with Estuary 2:33 Beginning connector creation 3:23 Creating a warehouse 3:59 Configuring a service principal 5:59 Creating a storage account 6:55 Wrapping up the connector 7:33 Outro
Stream Real‑Time Data to Databricks with Estuary
Learn how to stream real-time data from PostgreSQL into Databricks using Estuary — no code, no maintenance. In this demo, Dani walks through: • Setting up a Databricks SQL Warehouse and generating a personal access token • Capturing the users and transactions tables from PostgreSQL in Estuary • Materializing those tables directly into Databricks using the built-in connector • Monitoring live data replication and verifying the results in Databricks SQL Warehouse Highlights of Estuary: • Real-time change data capture with millisecond latency • Native support for Databricks Unity Catalog and Delta Lake • Zero-code pipeline setup with automatic backfill and continuous sync Why it matters Streaming live data into Databricks unlocks fresh analytics, real‑time dashboards, and feeding ML models with up‑to‑date data — all without complex ETL or scripting 🔗 Learn more & get started: • Official Estuary guide: https://estuary.dev/real-time-fraud-detection-databricks/ • https://estuary.dev/blog/load-data-into-databricks/ • Start building for free at: https://dashboard.estuary.dev/register If you have questions or need help, jump into our community Slack or check the docs. #databrickstutorial #databricks 00:00 - Introduction 00:49 - Materializing Data to Databricks 01:45 - Verifying Data in Databricks 02:04 - Conclusion
Seamless Data Integration, Unlimited Potential
Discover the simplest way to connect and move your data.Get hands-on for free, or schedule a demo to see the possibilities for your team.