Estuary 101: Right Time Data Integration for ClickHouse
Join Zulf Qureshi (Head of Sales Engineering @ Estuary) for a fast-paced walkthrough of how to design and ship right-time data pipelines for ClickHouse with Estuary.
In this session, you’ll get:
Context: What “right-time” really means, where Estuary fits among batch vs. streaming and managed vs. self-hosted options, and why unified ingestion reduces cost and complexity.
Live End-to-End Demo: Connect CDC sources, apply declarative transformations, and materialize data simultaneously into a warehouse, analytical engines, and object storage—plus a look at observability, error recovery, and real-world scenarios like schema drift and backfills.
Estuary helps organizations streamline their data integration for AI, analytics, and operational workflows without the constraints of a rigid architecture. As the right-time data platform, Estuary replaces fragmented CDC, streaming, and batch pipelines with one managed system and predictable pricing.
Try Estuary for free today: https://dashboard.estuary.dev/register/
More videos
Estuary | The Right Time Data Platform
Welcome to Estuary, the Right Time Data Platform built for modern data teams. With Estuary, you can move and transform data between hundreds of systems at sub second latency or in batch, depending on your business needs. • Capture data from source systems using pre built, no code connectors. • Automatically infer schemas and manage both real time and historical events in collections. • Materialize your data to any destination with ease and flexibility. • Choose your deployment model: fully SaaS, Bring Your Own Cloud, or private deployment with enterprise level security. Start streaming an ocean of data and get going today: 🌊 https://dashboard.estuary.dev/register/?utm_source=youtube&utm_medium=social&utm_campaign=overview_video Learn more: 🌐 On our site: https://www.estuary.dev/?utm_source=youtube&utm_medium=social&utm_campaign=flow_overview 📚 In our docs: https://docs.estuary.dev/?utm_source=youtube&utm_medium=social&utm_campaign=flow_overview Connect with us: 💬 On Slack: https://go.estuary.dev/slack 🧑💻 In GitHub: https://github.com/estuary ℹ️ On LinkedIn: https://www.linkedin.com/company/estuary-tech/ #righttimedata #datapipelines #streamingdata #realtimeanalytics #CDC #dataengineering #Estuary
![[WEBINAR] Estuary Agent Skills: From Setup to Success thumbnail](/static/0bfcc292be14e15168775e4eda941993/71d4d/maxresdefault.webp)
[WEBINAR] Estuary Agent Skills: From Setup to Success
In this webinar, Dani Palma (Developer Relations, Estuary) walks through the new Estuary Agent Skills product launch, from installation to building and debugging an Estuary pipeline. Agent Skills lets your AI assistant create and monitor production-grade Estuary pipelines in the tool your team already uses. You describe the pipeline using plain language prompts. The agent builds it. You review, approve, and run it. Agent Skills are compatible with Claude Code, Cursor, Codex, and 45+ other agents that support the open SKILL.md standard. We currently offer 37 skills that cover captures, materializations, pipeline operations, derivations, and schema management. Interesting in trying out Agent Skills? Here's how to get started: Agent Skills docs: https://docs.estuary.dev/guides/agent-skills/ Agent Skills overview: https://www.youtube.com/watch?v=lc71ILyd89g Agent Skills GitHub repo: https://github.com/estuary/agent-skills
PostgreSQL CDC: What, Why, and How
Set up PostgreSQL change data capture (CDC) without worrying about your WAL or replication slot loss. This step-by-step Postgres CDC guide covers WAL configuration, replication user setup, publication creation, and building your first real-time pipeline to Snowflake, Iceberg, or ClickHouse, no Kafka or Debezium required. We walk through why CDC beats batch replication for Postgres, how the WAL and replication slots actually work (and what goes wrong), and a full setup demo with Estuary, a real-time CDC platform that manages WAL advancement and decouples your source from your destination. LINKS - Set up your first data pipeline for free at Estuary: https://dashboard.estuary.dev/register/?utm_source=youtube&utm_medium=social&utm_campaign=postgres - Discover more PostgreSQL configuration options in Estuary’s docs: https://docs.estuary.dev/reference/Connectors/capture-connectors/PostgreSQL/ - Explore available destinations to continue your pipeline: https://docs.estuary.dev/reference/Connectors/materialization-connectors/ - Join Estuary on Slack: https://go.estuary.dev/slack - Contact us: https://estuary.dev/contact-us FAQ 1. What is Postgres CDC? CDC, or change data capture, is a method to record all updates made to a database. In conjunction with PostgreSQL, CDC often uses the Postgres write-ahead log for data replication. We start discussing Postgres CDC at 3:36 2. When should you use CDC? Is it just for real-time data? CDC can be beneficial for many different use cases since it has a low impact on the replicated Postgres database and handles replication more efficiently than batch queries. Since CDC is also very low latency, it’s a top choice for real-time use cases, though batch workflows will still benefit from an efficiency/low-impact standpoint. We go over CDC use cases at 2:29 3. Is CDC complicated to implement? CDC can be complex when creating and managing a DIY solution. You need to be careful that the WAL doesn’t grow beyond expected bounds. A managed solution can make setup and maintenance much simpler; just make sure that the integration platform follows CDC best practices, like keeping the source and destination decoupled. A brief implementation guide with Estuary starts at 5:55 Media resources used in this video are from Canva and the YouTube Studio Audio Library. CHAPTERS 0:00 Intro 0:45 On PostgreSQL 2:05 Change Data Capture 3:36 Postgres + CDC 5:55 Demo: Postgres setup 7:36 Demo: Basic Estuary setup 8:52 Demo: Additional settings 10:35 Wrap up
Building Autonomous Data Pipelines for the AI Era | ft. Dave Yaffe, CEO & Co-Founder of Estuary
Today, 90% of AI projects depend directly on data engineering pipelines, yet 30-40% of pipelines experience failures every week as data environments grow more complex. Database Trends and Applications (DBTA) brought together industry leaders to discuss: 1) Why traditional pipelines struggle to support modern AI workloads 2) How automation and observability are transforming data engineering 3) Strategies for improving pipeline reliability and data quality 4) Best practices for building scalable, AI-ready data pipelines Dave Yaffe, CEO & Co-Founder of Estuary, explained the architecture behind Estuary v2.0 and why building separate pipelines for analytics and operations creates unnecessary complexity. He also highlighted the importance of correcting data at the source, why the challenge with AI success might be due more to human habits than with the technology itself, and how Estuary's approach to AI has evolved over time. LINKS Set up your first data pipeline for free at Estuary: https://dashboard.estuary.dev/registe... Join Estuary on Slack: https://go.estuary.dev/slack Contact us: https://estuary.dev/contact-us Credit Music: びじえもん (BGMemon)
Dekaf: How to Use Kafka Minus the Kafka
Have you ever wanted Kafka’s real-time pub/sub benefits without implementing and maintaining a whole Kafka ecosystem yourself? Learn how with Dekaf. We’ll cover some Kafka basics to help explain how Estuary’s Kafka API compatibility layer fits seamlessly into a modern data architecture. - Register for a free Estuary account: https://dashboard.estuary.dev/register/?utm_source=youtube&utm_medium=social&utm_campaign=dekaf_video - Learn more about Dekaf: https://docs.estuary.dev/reference/Connectors/dekaf/ - Find the example kcat command: https://docs.estuary.dev/guides/dekaf_reading_collections_from_kafka/#2-set-up-your-kafka-client - Join us on Slack: https://go.estuary.dev/slack Media resources used in this video are from Pexels, Canva, and the YouTube Studio Audio Library. 0:00 Introduction 0:30 What is Kafka? 1:04 The Kafka Ecosystem 2:39 Integrating with Kafka Consumers... 3:26 ...using Dekaf 4:12 Dekaf Setup 6:16 kcat Test 6:45 Final Thoughts
How to Connect NetSuite to Estuary for Real-Time Data Integration
Learn how to connect NetSuite to Estuary and stream your ERP data in real-time to destinations like Snowflake, BigQuery, Databricks, Redshift, and more — no custom ETL code required. This tutorial walks through the full NetSuite setup (SuiteAnalytics, integration, role, user, and token-based authentication) and the Estuary capture configuration end-to-end. 👉 Try Estuary Free: https://dashboard.estuary.dev/register 👉 Book a Demo: https://estuary.dev/demo 👉 NetSuite Connector Docs: https://docs.estuary.dev/reference/Connectors/capture-connectors/netsuite/ ━━━━━━━━━━━━━━━━━━━━━━━ WHAT YOU'LL LEARN ━━━━━━━━━━━━━━━━━━━━━━━ ✅ How to enable SuiteAnalytics in NetSuite ✅ How to create a NetSuite integration with token-based authentication ✅ How to set up a role and assign permissions for Estuary access ✅ How to generate NetSuite access tokens (consumer key, secret, token ID, token secret) ✅ How to configure the Estuary NetSuite capture connector ✅ How to find your NetSuite account ID and internal role ID ━━━━━━━━━━━━━━━━━━━━━━━ CHAPTERS ━━━━━━━━━━━━━━━━━━━━━━━ 0:00 Intro: NetSuite to Estuary Integration 0:18 NetSuite Setup: SuiteAnalytics, Account ID, Integration 2:08 Create Role and Assign User Permissions 2:37 Generate NetSuite Access Tokens 3:25 Enable Internal IDs in NetSuite 4:02 Create the NetSuite Capture in Estuary 5:16 Endpoint Configuration and Authentication 5:43 Test, Save, and Publish the Capture ━━━━━━━━━━━━━━━━━━━━━━━ WHY NETSUITE WITH ESTUARY ━━━━━━━━━━━━━━━━━━━━━━━ NetSuite is one of the most widely used ERPs for finance, operations, and inventory data, but getting that data into a modern warehouse is notoriously painful. Estuary is a real-time data platform that streams NetSuite data into analytics destinations with sub-second latency — without requiring custom SuiteScript, scheduled CSV exports, or homegrown ETL pipelines. Common NetSuite pipelines built on Estuary: - NetSuite to Snowflake - NetSuite to BigQuery - NetSuite to Databricks - NetSuite to Redshift - NetSuite to PostgreSQL ━━━━━━━━━━━━━━━━━━━━━━━ CONNECT WITH ESTUARY ━━━━━━━━━━━━━━━━━━━━━━━ 🌐 Website: https://estuary.dev/ 💬 Slack Community: https://estuary-dev.slack.com/ 📚 Docs: https://docs.estuary.dev #NetSuite #DataIntegration #ERP #DataPipeline #Estuary #SuiteAnalytics #NetSuiteToSnowflake #DataEngineering #RealTimeData #ETL
Amazon RDS to Snowflake in Real Time (Postgres CDC)
Move data from Amazon RDS (PostgreSQL) to Snowflake in real time using change data capture, with no batch jobs and no custom code. In this short demo, Estuary captures live changes from an RDS Postgres database and materializes them into Snowflake in minutes. How do you sync RDS Postgres to Snowflake in real time? Estuary captures inserts, updates, and deletes from your Postgres database, lands them in a collection, and materializes that collection into Snowflake with low latency. Insert a row in Postgres and watch it appear in Snowflake seconds later, no pipeline to build or maintain. ⏱️ Chapters 0:00 Real-time data integration with Estuary 0:11 Setting up Amazon RDS PostgreSQL as a source 0:27 Configuring the Estuary source connector 0:51 Testing and publishing the source 1:09 Creating a collection for real-time data 1:36 Materializing data to Snowflake 2:01 Testing real-time data ingestion 🔑 What you'll learn - How real-time CDC differs from batch ETL for loading Snowflake - Connecting an Amazon RDS PostgreSQL source in Estuary - Creating a collection and materializing it to Snowflake - Verifying the live sync by inserting a record in Postgres 🔗 Try Estuary free: https://dashboard.estuary.dev/register 📚 Docs: https://docs.estuary.dev 📝 Tutorial: https://estuary.dev/blog/tutorial/ 🛠️ Tools: Amazon RDS, PostgreSQL, Snowflake, Estuary #Snowflake #Postgres #CDC #ChangeDataCapture #DataEngineering #RealTimeData #AmazonRDS #DataPipeline #ETL #Estuary
Stream Real‑Time Data to Databricks with Estuary
Learn how to stream real-time data from PostgreSQL into Databricks using Estuary — no code, no maintenance. In this demo, Dani walks through: • Setting up a Databricks SQL Warehouse and generating a personal access token • Capturing the users and transactions tables from PostgreSQL in Estuary • Materializing those tables directly into Databricks using the built-in connector • Monitoring live data replication and verifying the results in Databricks SQL Warehouse Highlights of Estuary: • Real-time change data capture with millisecond latency • Native support for Databricks Unity Catalog and Delta Lake • Zero-code pipeline setup with automatic backfill and continuous sync Why it matters Streaming live data into Databricks unlocks fresh analytics, real‑time dashboards, and feeding ML models with up‑to‑date data — all without complex ETL or scripting 🔗 Learn more & get started: • Official Estuary guide: https://estuary.dev/real-time-fraud-detection-databricks/ • https://estuary.dev/blog/load-data-into-databricks/ • Start building for free at: https://dashboard.estuary.dev/register If you have questions or need help, jump into our community Slack or check the docs. #databrickstutorial #databricks 00:00 - Introduction 00:49 - Materializing Data to Databricks 01:45 - Verifying Data in Databricks 02:04 - Conclusion
MongoDB to Snowflake in real-time (no Debezium)
In this video, Jeff from Estuary walks you through how to move data from MongoDB to Snowflake using Estuary, a real-time ETL platform. Learn the key benefits of using Estuary, including low-latency Change Data Capture (CDC) and automatic unpacking of nested documents. You'll also see a step-by-step guide to setting up a MongoDB Atlas database and creating a real-time data pipeline with Estuary. Key features covered: - Real-time data replication from MongoDB to Snowflake - Low-latency data movement and automatic flattening of nested documents - Backfilling data and setting up materializations to Snowflake in just a few clicks #MongodbtoSnowflake #changedatacapture If you have any questions, feel free to join our community Slack. Start building real-time data pipelines with Estuary today! Sign up for a free account: https://dashboard.estuary.dev/register Join our Slack community: https://estuary-dev.slack.com/join/shared_invite/zt-86nal6yr-VPbv~YfZE9Q~6Zl~gmZdFQ#/shared-invite/email Blog: https://estuary.dev/mongodb-to-snowflake/ 0:00 – Introduction: Moving Data to Snowflake with Estuary 0:12 – Key Benefits of Using Estuary: Real-Time Data Integration 1:18 – Automatic Flattening of Nested Data 2:08 – Testing Connection to the MongoDB Source 2:25 – Saving and Publishing the Real-Time Pipeline 2:54 – Sending Data to Snowflake and Other Destinations 3:12 – Real-Time Backfill and Data Materialization to Snowflake
PostgreSQL CDC Setup Tutorial: Real-Time Data Capture with Estuary (Step-by-Step)
Learn how to set up PostgreSQL Change Data Capture (CDC) with Estuary in under 10 minutes. This tutorial walks through enabling logical replication on Google Cloud SQL, creating the replication user, and connecting your Postgres database to Estuary for real-time streaming to Snowflake, BigQuery, Databricks, and more. ⚠️ Heads up: This video was recorded on an earlier version of Estuary. The core concepts (logical replication, replication user, Cloud SQL flags) still apply, but the UI and a few setup steps have been updated since. For the latest instructions, follow the docs: https://docs.estuary.dev/reference/Connectors/capture-connectors/PostgreSQL/ 👉 Try Estuary Free: https://dashboard.estuary.dev/register 👉 Book a Demo: https://estuary.dev/demo 👉 Full Docs: https://docs.estuary.dev/reference/Connectors/capture-connectors/PostgreSQL/ ━━━━━━━━━━━━━━━━━━━━━━━ WHAT YOU'LL LEARN ━━━━━━━━━━━━━━━━━━━━━━━ ✅ How to enable logical replication on Cloud SQL PostgreSQL ✅ How to create a dedicated replication user with correct privileges ✅ How to configure the Estuary PostgreSQL capture connector ✅ How to troubleshoot connection string and IP allowlist errors ━━━━━━━━━━━━━━━━━━━━━━━ CHAPTERS ━━━━━━━━━━━━━━━━━━━━━━━ 0:00 Intro: PostgreSQL CDC with Estuary 0:52 Enable Logical Replication on Cloud SQL PostgreSQL 2:45 Create Estuary Capture for PostgreSQL 3:00 PostgreSQL Endpoint Configuration (Host, Port, User) 3:55 Troubleshooting: No Collections Found in Source Database 5:14 Troubleshooting: PostgreSQL Connection String Errors ━━━━━━━━━━━━━━━━━━━━━━━ WHY POSTGRESQL CDC WITH ESTUARY ━━━━━━━━━━━━━━━━━━━━━━━ Estuary is a real-time data platform built for Change Data Capture. Unlike batch ETL tools, Estuary streams row-level changes from PostgreSQL's write-ahead log (WAL) to destinations like Snowflake, BigQuery, Databricks, Redshift, and ClickHouse with sub-second latency. Works with PostgreSQL on: - Google Cloud SQL - Amazon RDS / Aurora - Azure Database for PostgreSQL - Supabase, Neon, Heroku - Self-hosted PostgreSQL ━━━━━━━━━━━━━━━━━━━━━━━ CONNECT WITH ESTUARY ━━━━━━━━━━━━━━━━━━━━━━━ 🌐 Website: https://estuary.dev/ 💬 Slack Community: https://estuary-dev.slack.com/ 📚 Docs: https://docs.estuary.dev/ #PostgreSQL #CDC #ChangeDataCapture #DataEngineering #DataPipeline #RealTimeData #Estuary #CloudSQL #LogicalReplication #ETL
Seamless Data Integration, Unlimited Potential
Discover the simplest way to connect and move your data.Get hands-on for free, or schedule a demo to see the possibilities for your team.