Building Autonomous Data Pipelines for the AI Era | ft. Dave Yaffe, CEO & Co-Founder of Estuary
Today, 90% of AI projects depend directly on data engineering pipelines, yet 30-40% of pipelines experience failures every week as data environments grow more complex.
Database Trends and Applications (DBTA) brought together industry leaders to discuss:
1) Why traditional pipelines struggle to support modern AI workloads
2) How automation and observability are transforming data engineering
3) Strategies for improving pipeline reliability and data quality
4) Best practices for building scalable, AI-ready data pipelines
Dave Yaffe, CEO & Co-Founder of Estuary, explained the architecture behind Estuary v2.0 and why building separate pipelines for analytics and operations creates unnecessary complexity. He also highlighted the importance of correcting data at the source, why the challenge with AI success might be due more to human habits than with the technology itself, and how Estuary's approach to AI has evolved over time.
LINKS
Set up your first data pipeline for free at Estuary: https://dashboard.estuary.dev/registe...
Join Estuary on Slack: https://go.estuary.dev/slack
Contact us: https://estuary.dev/contact-us
Credit
Music: びじえもん (BGMemon)
More videos
Estuary | The Right Time Data Platform
Welcome to Estuary, the Right Time Data Platform built for modern data teams. With Estuary, you can move and transform data between hundreds of systems at sub second latency or in batch, depending on your business needs. • Capture data from source systems using pre built, no code connectors. • Automatically infer schemas and manage both real time and historical events in collections. • Materialize your data to any destination with ease and flexibility. • Choose your deployment model: fully SaaS, Bring Your Own Cloud, or private deployment with enterprise level security. Start streaming an ocean of data and get going today: 🌊 https://dashboard.estuary.dev/register/?utm_source=youtube&utm_medium=social&utm_campaign=overview_video Learn more: 🌐 On our site: https://www.estuary.dev/?utm_source=youtube&utm_medium=social&utm_campaign=flow_overview 📚 In our docs: https://docs.estuary.dev/?utm_source=youtube&utm_medium=social&utm_campaign=flow_overview Connect with us: 💬 On Slack: https://go.estuary.dev/slack 🧑💻 In GitHub: https://github.com/estuary ℹ️ On LinkedIn: https://www.linkedin.com/company/estuary-tech/ #righttimedata #datapipelines #streamingdata #realtimeanalytics #CDC #dataengineering #Estuary
Estuary 101: Right Time Data Integration for ClickHouse
Join Zulf Qureshi (Head of Sales Engineering @ Estuary) for a fast-paced walkthrough of how to design and ship right-time data pipelines for ClickHouse with Estuary. In this session, you’ll get: Context: What “right-time” really means, where Estuary fits among batch vs. streaming and managed vs. self-hosted options, and why unified ingestion reduces cost and complexity. Live End-to-End Demo: Connect CDC sources, apply declarative transformations, and materialize data simultaneously into a warehouse, analytical engines, and object storage—plus a look at observability, error recovery, and real-world scenarios like schema drift and backfills. Estuary helps organizations streamline their data integration for AI, analytics, and operational workflows without the constraints of a rigid architecture. As the right-time data platform, Estuary replaces fragmented CDC, streaming, and batch pipelines with one managed system and predictable pricing. Try Estuary for free today: https://dashboard.estuary.dev/register/
PostgreSQL CDC: What, Why, and How
Set up PostgreSQL change data capture (CDC) without worrying about your WAL or replication slot loss. This step-by-step Postgres CDC guide covers WAL configuration, replication user setup, publication creation, and building your first real-time pipeline to Snowflake, Iceberg, or ClickHouse, no Kafka or Debezium required. We walk through why CDC beats batch replication for Postgres, how the WAL and replication slots actually work (and what goes wrong), and a full setup demo with Estuary, a real-time CDC platform that manages WAL advancement and decouples your source from your destination. LINKS - Set up your first data pipeline for free at Estuary: https://dashboard.estuary.dev/register/?utm_source=youtube&utm_medium=social&utm_campaign=postgres - Discover more PostgreSQL configuration options in Estuary’s docs: https://docs.estuary.dev/reference/Connectors/capture-connectors/PostgreSQL/ - Explore available destinations to continue your pipeline: https://docs.estuary.dev/reference/Connectors/materialization-connectors/ - Join Estuary on Slack: https://go.estuary.dev/slack - Contact us: https://estuary.dev/contact-us FAQ 1. What is Postgres CDC? CDC, or change data capture, is a method to record all updates made to a database. In conjunction with PostgreSQL, CDC often uses the Postgres write-ahead log for data replication. We start discussing Postgres CDC at 3:36 2. When should you use CDC? Is it just for real-time data? CDC can be beneficial for many different use cases since it has a low impact on the replicated Postgres database and handles replication more efficiently than batch queries. Since CDC is also very low latency, it’s a top choice for real-time use cases, though batch workflows will still benefit from an efficiency/low-impact standpoint. We go over CDC use cases at 2:29 3. Is CDC complicated to implement? CDC can be complex when creating and managing a DIY solution. You need to be careful that the WAL doesn’t grow beyond expected bounds. A managed solution can make setup and maintenance much simpler; just make sure that the integration platform follows CDC best practices, like keeping the source and destination decoupled. A brief implementation guide with Estuary starts at 5:55 Media resources used in this video are from Canva and the YouTube Studio Audio Library. CHAPTERS 0:00 Intro 0:45 On PostgreSQL 2:05 Change Data Capture 3:36 Postgres + CDC 5:55 Demo: Postgres setup 7:36 Demo: Basic Estuary setup 8:52 Demo: Additional settings 10:35 Wrap up
Stream Real‑Time Data to Databricks with Estuary
Learn how to stream real-time data from PostgreSQL into Databricks using Estuary — no code, no maintenance. In this demo, Dani walks through: • Setting up a Databricks SQL Warehouse and generating a personal access token • Capturing the users and transactions tables from PostgreSQL in Estuary • Materializing those tables directly into Databricks using the built-in connector • Monitoring live data replication and verifying the results in Databricks SQL Warehouse Highlights of Estuary: • Real-time change data capture with millisecond latency • Native support for Databricks Unity Catalog and Delta Lake • Zero-code pipeline setup with automatic backfill and continuous sync Why it matters Streaming live data into Databricks unlocks fresh analytics, real‑time dashboards, and feeding ML models with up‑to‑date data — all without complex ETL or scripting 🔗 Learn more & get started: • Official Estuary guide: https://estuary.dev/real-time-fraud-detection-databricks/ • https://estuary.dev/blog/load-data-into-databricks/ • Start building for free at: https://dashboard.estuary.dev/register If you have questions or need help, jump into our community Slack or check the docs. #databrickstutorial #databricks 00:00 - Introduction 00:49 - Materializing Data to Databricks 01:45 - Verifying Data in Databricks 02:04 - Conclusion
PostgreSQL CDC Setup Tutorial: Real-Time Data Capture with Estuary (Step-by-Step)
Learn how to set up PostgreSQL Change Data Capture (CDC) with Estuary in under 10 minutes. This tutorial walks through enabling logical replication on Google Cloud SQL, creating the replication user, and connecting your Postgres database to Estuary for real-time streaming to Snowflake, BigQuery, Databricks, and more. ⚠️ Heads up: This video was recorded on an earlier version of Estuary. The core concepts (logical replication, replication user, Cloud SQL flags) still apply, but the UI and a few setup steps have been updated since. For the latest instructions, follow the docs: https://docs.estuary.dev/reference/Connectors/capture-connectors/PostgreSQL/ 👉 Try Estuary Free: https://dashboard.estuary.dev/register 👉 Book a Demo: https://estuary.dev/demo 👉 Full Docs: https://docs.estuary.dev/reference/Connectors/capture-connectors/PostgreSQL/ ━━━━━━━━━━━━━━━━━━━━━━━ WHAT YOU'LL LEARN ━━━━━━━━━━━━━━━━━━━━━━━ ✅ How to enable logical replication on Cloud SQL PostgreSQL ✅ How to create a dedicated replication user with correct privileges ✅ How to configure the Estuary PostgreSQL capture connector ✅ How to troubleshoot connection string and IP allowlist errors ━━━━━━━━━━━━━━━━━━━━━━━ CHAPTERS ━━━━━━━━━━━━━━━━━━━━━━━ 0:00 Intro: PostgreSQL CDC with Estuary 0:52 Enable Logical Replication on Cloud SQL PostgreSQL 2:45 Create Estuary Capture for PostgreSQL 3:00 PostgreSQL Endpoint Configuration (Host, Port, User) 3:55 Troubleshooting: No Collections Found in Source Database 5:14 Troubleshooting: PostgreSQL Connection String Errors ━━━━━━━━━━━━━━━━━━━━━━━ WHY POSTGRESQL CDC WITH ESTUARY ━━━━━━━━━━━━━━━━━━━━━━━ Estuary is a real-time data platform built for Change Data Capture. Unlike batch ETL tools, Estuary streams row-level changes from PostgreSQL's write-ahead log (WAL) to destinations like Snowflake, BigQuery, Databricks, Redshift, and ClickHouse with sub-second latency. Works with PostgreSQL on: - Google Cloud SQL - Amazon RDS / Aurora - Azure Database for PostgreSQL - Supabase, Neon, Heroku - Self-hosted PostgreSQL ━━━━━━━━━━━━━━━━━━━━━━━ CONNECT WITH ESTUARY ━━━━━━━━━━━━━━━━━━━━━━━ 🌐 Website: https://estuary.dev/ 💬 Slack Community: https://estuary-dev.slack.com/ 📚 Docs: https://docs.estuary.dev/ #PostgreSQL #CDC #ChangeDataCapture #DataEngineering #DataPipeline #RealTimeData #Estuary #CloudSQL #LogicalReplication #ETL
Seamless Data Integration, Unlimited Potential
Discover the simplest way to connect and move your data.Get hands-on for free, or schedule a demo to see the possibilities for your team.