Stream Data to Apache Iceberg with Estuary
Learn about the Apache Iceberg table format, why it’s essential for organizing your data lake, and how to load data into Iceberg using Estuary. We’ll cover a brief intro to Iceberg before demoing the connector setup with Estuary, Amazon S3, and AWS Glue for real-time and batch data integration.
With Estuary, you can stream structured or unstructured data directly into Iceberg tables — whether your source is PostgreSQL, Kafka, Snowflake, MongoDB, or many others — making it easy to build a scalable, query-ready data lakehouse architecture.
Find more at Estuary’s:
- Website: https://estuary.dev/
- Docs: https://docs.estuary.dev/
- Introduction to Iceberg: https://estuary.dev/apache-iceberg-tutorial-guide/
- Iceberg connector documentation: https://docs.estuary.dev/reference/Connectors/materialization-connectors/amazon-s3-iceberg/
#ApacheIceberg #datalakehouse
Media resources used in this video are from Pexels and the YouTube Studio Audio Library.
0:00 Intro
1:00 What is Iceberg?
2:30 Beginning connector setup in Estuary
3:17 AWS resources
5:00 Additional config and catalogs
6:05 Wrapping up connector creation
6:36 Review and outro
More videos
Streaming Data Lakehouse Tutorial: MongoDB to Apache Iceberg
Learn how to connect MongoDB to Apache Iceberg in Iceberg table format using Estuary Flow. In this step-by-step demo, we show you how to: 1. Set up a MongoDB source and configure secure connections. 2. Create real-time pipelines to load data into Amazon S3. 3. Leverage the AWS S3 Iceberg Connector with AWS Glue for table cataloging. Estuary Flow simplifies real-time data integration with powerful features like advanced security connections, automated materialization, and streamlined pipeline management. Whether you're handling transactional data or syncing complex data streams, Estuary Flow has you covered. 👉 Try Estuary Flow: https://dashboard.estuary.dev/register 👉 Read the Documentation: https://docs.estuary.dev/ #MongoDBtoIceberg 0:00 - Introduction: Overview of the demo and Estuary Flow. 0:07 - Step 1: Setting Up MongoDB Source: Configuring MongoDB as the data source. 0:44 - Step 2: Reviewing Collections: Selecting collections to sync. 1:03 - Step 3: Setting Up S3 Destination: Configuring the AWS S3 Iceberg connector. 1:37 - Step 4: Testing and Publishing Pipeline: Testing the connection and publishing the pipeline. 2:07 - Final Verification: Verifying MongoDB data in S3 as Iceberg tables.
PostgreSQL to Iceberg - Streaming Lakehouse Foundations
Stream Real-Time Data from Postgres to Iceberg with Change Data Capture and Estuary Flow In this step-by-step tutorial, we demonstrate how to set up and stream real-time data from a PostgreSQL database into Iceberg tables using change data capture (CDC) with Estuary Flow. Learn how to capture, ingest, and materialize data using Estuary Flow's seamless integration. This demo uses a sales database to showcase how changes in a PostgreSQL table are tracked and replicated into an Iceberg table stored in AWS S3. Check out Estuary Flow's Iceberg integration: https://estuary.dev/destination/s3-iceberg/ Join Estuary Flow's community Slack: https://estuary-dev.slack.com/join/shared_invite/zt-86nal6yr-VPbv~YfZE9Q~6Zl~gmZdFQ#/shared-invite/email 00:00 - Introduction: Streaming Data from Postgres to Iceberg 00:18 - Postgres Sales Database Overview 01:08 - Starting Change Data Capture (CDC) with Estuary Flow 02:09 - Materializing Data into Apache Iceberg 04:17 - Backfilling Data into Iceberg 05:21 - Querying Iceberg Tables with Python 06:10 - Conclusion: Demo Recap
Seamless Data Integration, Unlimited Potential
Discover the simplest way to connect and move your data.Get hands-on for free, or schedule a demo to see the possibilities for your team.