How to Load Data into Apache Iceberg (S3 + Glue Tutorial)
Learn about the Apache Iceberg table format, why it’s essential for organizing your data lake, and how to load data into Iceberg using Estuary. We’ll cover a brief intro to Iceberg before demoing the connector setup with Estuary, Amazon S3, and AWS Glue for real-time and batch data integration.
With Estuary, you can stream structured or unstructured data directly into Iceberg tables — whether your source is PostgreSQL, Kafka, Snowflake, MongoDB, or many others — making it easy to build a scalable, query-ready data lakehouse architecture.
Find more at Estuary’s:
- Website: https://estuary.dev/
- Docs: https://docs.estuary.dev/
- Introduction to Iceberg: https://estuary.dev/apache-iceberg-tutorial-guide/
- Iceberg connector documentation: https://docs.estuary.dev/reference/Connectors/materialization-connectors/amazon-s3-iceberg/
#ApacheIceberg #datalakehouse
Media resources used in this video are from Pexels and the YouTube Studio Audio Library.
0:00 Intro
1:00 What is Iceberg?
2:30 Beginning connector setup in Estuary
3:17 AWS resources
5:00 Additional config and catalogs
6:05 Wrapping up connector creation
6:36 Review and outro
More videos
Streaming Data Lakehouse Tutorial: MongoDB to Apache Iceberg
Learn how to connect MongoDB to Apache Iceberg in Iceberg table format using Estuary. In this step-by-step demo, we show you how to: 1. Set up a MongoDB source and configure secure connections. 2. Create real-time pipelines to load data into Amazon S3. 3. Leverage the AWS S3 Iceberg Connector with AWS Glue for table cataloging. Estuary simplifies real-time data integration with powerful features like advanced security connections, automated materialization, and streamlined pipeline management. Whether you're handling transactional data or syncing complex data streams, Estuary has you covered. 👉 Try Estuary: https://dashboard.estuary.dev/register 👉 Read the Documentation: https://docs.estuary.dev/ #MongoDBtoIceberg 0:00 - Introduction: Overview of the demo and Estuary. 0:07 - Step 1: Setting Up MongoDB Source: Configuring MongoDB as the data source. 0:44 - Step 2: Reviewing Collections: Selecting collections to sync. 1:03 - Step 3: Setting Up S3 Destination: Configuring the AWS S3 Iceberg connector. 1:37 - Step 4: Testing and Publishing Pipeline: Testing the connection and publishing the pipeline. 2:07 - Final Verification: Verifying MongoDB data in S3 as Iceberg tables.
PostgreSQL to Iceberg - Streaming Lakehouse Foundations
Stream Real-Time Data from Postgres to Iceberg with Change Data Capture and Estuary. In this step-by-step tutorial, we demonstrate how to set up and stream real-time data from a PostgreSQL database into Iceberg tables using change data capture (CDC) with Estuary. Learn how to capture, ingest, and materialize data using Estuary's seamless integration. This demo uses a sales database to showcase how changes in a PostgreSQL table are tracked and replicated into an Iceberg table stored in AWS S3. Check out Estuary's Iceberg integration: https://estuary.dev/destination/s3-iceberg/ Join Estuary's community Slack: https://estuary-dev.slack.com/join/shared_invite/zt-86nal6yr-VPbv~YfZE9Q~6Zl~gmZdFQ#/shared-invite/email 00:00 - Introduction: Streaming Data from Postgres to Iceberg 00:18 - Postgres Sales Database Overview 01:08 - Starting Change Data Capture (CDC) with Estuary 02:09 - Materializing Data into Apache Iceberg 04:17 - Backfilling Data into Iceberg 05:21 - Querying Iceberg Tables with Python 06:10 - Conclusion: Demo Recap
Seamless Data Integration, Unlimited Potential
Discover the simplest way to connect and move your data.Get hands-on for free, or schedule a demo to see the possibilities for your team.