S3 Tables Support for The Iceberg Connector is Now Available

Product Update: Estuary Flow Now Supports S3 Tables with Apache Iceberg

We're excited to announce that Estuary Flow's Iceberg Connector now supports AWS S3 Tables, significantly enhancing your ability to manage data lakes and optimize data operations directly in your S3 buckets.

What's New?

Estuary Flow users can now materialize data seamlessly into S3 Table Buckets via Apache Iceberg. Whether your S3 Tables are integrated with AWS Glue Data Catalog or directly managed through the S3 Tables Iceberg REST Endpoint, Estuary Flow simplifies real-time and batch data integration.

Key Features

Flexible Connectivity: Materialize your data directly to the S3 Tables Iceberg REST Endpoint or via AWS Glue Data Catalog, leveraging AWS’s scalable infrastructure.

Secure Access: Estuary supports AWS SigV4 authentication, ensuring secure connections using AWS IAM credentials.

Seamless Configuration: Easy setup using a simple REST endpoint and table bucket ARN configuration, with clear IAM policy examples provided for quick permissions setup.

How It Works

To connect directly to the S3 Tables Iceberg REST Endpoint:

Use the Base URL format: https://s3tables.<region>.amazonaws.com/iceberg, replacing <region> with your bucket's AWS region.

Specify the Warehouse with your bucket ARN in this format: arn:aws:s3tables:<region>:<accountID>:bucket/<bucketname>.

Authenticate securely with AWS SigV4 Authentication and your IAM user's AWS Access Key ID and Secret Access Key.

Permissions Simplified

Ensure your IAM user and EMR Execution Role have the appropriate permissions. Below is a policy example that you can use directly or customize as needed:

plaintext
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "S3TablesAccess",
            "Effect": "Allow",
            "Action": [
                "s3tables:GetTableBucket",
                "s3tables:ListNamespaces",
                "s3tables:CreateNamespace",
                "s3tables:ListTables",
                "s3tables:CreateTable",
                "s3tables:PutTableData",
                "s3tables:GetTableData",
                "s3tables:UpdateTableMetadataLocation",
                "s3tables:GetTableMetadataLocation",
                "s3tables:DeleteTable"
            ],
            "Resource": [
                "arn:aws:s3tables:<region>:<aws-account-id>:bucket/<bucket>",
                "arn:aws:s3tables:<region>:<aws-account-id>:bucket/<bucket>/*"
            ]
        }
    ]
}

Benefits

Simplified and cost-effective data lake management.

Accelerated real-time analytics and batch processing capabilities.

Improved governance and security through native AWS integration.

Start leveraging S3 Tables support with Apache Iceberg today for a more efficient, secure, and streamlined data integration experience with Estuary Flow.

Learn more

Check out the docs page for more information.