How Prodege Reduced Costs by 60% with Estuary Flow and Apache Iceberg

Introduction

Prodege, a leading digital services company known for delivering innovative consumer insights and rewards-based solutions, has been at the forefront of leveraging data for a competitive edge. Handling significant data ingestion and transformation, Prodege needed a high-performance, scalable, and cost-effective data infrastructure to keep up with its growth and evolving data needs.

Their solution? Estuary Flow in combination with Apache Iceberg—a modern approach to data movement and optimizing storage costs.

Challenges

Prodege’s data platform provided a reliable and scalable foundation that supported their early data needs. However, as data volumes expanded, so did the need for a more cost-effective and flexible approach.

Managing rising ingestion costs while ensuring high performance became a priority, prompting Prodege to explore ways to evolve its architecture.

At the same time, Prodege’s transition to dbt for data transformations emphasized the importance of streamlined, version-controlled workflows that could support both real-time and batch processing. Recognizing these needs, Prodege aimed to build a scalable, open-architecture platform, allowing diverse data workflows, effective cost management, and readiness for future growth.

The Solution: Moving to Iceberg with Estuary Flow

Prodege embarked on a two-pronged migration:

1. Replication to Estuary Flow: Transitioning to Estuary Flow as the primary data replication tool, replacing existing replication solutions.
2. Transition to Iceberg with Starburst Galaxy: Migrating base tables to a data lake using Apache Iceberg from a centralized data warehouse, with Starburst Galaxy as the primary engine for running dbt models and transformations.

Estuary Flow's ability to support Apache Iceberg as a direct target allows Prodege to continuously load data to the Iceberg tables in their Amazon S3 lake, benefiting from Iceberg’s support for schema evolution and partitioning. Estuary Flow’s pipeline flexibility ensures that data can be processed in real-time, minimizing latency for downstream processing.

Why Estuary Flow?

Prodege chose Estuary Flow for its robust features tailored to enterprise needs, including:

• Cost Efficiency: Lightweight infrastructure and flexible pricing reduced replication costs by 60%.
• Enterprise-Grade Security: Private deployments within Prodege’s VPC ensure stringent data governance and compliance.
• Native Apache Iceberg Support: Flow simplifies real-time loading to Iceberg tables with schema evolution and partitioning.
• Future-Proof Architecture: Integrates with popular tools in the data ecosystem, such as dbt and Starburst Galaxy, ensuring long-term adaptability.

Estuary Flow empowers Prodege with a cost-effective, secure, scalable platform that bridges real-time innovation with enterprise-level reliability.

Cost Savings and Operational Efficiency

With Estuary Flow, Prodege has been able to cut ingestion costs dramatically:

• 60% Cost Savings in Replication: Estuary Flow’s flexible pricing and lightweight infrastructure have reduced replication costs by up to 60% compared to previous monthly active rows-based pricing.
• 30% Reduction in Snowflake Ingestion Costs: By moving base tables to Iceberg tables in S3, Prodege saves an estimated 30% on Snowflake ingestion expenses. This is not only due to lower storage costs but also because of reduced reliance on Snowflake for staging and ingestion.

The integration of Iceberg has also reduced the operational burden associated with managing historical data versions and schema changes. Iceberg’s advanced capabilities, such as schema evolution, align with Prodege's goals of creating a flexible, long-term storage solution that can adapt as their data requirements grow.

Accelerated Data Transformation with dbt and Starburst Galaxy

In addition to optimizing data movement, Prodege was also keen on enhancing its transformation layer. By shifting their business logic implementation from simple batch SQL to dbt, Prodege is building modular, version-controlled data processing workflows that are easy to maintain.

Starburst Galaxy, running directly on top of the Iceberg data lake, pairs perfectly with dbt, providing a highly scalable, efficient execution engine that benefits from Iceberg’s structure and metadata management capabilities.

Future-Ready Data Architecture

Prodege’s transformation with Estuary Flow and Iceberg has paved the way for an agile, scalable data infrastructure that enables them to:

• Reduce Total Cost of Ownership (TCO): Through cost-effective storage, lower ingestion costs, and streamlined processing.
• Achieve Unified Real-Time and Batch Data Integration: With Estuary Flow, Prodege can ingest and transform data in real time while supporting batch processing for dbt-based transformations.
• Ensure Data Lake Scalability: Iceberg on S3 provides Prodege with a high-performance data lake capable of handling growing data volumes and complex schema requirements.

Conclusion

Prodege’s partnership with Estuary Flow highlights the potential of combining innovative data integration with modern lakehouse architecture. Together with Apache Iceberg, Estuary Flow has enabled Prodege to reduce costs significantly, simplify operations, and scale effortlessly, supporting a future of data-driven decision-making across their organization.

By implementing Estuary Flow and embracing a lakehouse approach with Apache Iceberg, Prodege has built a resilient, cost-efficient data pipeline primed to grow with their business needs.