How Forward reduced real-time analytics costs by 50% with Estuary

Overview

Forward, an innovative fintech company specializing in embedded payment solutions, found itself at a pivotal moment when its primary operational analytics provider, Rockset, announced it would no longer be available for public use. In urgent need of a robust replacement, Forward required a solution that could integrate data from multiple sources—such as DynamoDB, MySQL, and Snowflake—while supporting modern analytics tools like MotherDuck, BigQuery for AI, and PostgreSQL (AWS Aurora).

Forward's complex ecosystem needed data movement, real-time transformations, many-to-many data routing, and cost-effective scalability to support real-time dashboards exposed to users.

"Estuary’s collections as a portable semantic layer enables seamless adaptation to the best tools for any workload. It powers real-time pipelines across our operational, analytics, and AI data stores. With Kafka materialization, we unify CDC sources, extending our platform’s flexibility into the broader Kafka ecosystem."
 

Alex Mays, Principal Engineer, Forward

The Challenge

Replacing Rockset Efficiently

Rockset’s deprecation forced Forward to find an alternative quickly that could match or surpass Rockset’s real-time capabilities without incurring excessive costs. Traditional ETL tools like Fivetran were evaluated but proved too expensive, while Hevo lacked the transformation capabilities Forward needed.

Integration Complexity

Forward's data comes from various sources, including DynamoDB, MySQL, and Snowflake. 

Rockset provided the real-time ingestion pipelines as well as the data warehouse. In order to quickly evaluate new warehouse solutions, they first needed to decouple ingestion from the underlying warehouse. Engineering a bespoke ingestion pipeline that could combine disparate datasets and materialize them in multiple target warehouses posed a risk of overrunning the Rockset deadline due to the high degree of complexity and overhead required.

One significant hurdle was the use of a single table design in DynamoDB - where 30 different entities coexist in a single table with distinct column sets. Traditional ingestion methodologies and several vendor solutions struggled to transform these entity records into normalized tables. 

Performance Bottlenecks

Running complex analytical queries directly on PostgreSQL (AWS Aurora) introduced delays and resource contention. PostgreSQL struggled with heavy aggregation queries and lacked support for continuous materializations, requiring multiple derivations to optimize performance.

Real-Time Analytics Demand

Forward needed real-time analytics for operational dashboards, partner APIs, and user-facing analytics. Delays or data inconsistencies could disrupt client experiences and business operations.

Why Estuary Flow?

Forward chose Estuary Flow to power its operational analytics for several strategic reasons:

Cost Efficiency

Estuary Flow's lean infrastructure and transparent pricing reduced Forward's real-time analytics costs by over 50% per month compared to Rockset. This cost reduction freed up the budget for other growth initiatives.

Real-Time Transformations

Unlike traditional ETL tools, Estuary Flow enables complex real-time transformations. Forward can move joins and data derivations upstream from PostgreSQL, significantly improving query performance and reducing the load on operational databases.

Many-to-Many Connectivity

Flow’s architecture supports many-to-many data routing, allowing Forward to seamlessly stream data from sources like DynamoDB,MySQL, and Snowflake into multiple destinations, including PostgreSQL, MotherDuck for advanced analytics, and BigQuery for AI workloads.

The Results

• 50% Reduction in Data Movement Costs: Forward cut its data movement costs in half after transitioning from Rockset to Estuary Flow.
• Simplified Data Integration: Native connectors eliminated the need for complex, custom-built pipelines, streamlining data workflows.
• Improved Query Performance: By shifting joins and transformations upstream to Estuary, PostgreSQL was optimized for speed layer workloads.
• Real-Time API Enablement: Real-time data pipelines empowered Forward to expose APIs for users, enhancing their embedded analytics offerings.
• Modular, Future-Ready Data Stack: With Estuary, Forward can seamlessly shift workloads to new destinations like MotherDuck or Iceberg as business needs evolve.

Conclusion

Forward’s adoption of Estuary Flow has transformed its real-time analytics landscape. By replacing Rockset, Forward achieved a 50% reduction in operational analytics costs while significantly simplifying data integration and boosting query performance. With MotherDuck managing heavy aggregation workloads, BigQuery supporting AI initiatives, and PostgreSQL optimized for last-mile operational analytics, Forward is poised for continued growth.

Estuary Flow's flexibility and modular architecture lay the foundation for Forward’s future: a scalable, cost-effective platform that supports the company’s mission to deliver cutting-edge payment solutions with maximum efficiency and minimal complexity.