Connect&GO lowers MySQL to Snowflake latency up to 180x, improves productivity 4x with Estuary Flow
Challenge
- Connect&GO empowers attractions with insights for better management. Historically this meant replicating 12 large MySQL databases to Snowflake using a self-hosted ELT solution that was batching in data.
- Clients were demanding faster data updates during peak season, while the data eng team struggled to resource the in-house implementation.
Solution
- Connect&GO implements Estuary to stream in change events from MySQL to Snowflake without standing up Kafka/Debezium, or having to settle for fully-managed batch ELT/ETL.
Results
- Up to 180x lower latency from 45 minutes down to 15 seconds.
- 4x greater productivity, resulting in more time spent on delivering new capabilities.
- 5x lower vendor costs.
About Connect&GO
location
Montreal, Canada
INDUSTRY
Attractions
GOALS
Re-establish a robust data pipeline to conform with their growing needs
Connect&GO is an all-in-one visitor and data management platform for the attractions industry. By replacing self-hosted batch-based ELT with Estuary Flow, Connect&GO is able to give their clients the near-real-time visibility they need to manage museums, amusement parks, and festivals minute by minute.
The Challenge
When you’re running an attraction for thousands of visitors, situations can change in minutes or seconds. Connect&GO’s clients needed reports and data feeds they can use to proactively direct resources and staff. Before finding Estuary, Connect&GO integrated its 12+ MySQL instances into Snowflake using a self-hosted ELT solution that often overloaded the small data engineering team and required them to use other teams' resources. More importantly, data was moved in batch every 45 minutes from MySQL to Snowflake, which was too big of a delay for their customers who wanted more real-time visibility.
The team knew they needed streaming data pipelines, but switching to a more complex self-hosted solution, like Kafka, would have been an even greater strain on tight resources. Meanwhile, the managed batch ELT services they considered — while an improvement, would add too much latency for Connect&GO’s customers.
All the other vendors out there are batch-based, so you'd have five-minute turnarounds for data refreshes. For us, finding something that was both pretty cost-effective [with latency] close to the second was very attractive.
Alexandre Pelletier, Senior Data Engineer, Connect&GO
The Solution
The team was searching for a fully managed solution for capturing and streaming in change data from MySQL to Snowflake. They required that the solution wouldn’t impact their production databases, and so preferred a platform capturing change data from a write-ahead log. And given the tight budget constraints most data teams have come under, they wanted to find something cost-effective and sustainably priced as they scale. During their search, they discovered Estuary. Estuary was the only solution they found that combined the ease of use of a managed ELT tool with true data streaming.
Connect&GO connected their MySQL databases through the Estuary UI and quickly saw their large existing DBs backfill within a couple hours on top of Estuary’s distributed stream processing framework. Once connected, Connect&GO was able to quickly set up a Snowflake destination and lock in their desired write schema for the data. Data typically landed from their MySQL DB to Snowflake within 15 seconds... the vast majority of this latency deriving from limitations on Snowflake’s ability to ingest. Beyond just a typical point-to-point ELT/ETL setup, Connect&GO’s team felt ready to cost-efficiently scale knowing that data ingested using Estuary lands in their data lake, allowing them to spin up a new fully-backfilled pipeline in minutes.
Using Estuary’s low-latency connectors, Connect&GO empowers customers to access to their most highly requested datasets in seconds. Through real-time reports, Connect&GO enabled Parks are driving higher revenue and optimized guest experience with real-time insight on capacity availability and foot traffic by access zone.