From SAP on SQL Server to Snowflake: How Coltene Built Reliable Replication with Estuary

Meet the Coltene Group

A developer and manufacturer of a range of dental products, Coltene is dedicated to helping dental professionals and the patients they serve.

The international company employs thousands of professionals across Europe and North America. With worldwide distribution to all major markets, you may have already had firsthand experience with their products.

To manufacture such precise products and distribute them so widely, Coltene needed a top-of-the-line data stack, which is why they started looking at Snowflake as their central data solution. Snowflake would be a perfect platform to:

• Integrate AI features
• Perform data analysis
• Share data with partners

The only blocker? Reliably transferring their SAP system data into Snowflake.

Researching the Market

Alexander Grutsch, Director of IT at the Coltene Group, began an involved evaluation process, reviewing a number of data movement solutions. Many of the platforms struggled, and none were perfectly suited to Coltene's requirements.

Coltene needed to find a solution that fit all their criteria:

• Could the solution connect to an on-premises hosted system?
• Could it process Coltene’s huge number of SAP tables?
• And could it do so in a reasonably prompt manner?
• How helpful was the support team?
• And, of course, how much would the solution cost?

After trialing platforms that choked on or stalled Coltene’s data, Alexander found himself reviewing lists of data movement platforms once more. One mentioned Estuary, and after a cursory review of the connector options and pricing, he decided to try out the platform and see how it handled SAP data.

SAP System Setup with Estuary

To test out the pipeline, Alexander needed to get Coltene’s SAP-system-formatted data into Estuary and then to Snowflake. Comprising 120,000 SQL Server tables, the SAP system had its own unique integration challenges.

The first challenge was relatively simple. Estuary’s auto discovery was having trouble accessing system schema information since it expected lowercase names. Estuary support quickly identified that Alexander could add the uppercase_discovery_queries feature flag to the capture to solve the problem.

With that minor hurdle out of the way, Alexander could focus on the main obstacle: extracting a select set of useful data from a sea of 100,000 tables.

To limit the number of returned tables, he configured 100 tables that he wanted replicated for Change Data Capture. He figured this would pair well with Estuary’s “Discover Only CDC-Enabled Tables” option in the SQL Server capture configuration.

Instead of displaying the pared-down list of 100 tables, however, Estuary displayed an error. The auto discovery process failed.

The Estuary engineering team was called in to take a look. Will Donnelly, Senior Software Engineer, swiftly realized that “Discover Only CDC-Enabled Tables” didn’t actually pre-filter discovery results. Estuary was still trying to return the full list of 120,000 tables before filtering down to the 100 that supported CDC, resulting in out of memory errors.

Intent on optimizing behavior and providing a better user experience, Will immediately dug deeper. Within several hours, he already had a fix up for review. It would both flip connector behavior to apply filters first in discovery and streamline case-sensitive handling.

Alexander tested out a freshly-built connector image with the fixes in place and was able to successfully complete capture setup. SQL Server CDC started streaming into Estuary.

Completing the Pipeline with Snowflake

After the excitement with SQL Server capture for Coltene’s SAP system, setting up the rest of the pipeline was straightforward, especially since it was all going to one place: Snowflake.

The Snowflake materialization lined up perfectly with Coltene’s requirements, creating a 1:1 copy of their SAP tables in Snowflake. Other solutions had transformed JSON documents into Snowflake tables, racking up credit consumption. With Estuary, they could eliminate this unnecessary transformation step, keeping costs down.

Estuary’s delta updates also offered low latency by using Snowflake’s Snowpipe Streaming.

"Estuary’s materialization features match exactly what we need in our pipeline."

—Alexander Grutsch, Director of IT at Coltene

With pipeline setup and initial testing complete, Coltene continued with a full backfill of their SAP data into Snowflake.

Estuary routed 2 TB of data through the pipeline at a steady rate, completing the initial sync after a couple days. The pipeline then settled into the 350 GB per month of new data that Coltene expected.

Coltene had finally found an integration to support their Snowflake-powered data stack.

By the Numbers

With his methodical research and evaluation phase, Alexander had the chance to compare several leading data solutions and see how they stacked up. He had tried out a leading solution in the data movement space with a managed platform, a leading open-source solution, and Estuary.

Here’s what he found.

Sync times

While Estuary started out 2x slower than the leading competitor, this only lasted as long as the initial sync. Once tables were fully backfilled, Estuary’s delta updates beat out everyone else for load times.

• With Estuary’s unique Snowpipe Streaming integration, data can land in Snowflake in under a second, breaking into millisecond latency territory.
• In comparison, the lowest possible latency for the leading competitor is 1 minute.
• Estuary’s solution is thus an order of magnitude faster, making for truly real-time data.

As for the open-source platform, it couldn’t keep up at all with the SAP system data. After slowly chugging along, it stopped loading after a couple days.

Support response

Estuary cut wait times by at least half, with an initial response within an hour and a full solution from the engineering team within a day.

With the leading competitor, Coltene sometimes had to wait 2-3 days for a response.

Snowflake cost

With Estuary, Coltene’s daily consumption of Snowflake credits reduced by 50.

With the leading competitor, “we lost too much money on ingestion instead of data analytics.”

Overall cost

In total, Coltene’s data integration solution with Estuary cost 85% less than the leading competitor. Over a year, that would rack up to five figures of savings.

The Result: Dependable, Right-Time Data

Alexander’s research had uncovered faster data, lower cost, and reliable pipelines with Estuary. But a successful integration is more than the numbers. Overall, Coltene enjoyed several main benefits of working with Estuary:

Responsive support

Setting Estuary up to work with SAP data was more like a collaboration than a customer service relationship.

"This was a good chance to challenge the support of Estuary and this was a marvelous experience. It turned out to be a great collaboration."

—Alexander Grutsch, Director of IT at Coltene

The support team listened to Alexander’s remarks and ideas, and the engineering team quickly implemented them.

Integrations that just work

After the initial setup hiccups, Coltene’s integration from SAP to Snowflake proved its reliability.

"After the initial sync, data replication just worked seamlessly."

—Alexander Grutsch, Director of IT at Coltene

Coltene experienced some network issues of their own, including a planned update. Estuary took it all in stride, automatically picking back up where it had last left off for seamless data integration.

Reasonable costs, even beyond the Estuary platform

Besides Estuary’s intuitive usage-based pricing, Estuary helps lower costs in connected systems. With Snowflake, Estuary can do this in a few ways:

• Using low-cost Snowpipe Streaming with delta updates
• Configuring scheduled batches that only wake the warehouse when you need it
• Employing efficient merge queries to update tables

Because Estuary focuses on efficient integrations, Coltene cut their Snowflake ingestion costs and transformation credit consumption.

With all of these benefits powering a dependable data stack, Coltene's data team could focus back on the big picture: what they could do with Snowflake to prepare valuable insights and collaborate with partners, not just how they could get data into Snowflake.