Snowflake is a powerful platform for modern analytics, but Snowflake ingestion costs can quickly add up. Processing large data volumes with high velocities requires data engineers to stage files and run COPY commands that require virtual warehouses, which all contribute to cost consumption. In addition to high cost, the data that lands in Snowflake tables often lags in freshness.
Estuary offers a different approach. By utilizing Snowflake’s streaming API (the foundation of Snowpipe Streaming) under the hood, it takes any data source, including files, databases, or APIs, and turns it into a continuous stream. The result is lower Snowflake ingestion cost and fresher data, without the complexity of implementing streaming functionality.
In this article, we’ll explore why most ETL tools increase Snowflake ingestion costs, how Snowpipe Streaming helps reduce them, and how Estuary enables real-time, cost-efficient analytics without added complexity.
Why Most ETL Tools Drive Up Snowflake Ingestion Costs
Most ETL tools ingest data in batch or micro-batch mode, which adds overhead and ultimately increases Snowflake ingestion costs. Even tools that support streaming typically ingest data from inherently streaming sources such as Kafka or Kinesis, but not from sources that most organizations use on a daily basis, including databases, files, or APIs.
The reason ETL tools are designed that way is that they reflect how Snowflake positions the Snowpipe functionality, intended for continuous data ingestion. There are two options:
- Snowpipe is designed for continuous ingestion of files in a stage. When new files arrive, Snowpipe executes the COPY command, which requires a virtual warehouse to execute.
- Snowpipe Streaming is intended for row-level streaming. Data is ingested into Snowflake tables as individual rows instead of being staged in files. In this scenario, data must already be in a streaming format, such as a Kafka topic or a custom Java or Scala application built using the Snowpipe Streaming SDK.
The diagram below illustrates the various Snowpipe-based ingestion methods in Snowflake, depending on the source:
- With Kafka, you can use Snowpipe Streaming with the Kafka Connector
- With other streaming platforms, you can use Snowpipe Streaming, but you must develop your own ingestion using the client SDK
- With any other source, such as databases, files, or APIs, you can use the regular Snowpipe which executes the COPY command and adds extra cost overhead.
Snowflake ingestion methods: Snowpipe for files, Snowpipe Streaming with Kafka connector or client SDK for streaming sources.
As a result of this Snowflake architecture, data pipelines that ingest sources that are not streaming in nature are implemented as batch or micro-batch. Although the micro-batch approach with Snowpipe has low latency because it processes each file as soon as it is available in a stage, it still does not provide up-to-date data as with streaming ingestion where each row is ingested immediately.
Enter Estuary Flow: Stream Data Into Snowflake From Any Source
This is where Estuary differs from most ETL tools. Instead of batch ingestion for non-streaming data sources, it gives you the option to choose streaming ingestion for any source, including files in object storage, databases, APIs, and others.
Even when you choose streaming ingestion, Estuary Flow doesn’t require you to have a running Kafka cluster or any streaming infrastructure, nor do you have to write any custom applications using the Snowpipe Streaming SDK. All that complexity is abstracted and handled by Estuary under the hood.
This means that setting up data pipelines in streaming mode is easy and straightforward. You only configure the source and the target, choose the streaming option, and Estuary Flow does the rest. The result is cost-efficient, true row-level streaming ingestion from any source, either streaming in nature or not, all without complexity or engineering overhead.
Estuary Flow streams data from any source into Snowflake using Snowpipe Streaming without custom SDKs or infrastructure.
Ready to cut your Snowflake bill and unlock real-time analytics? Start streaming with Estuary Flow today
How Snowpipe Streaming Lowers Snowflake Ingestion Cost
Snowpipe Streaming is Snowflake’s API for continuous, row-level ingestion. Unlike file-based Snowpipe, it ingests rows of data directly into Snowflake tables without staging them first in files. It also doesn’t require a virtual warehouse. The result is lower Snowflake ingestion cost, reduced latency, simpler data engineering pipelines, and charges only for the serverless compute utilized in streaming ingestion instead of virtual warehouse compute.
Snowflake itself has benchmarked the difference in cost. When compared to file-based Snowpipe ingestion, Snowpipe Streaming was at least 50% less expensive, as documented in Best Practices for Data Ingestion with Snowflake: Part 3, where they say:
Snowpipe Streaming is a more cost-effective data ingestion method compared to Snowpipe, specifically for offering high-throughput, low-latency streaming data ingestion based on our benchmarks at a low cost.
Estuary Flow takes advantage of these cost savings. By eliminating batch overhead such as staging data in files, and by utilizing serverless compute resources instead of virtual warehouses, Estuary Flow customers have seen ingestion costs lower by as much as 70%.
The benefits are not just in Snowflake cost savings. With Estuary, the complexity of building and maintaining the streaming logic in data pipelines is implemented by the tool, not by the data engineers. This means that pipelines are less complex and easier to operate and maintain. At the same time, data in Snowflake tables is fresher.
Real-Time Analytics in Snowflake Without High Ingestion Cost
Many organizations accept batch ingestion because it feels “good enough.” But batch pipelines inevitably mean stale data and longer time to decisions. With Estuary Flow and Snowpipe Streaming, there is no compromise in analytics. Business users can see dashboards with the freshest metrics. Operational use cases like fraud detection, anomaly alerts, and IoT monitoring can run directly on live data in Snowflake. The cost savings and performance benefits go hand in hand, and the result is analytics that are both faster and cost less.
Estuary Flow implemented with Snowpipe Streaming can be efficiently paired with Snowflake Dynamic Tables, which enable near real-time analytics. Dynamic tables are automatically refreshed in Snowflake based on target freshness which simplifies data transformation and pipeline management without requiring manual updates or custom scheduling.
Since dynamic tables support incremental processing by updating only the data that has changed, this creates a highly efficient incremental processing pipeline: only the rows that change are streamed in and updated downstream. That means faster queries, lower costs, and analytics in near real time.
For example, a retail company that tracks inventory across thousands of stores doesn’t have to reload the entire inventory table every few minutes. Instead, Estuary Flow streams only the new rows that represent new sales, stock replenishments, or customer returns. Snowflake Dynamic Tables used in dashboards are then incrementally updated so that store managers and supply chain teams have a live view of product availability.
The takeaway is that real-time analytics doesn’t have to come with trade-offs. Estuary Flow ensures lower cost, fresher data, and simpler data pipelines to enable real-time analytics.
From IoT Sensors to Real-Time Decisions
A common scenario is IoT devices that generate data continuously, for example sensors in a manufacturing plant that save data as JSON files in cloud object storage. With traditional ETL tools, these files might only be ingested every 10 minutes, which sounds close to real time. However, it doesn’t ensure near-real time data for downstream consumers and can increase Snowflake costs due to the overhead of batch ingestion.
With Estuary Flow, the same sensor data can be ingested as a continuous data stream. Instead of waiting for files to accumulate, the data pipeline can read each file and ingest the data as individual rows that are saved directly in Snowflake tables via Snowpipe Streaming.
The benefits are fresher insights because the data teams can monitor nearly live sensor data instead of snapshots from several minutes ago, helping them to identify any potential disruptions in the manufacturing process as soon as they start to appear. At the same time, Snowflake cost is lower due to utilizing serverless compute instead of staging files and virtual warehouses.
Whether it’s monitoring manufacturing processes, detecting anomalies in smart meters, or tracking vehicle telemetry, organizations can take advantage of real-time decision-making while cutting ingestion costs at the same time.
Want to talk through your scenario? Contact our team
No More Choosing Between Cost and Speed
Traditionally, data teams have had to choose between incurring higher Snowflake costs for fresher data, or saving money by accepting stale analytics. With Estuary Flow, there is no more trade-off.
By making Snowpipe Streaming accessible for any source, regardless whether it’s a true streaming source like Kafka or a file-based source or relational database, Estuary Flow enables streaming row-level ingestion without engineering overhead. This can result in up to 70% lower ingestion costs compared to batch pipelines. It means simpler data pipelines with less operational overhead. And it ensures real-time analytics for dashboards and other mission-critical applications.
The possibilities where real-time data makes a difference include real-time recommendation engines that react to the latest clicks and purchases, live fraud detection that blocks suspicious transactions before they complete, or continuous monitoring of connected devices and equipment in real-time.
With Estuary Flow, you don’t have to choose between cost and speed. You get both, without the complexity.
Talk to our team about your pipelines — Share your current Snowflake setup and we’ll show you where streaming can cut costs.
About the author
Maja is a seasoned data architect with more than 30 years of experience in data analytics, data warehousing, business intelligence, data engineering, and data modeling. She has written extensively on these topics, helping practitioners and organizations take control of their data. Most recently, she authored Snowflake Data Engineering, where she shares practical insights into building and managing data engineering pipelines in modern cloud-based data platforms.
