Hevo Data VS Meltano

Estuary was founded in 2019. But the core technology, the Gazette open source project, has been evolving for a decade within the Ad Tech space, which is where many other real-time data technologies have started.

Estuary Flow is the only real-time and ETL data pipeline vendor in this comparison. There are some other ETL and real-time vendors in the honorable mention section, but those are not as viable a replacement for Fivetran. Estuary Flow is also a great option for batch sources and targets.

Where Estuary Flow really shines is in any combination of change data capture (CDC), real-time and batch ETL or ELT, and loading multiple destinations with the same pipeline. Estuary Flow currently is the only vendor to offer a private cloud deployment, which is the combination of a dedicated data plane deployed in a private customer account that is managed as SaaS by a shared control plane. It combines the security and dedicated compute of on-prem with the simplicity of SaaS.

CDC works by reading record changes written to the write-ahead log (WAL) that records each record change exactly once as part of each database transaction. It is the easiest, lowest latency, and lowest-load for extracting all changes, including deletes, which otherwise are not captured by default from sources. Unfortunately ELT vendors like Airbyte, Fivetran, Meltano, and Hevo all rely on batch mode for CDC. This puts a load on a CDC source by requiring the write-ahead log to hold onto older data. This is not the intended use of CDC and can put a source in distress, or lead to failures.

Estuary Flow has a unique architecture where it streams and stores streaming or batch data as collections of data, which are transactionally guaranteed to deliver exactly once from each source to the target. With CDC it means any (record) change is immediately captured once for multiple targets or later use. Estuary Flow uses collections for transactional guarantees and for later backfilling, restreaming, transforms, or other compute. The result is the lowest load and latency for any source, and the ability to reuse the same data for multiple real-time or batch targets across analytics, apps, and AI, or for other workloads such as stream processing, or monitoring and alerting.

Estuary Flow also has broad packaged and custom connectivity, making it one of the top ETL tools. It has 150+ native connectors that are built for low latency and/or scale. While this number may seem low, these are high-quality, standardized connectors. In addition, Estuary is the only vendor to support Airbyte, Meltano, and Stitch connectors, which easily adds 500+ more connectors. Getting official support for the connector is a quick “request-and-test” with Estuary to make sure it supports the use case in production. Most of these connectors are not as scalable as Estuary-native, Fivetran, or some ETL connectors, so it’s important to confirm they will work for you. Flow’s support for TypeScript and SQL transformations also enables ETL.

• Modern data pipeline: Estuary Flow has the best support for schema drift, evolution, and automation, as well as modern DataOps.
• Modern transforms: Flow is also both low-code and code-friendly with support for SQL and TypeScript (with Python on the way) for ETL, and dbt for ELT.
• Lowest latency: Several ETL vendors support low latency. But of these Estuary can achieve the lowest, with sub-100ms latency. ELT vendors generally are batch only. 
• High scale: Unlike most ELT vendors, leading ETL vendors do scale. Estuary is proven to scale with one production pipeline moving 7GB+/sec at sub-second latency.
• Most efficient: Estuary alone has the fastest and most efficient CDC connectors. It is also the only vendor to enable exactly-and-only-once capture, which puts the least load on a system, especially when you’re supporting multiple destinations including a data warehouse, high performance analytics database, and AI engine or vector database.
• Deployment options: Of the ETL and ELT vendors, Estuary is currently the only vendor to offer open source, private cloud, and public multi-tenant SaaS.
• Reliability: Estuary’s exactly-once transactional delivery and durable stream storage makes it very reliable.
• Ease of use: Estuary is one of the easiest to use tools. Most customers are able to get their first pipelines running in hours and generally improve productivity 4x over time. 
• Lowest cost: For data at any volume, Estuary is the clear low-cost winner in this evaluation. Rivery is second.
• Great support: Customers consistently cite great support as one of the reasons for adopting Estuary.

• On premises connectors: Estuary has 150+ native connectors and supports 500+ Airbyte, Meltano, and Stitch open source connectors. But if you need on-premises app or data warehouse connectivity, make sure you have all the connectivity you need.
• Graphical ETL: Estuary has been more focused on SQL and dbt than graphical transformations. While it does infer data types and convert between sources and targets, there is currently no graphical transformation UI.

Of the various ELT and ETL vendors, Estuary is the lowest total cost option. Estuary only charges $0.50 per GB of data moved from each source or to each target, and $100 per connector per month. Rivery, the next lowest cost option, is the only other vendor that publishes pricing of 1 RPU per 100MB, which is $7.50 to $12.50 per GB depending on the plan you choose. Estuary becomes the lowest cost option by the time you reach the 10s of GB/month. By the time you reach 1TB a month Estuary is 10x lower cost than the rest.

Hevo is a cloud-based ETL/ELT service for building data pipelines that, unlike Fivetran, started as a cloud service in 2017, making it more mature than Airbyte. Like Fivetran, Hevo is designed for “low code”, though it does provide a little more control to map sources to targets, or add simple transformations using Python scripts or a new drag-and-drop editor in ETL mode. Stateful transformations such as joins or aggregations, like Fivetran, should be done using ELT with SQL or dbt.

While Hevo is a good option for someone getting started with ELT, as one user put it, “Hevo has its limits”.

• Ease of use: Like several other modern ELT tools, Hevo is intuitive and easy to use, especially compared to traditional ETL tools. 
• ELT and ETL: Hevo has started to add ETL support including Python scripts and a new drag-and-drop editor. This is limited mostly to row-level transformations. Hevo’s main transformation support is dbt (ELT).
• Reverse ETL: Hevo supports the ability to insert source data back into the source once it’s been cleansed. This might be good for you if you’re looking for this feature. It is a very specific use case where you write modified data back directly into the source. A more general-purpose solution is to have a pipeline write back to the sources, which is not supported by most modern ETL/ELT vendors. It is supported by iPaaS vendors.

• Connectivity: Hevo has one of the lowest number of connectors at slightly over 150. You should consider what sources and destinations you need for your current and future projects to make sure it will support your needs. 
• Latency: Hevo is still mostly batch-based connectors on a streaming Kafka backbone. While data is converted into “events” that are streamed, and streams can be processed if scripts are written for any basic row-level transforms, Hevo connectors to sources, even when CDC is used, is batch. There are starting to be a few exceptions. For example, you can use the streaming API in BigQuery, not just the Google Cloud Storage staging area. But you still have a 5 minute or more delay at the source. Also, there is currently no common  scheduler. Each source and target frequency is different. So latency can be longer than the source or target when they operate at different intervals.  
• Costs: Hevo can be comparable to Estuary for low data volumes in the low GBs per month. But it becomes more expensive than Estuary and Airbyte as you reach 10s of GBs a month. Costs will also be much more as you lower latency because several Hevo connectors do not fully support incremental extraction. As you reduce your extract interval you capture more events multiple times, which can make costs soar.
• Reliability: CDC is batch mode only, with the minimum interval being 5 minutes. This can load the source and even cause failures. Customers have complained about Hevo bugs that make it into production and cause downtime.
• Scalability: Hevo has several limitations around scale. Some are adjustable. For example, you can get the 50MB Excel, and 5GB CSV/TSV file limits increased by contacting support. 
  But most limitations are not adjustable, like column limits. MongoDB can hit limits more often than others. A standalone MongoDB instance without replicas is not supported. You need 72 hours or more of OpsLog retention. And there is a 4090 columns limit that is more easily hit with MongoDB documents. 
  There are ingestion limits that cause issues, like a 25 million row limit per table on initial ingestion. In addition there are scheduling limits that customers hit, like not being able to have more than 24 custom times.
  For API calls, you cannot make more than 100 API calls per minute.
• DataOps: Like Airbyte, Hevo is not a great option for those trying to automate data pipelines. There is no CLI or “as code” automation support with Hevo. You can map to a destination table manually, which can help. But while there is some built-in schema evolution that happens when you turn on auto mapping, you cannot fully automate schema evolution or control the rules. There is no schema testing or evolution control. New tables can be passed through, but many column changes can lead to data not getting loaded in destinations and moved to a failed events table that must be fixed within 30 days or the data is permanently lost. Hevo used to support a concept of internal workflows, but it has been discontinued for new users. You cannot modify folder names for the same “events”. 

Hevo is more expensive than Airbyte and Estuary, but still less expensive than Fivetran and various ETL vendors.

• Free: Limited to 1 million free events per month with free initial load, 50+ connectors, and unlimited models
• Starter ($239/mo for 5M rows): Offers 150+ connectors, on-demand events, and 12 hours of support as an SLA. Additional rows are $10 or more per million (~1GB)
• Business (Custom Pricing): HIPAA compliance with a dedicated data architect and dedicated account manager

Meltano was founded in 2018 as an open source project within GitLab to support their data and analytics team. It’s a Python framework built on the Singer protocol. The Singer framework was originally created by the founders of Stitch, but their contribution slowly declined following the acquisition of Stitch by Talend (which in turn was later acquired by Qlik).

Meltano is focused on configuration-based ELT using YAML and the CLI.

• Open source ELT: Meltano is the main successor to Stitch if you’re looking for a Singer-based framework.
• Configuration-driven: If you are looking for a configure-driven approach to ELT, Meltano may be a great option for you.
• Connectivity: Meltano and Airbyte collectively have the most connectors, which makes sense given their open source history with Singer. Meltano supports Singer and has an SDK wrapper for Airbyte, giving it 600+ open source connectors in total. Open source connectors have their limits, so it’s important to test out carefully based on your needs.

• Not low-code: If you’re looking for a more graphical, low-code approach to integration, Meltano is not a good choice.
• Latency: Meltano is batch-only. It does not support streaming. While you can reduce polling intervals down to seconds, there is no staging area. The extract and load intervals need to be the same. Meltano is best suited for supporting historical analytics for this reason.
• Reliability: Some will say Meltano has less issues when compared to Airybte. But it is open source. Connectors may not be maintained and if you have issues you can only rely on the open source community for support.
• Scalability: There isn’t as much documentation to help with scaling Meltano, and it’s not generally known for scalability, especially if you need low latency. Various benchmarks show that larger batch sizes deliver much better throughput. But it’s still not the level of throughput of Estuary or Fivetran. It’s generally minutes even in batch mode for 100K rows.
• ELT only: Meltano supports open source dbt and can import existing dbt projects. Its support for dbt is considered good. It also has the ability to extract data from dbt cloud. Meltano does not support ETL.
• Deployment options: Meltano is deployed as self-hosted open source. There is no Meltano Cloud, though Arch is offering a broader service with consulting.
• DataOps: Data engineers generally automate using the CLI or the Meltano API. While it is straightforward to automate pipelines, there isn’t much support for schema evolution and automating responses to schema changes.

Meltano is open source. There is no pricing. But it’s not really free. You’ll need to spend more on data engineering resources to stand up, build, and maintain Meltano. If you need scalability, there isn’t a lot of documentation on how to scale. Make sure you evaluate carefully and find some Meltano expertise.