Announcing Dekaf: Kafka-API Compatibility for Estuary Flow

Apache Kafka is great, but it’s hard to manage. Imagine if you could get all the benefits of Kafka from a SaaS product with the simplicity of ELT.

We’re excited to introduce Dekaf, Estuary Flow's latest feature that brings Kafka API compatibility to our real-time data streaming platform. With Dekaf, you can now connect to Estuary Flow as if it were Kafka, enabling integration with the ecosystem of tools and services that rely on Kafka for real-time data processing.

Why Dekaf Matters

As the demand for real-time data grows, so does the need for versatile and robust data streaming solutions. Kafka has become a cornerstone for many organizations, providing a scalable and reliable way to manage real-time data streams. By introducing Kafka API compatibility, Estuary Flow expands its connectivity, allowing users to leverage existing Kafka-based infrastructures and tools with minimal friction.

With Dekaf, we’re not just adding a new feature; we’re unlocking a world of possibilities. Here’s what this means for you:

• Integrations, Integrations, and Integrations: Whether you're using data platforms like Tinybird, SingleStore, ClickHouse, or other services that can consume data from Kafka, Dekaf allows you to tap into Estuary Flow's real-time data streams with ease.
• Enhanced Flexibility: Dekaf bridges the gap between Estuary Flow collections and Kafka topics, making it easier than ever to incorporate new real-time data sources into your existing workflows. Any vendor capable of consuming Kafka topics can now access Estuary Flow's rich, real-time datasets.
• Schema Management Simplified: Alongside Kafka topic emulation, Dekaf also provides a schema registry API, allowing you to manage and retrieve schemas just as you would with Confluent’s Schema Registry. This means less hassle and more consistency in how you handle your data.

How It Works

Dekaf allows you to connect to Estuary Flow using familiar Kafka APIs. Whether you’re configuring your Kafka client or using tools like kcat, Dekaf makes the process straightforward. Simply plug in your Estuary Flow credentials, and you’re ready to start consuming real-time data from Estuary Flow collections as if they were Kafka topics.

We’ve designed Dekaf to be intuitive and easy to integrate, ensuring that your focus remains on building impactful real-time applications, not wrestling with infrastructure and configuration issues.

Here’s a simple example in Python:

python
from kafka import KafkaConsumer

# Create Consumer instance
consumer = KafkaConsumer(
    'Estuary_Collection_Name',
    bootstrap_servers='dekaf.estuary.dev:9092',
    security_protocol='SASL_SSL',
    sasl_mechanism='PLAIN',
    sasl_plain_username='{}',
    sasl_plain_password='Estuary_Refresh_Token',
)

# Poll for messages
for collection in consumer:
    print(f"Received message: {collection.value}")

Check out the documentation page for more details.

Launch Partners

Dekaf enables many integrations in the real-time and streaming ecosystem. Dekaf is compatible with any Kafka consumer, but we made sure to validate integrations with some of our friends in the real-time space. 

We’re proud to launch with the following partners: Tinybird, SingleStore, ClickHouse, StarTree, Materialize, and Bytewax. 

But wait.. there’s more!

Dekaf also opens to door to the Kafka Ecosystem at large. You’ll be able to load data in real-time into services such as Apache Druid, StarRocks, Kafka Connect, MirrorMaker2, Cassandra, and many more!

A New Era of Real-Time Data Streaming

Dekaf represents a significant leap forward for Estuary Flow and our users. By enabling Kafka API connectivity, we’re opening doors to countless new integrations and use cases. Whether you’re a system integrator, a data engineer, or a vendor looking to expand your real-time capabilities, Dekaf empowers you to do more with Estuary Flow.

We invite you to explore the possibilities that Dekaf brings to your data infrastructure. Start integrating today and see how Estuary Flow can supercharge your real-time data processing needs.

For detailed documentation and a step-by-step guide on getting started with Dekaf, visit our documentation page. If you have any questions or need support, our team is here to help.

Join us as we continue to push the boundaries of real-time data streaming with Estuary Flow!