Webhooks to Kafka: How to Receive Events and Publish Them to Topics

Kafka has no endpoint a webhook provider can call, so something must receive each HTTP POST, check it and produce it to a topic. There are four ways to run that receiver: your own service or an HTTP gateway, Amazon API Gateway with AWS Lambda, a webhook source connector in Kafka Connect, or a managed pipeline that hosts the endpoint. A receiver service gives full control, Lambda suits AWS-native teams, a Connect source suits teams already running Connect workers, and a managed pipeline takes the endpoint off your hands.

The hard parts sit at the edge: verify signatures against the raw body, answer within the provider's deadline (five seconds for Shopify), and deduplicate, because providers deliver at least once. Sending Kafka events out as webhooks is the reverse job, covered at the end. Estuary is one managed option: its HTTP Webhook capture hosts the URL, checks a token or signature and answers once each request is stored, and its Kafka materialization produces the events to topics in your cluster.

Choosing a webhook to Kafka method

Kafka's own HTTP front ends don't replace the receiver. Confluent REST Proxy and the Strimzi HTTP Bridge accept only a Kafka-specific envelope ({"records":[…]}), and Confluent's HTTP Source connectors poll an API on an interval instead of listening, so a provider can't post its native payload to any of them.

How webhook to Kafka delivery works

Every method runs the same five steps between the provider's POST and a consumer:

1. Receive. An HTTPS endpoint accepts the POST, usually on one path per provider.
2. Verify. Compute the HMAC over the raw bytes and compare it with the signature header. Reject stale signed timestamps (Stripe's libraries default to a 5-minute tolerance).
3. Acknowledge. Produce to Kafka, wait for the broker's acknowledgement (acks=all), then return 2xx. If the write fails or nears the deadline, return 5xx so the provider retries.
4. Key and route. Choose the topic and use the provider's delivery ID as the message key.
5. Consume. Services in an event-driven architecture read the topic and skip deliveries they have already processed.

The guarantee is at least once. Kafka's idempotent producer, on by default in the Java client from Kafka 3.2 (and 3.0.1, 3.1.1), removes duplicates from the producer's own retries; a provider's redelivery is a new request and becomes a new message.

Webhook provider requirements

From each provider you need the signing secret, its signature scheme, its response deadline and the ID that identifies an event or delivery:

• Choose JSON delivery. GitHub's application/x-www-form-urlencoded option wraps the JSON in a form field called payload; pick application/json.
• Plan for size. GitHub caps payloads at 25 MB, far above Kafka's default 1 MB message limit.
• Check for bus delivery. Shopify can deliver to Amazon EventBridge or Google Cloud Pub/Sub, and Stripe to Amazon EventBridge or Azure Event Grid, instead of HTTPS.

Kafka requirements

• Topics with the partitions and retention you need, for example webhooks.stripe and webhooks.github. Partitions cap consumer parallelism.
• A producer principal with SASL credentials over TLS and write ACLs on those topics. Brokers older than 2.8.0 also need IDEMPOTENT_WRITE for the idempotent producer.
• A size decision. Webhook payloads meet two limits: message.max.bytes on the broker (default 1,048,588 bytes) and max.request.size in the Java producer (default 1,048,576). Raise the topic's max.message.bytes and the producer limit together, or store oversized payloads elsewhere and produce a pointer.
• A schema registry if consumers want Avro, Protobuf or JSON Schema instead of raw JSON.
• A Kafka client wherever the receiver runs. AWS documents no native HTTP produce endpoint for Amazon MSK, so even a Lambda receiver needs a producer library.

Setting up webhook to Kafka delivery with a receiver service

1. Create the topics, one per provider.
2. Deploy an HTTPS endpoint with one path per provider, behind a load balancer so deploys don't drop requests.
3. Verify each signature against the raw body, and reject stale timestamps where the provider signs one (Stripe does).
4. Produce the raw payload with the delivery ID as the key and the event type as a header.
5. Return 2xx only after Kafka acknowledges, within the provider's deadline; otherwise return 5xx.
6. Park payloads that pass verification but can't be produced (too large, malformed) in a dead letter topic or object storage, with the error.
7. Validate. Send a test event from the provider's dashboard, resend it, and confirm consumers process it once.

A sketch of steps 3 to 5 for Shopify, with the confluent-kafka Python client:

python
import base64, hashlib, hmac
from confluent_kafka import Producer

producer = Producer({"bootstrap.servers": BROKERS, "acks": "all",
                     "enable.idempotence": True})   # plus SASL and TLS settings

def handle_shopify(raw_body: bytes, headers) -> int:
    digest = hmac.new(SHOPIFY_CLIENT_SECRET, raw_body, hashlib.sha256).digest()
    if not hmac.compare_digest(base64.b64encode(digest).decode(),
                               headers.get("X-Shopify-Hmac-SHA256", "")):
        return 401                                   # checked on raw bytes, before parsing
    errors = []
    producer.produce("webhooks.shopify", value=raw_body,
                     key=headers["X-Shopify-Webhook-Id"],
                     headers={"event-type": headers["X-Shopify-Topic"]},
                     on_delivery=lambda err, msg: errors.append(err))
    producer.flush(3)                                # stay inside Shopify's 5 seconds
    return 200 if errors == [None] else 503          # anything else: Shopify retries

Flushing per request keeps the sketch short; a busy receiver batches produces and answers each request from its own delivery callback.

What happens after the initial load?

• There is no initial load. A subscription carries events from its creation on; history comes from the provider's REST API.
• Retries follow a non-2xx response or a timeout, on each provider's schedule. GitHub doesn't redeliver automatically.
• Duplicates arrive as separate requests and become separate messages.
• Order isn't guaranteed (Stripe says so explicitly). Use payload timestamps or versions, not arrival order.
• New fields and event types appear over time; consumers must tolerate them, and a schema registry needs a compatibility mode that allows added optional fields.
• Deletes arrive as ordinary events with their own type, not as Kafka tombstones.
• Outages longer than a provider's retry window lose events unless you resend them from the provider, and recovery brings a burst of retries.

Mapping webhook payloads to Kafka messages

Each verified request becomes one message:

Choosing the message key

A random key spreads a redelivery to another partition, so deduplication needs a store shared by every consumer instance. The delivery ID keeps an event and its retries on one partition, and on a compacted topic Kafka eventually keeps only the latest message per key. An entity ID gives per-entity order, but compaction would then keep only each entity's last event, so don't compact an entity-keyed event topic.

Performance, latency and Kafka cost

Latency through a receiver is verification plus the Kafka write; the limit that matters is the provider's deadline, not Kafka's throughput.

• Keep producers warm. A new connection and SASL handshake per request eat into Shopify's five seconds.
• Watch acks=all under stress. A slow or rebalancing cluster shows up as provider timeouts; return 5xx before the deadline rather than letting the provider time out.
• Size for bursts. Providers retry their whole backlog after an outage. Scale the receiver horizontally, and on Lambda check the concurrency limit.
• Cost follows message volume, retention and replication factor; for small JSON payloads, producer compression and retention sized to your replay needs matter most.

Production considerations

• Recovery. A receiver that answers before producing loses events when it crashes in between, so return 2xx only after a durable write.
• Authentication. Check every request's signature or token; an open endpoint lets anyone write to your topics.
• Replay protection. Reject signed timestamps older than a few minutes, and keep clocks in sync.
• Monitoring. Alert on the receiver's non-2xx rate and response time, failed deliveries in each provider's dashboard, producer errors and consumer lag.

Common problems and troubleshooting

Signature checks fail on valid deliveries

Every delivery from one provider gets a 401. Log the length and SHA-256 of the bytes you verify, and compare them with the request as received.

• The body was parsed first. JSON middleware reorders keys or changes whitespace; read the raw bytes before any parser runs.
• Wrong encoding or prefix. Shopify sends base64; GitHub sends hex after sha256=; Stripe packs t= and one or more v1= values into one header.
• Wrong secret. Shopify signs with the app's client secret; each Stripe endpoint has its own signing secret.
• Clock skew. Stripe signs each retry with a fresh timestamp, so retries don't cure a drifting server clock.

RecordTooLargeException on large payloads

The Java producer throws RecordTooLargeException: The message is N bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration.; librdkafka clients report MSG_SIZE_TOO_LARGE ("Broker: Message size too large"). Raise the topic's max.message.bytes and the producer limit together, or move the payload to object storage and produce a pointer. Set the limits from each provider's maximum payload size.

Events missing after an endpoint outage

Stripe and Shopify retry for a limited window; GitHub doesn't retry. List failed deliveries in the provider's dashboard or API and redeliver them, and read anything older from the provider's REST API. Shopify can remove a subscription created through its Admin API after repeated failures, so check subscriptions after a long outage.

Using Estuary for webhooks to Kafka

On this route the HTTP Webhook capture is the receiver, writing each request to a collection, and the Apache Kafka materialization produces each collection to a topic in your cluster. Against the concepts above:

• Acknowledgement: a 200 goes back only after the request's documents are committed to the collection. A body that isn't JSON, fails the collection schema, or lacks a required ID header gets 400, and none of it is captured. During a restart, usually after a publication, the endpoint can return 5XX for a few seconds; Stripe and Shopify retry those, GitHub doesn't.
• Authentication: an optional bearer token (wrong token 403, missing 401). Without a token or signature check, anyone with the URL can write to the collection.
• Signatures: HMAC SHA-256 or ECDSA P-256, built in for Zoom, Knock and Twilio SendGrid, plus a custom setting. The custom setting reads the whole header as hex or base64, so it verifies Shopify but not GitHub's sha256= prefix or Stripe's combined t=/v1= header. One scheme covers every path in a capture.
• Keys: the document key is _meta/webhookId, a random UUID unless the binding's idFromHeader names a header such as X-GitHub-Delivery, which every request must then carry. Key on a delivery ID, not an entity ID: documents with the same key can be combined, keeping only the latest.
• Messages: each message is keyed by _meta/webhookId. The value holds the payload's top-level fields plus _meta, which carries the request headers (all but Authorization, Cookie, Content-Type and Accept), query and path parameters, and the receive time. No Kafka record headers are set, so the event type travels inside _meta. Format is JSON, or Avro through your schema registry.
• Delivery: at least once, with no sync schedule: documents are produced as each transaction commits.
• Limits: request bodies up to 64 MiB after decompression, as a JSON object or an array of objects (one document each). Messages truncate strings over 64 KiB and arrays or objects over 1,000 entries, with no setting to change that, and a message over Kafka's default 1 MB limit fails the materialization until its large field is excluded under field selection (below).
• Security: Estuary is SOC 2 Type II certified and HIPAA-compliant, and its security page shows a GDPR compliance badge.

Implementing webhooks to Kafka with Estuary

These steps adapt Estuary's general data flow guide to this route.

1. Complete the prerequisites in Webhook provider requirements and Kafka requirements. Estuary also needs:
   • Brokers reachable from Estuary's IP addresses, with SASL PLAIN, SCRAM-SHA-256 or SCRAM-SHA-512 over TLS. There's no SSH tunnel or AWS IAM option, so use SCRAM on Amazon MSK.
   • One capture per signing scheme.
2. Create the capture. On the Sources page, click New Capture, choose HTTP Webhook, name the capture, and fill in the endpoint properties:
   • URL paths: one per provider, for example /shopify. Each path becomes a binding and a collection.
   • Authentication token: for senders that can set an Authorization: Bearer header. With a token, set CORS Allowed Origins to an empty list (it can't be *).
   • Signature Verification: for Shopify, Custom with Algorithmhmac_sha256, Public Key set to the app's client secret, Signature HeaderX-Shopify-Hmac-SHA256, Signature Encodingbase64 and Signing String Template{PAYLOAD}. Unsigned or mis-signed requests get 401.

3. Set delivery IDs and publish. Click Next. Estuary lists one binding per path under Output Collections. In each binding's resource configuration, set idFromHeader to the provider's delivery header, such as X-Shopify-Webhook-Id or X-GitHub-Delivery, and leave the collection key at /_meta/webhookId. Click Save and publish.
4. Register the URL. On the capture details page, the Endpoints section shows the Public endpoint base URL. Append the path, for example /shopify, and paste the full URL into the provider's webhook settings.
5. Create the Kafka materialization. On the capture's details page, click Materialize, choose Apache Kafka, and fill in the endpoint properties:
   • Bootstrap Servers: comma-separated host:port list.
   • Credentials: SASL (User & Password) with SASL Mechanism, Username and Password; keep TLS Settings on system certificates.
   • Message Format: JSON, or Avro with Schema Registry Endpoint, Schema Registry Username and Schema Registry Password.
   • Topic Partitions and Topic Replication Factor (defaults 6 and 3) apply only to topics the connector creates. Compression defaults to lz4.

6. Map collections to topics and choose fields. Click Next. Each binding's Topic defaults to the collection name; set it to your topic, such as webhooks.shopify. Under Field Selection:
   • Select the root document field (flow_document) to carry the whole payload as one JSON string beside the top-level fields.
   • Exclude any field that can push a message past 1 MB. The producer keeps Kafka's default message size limit, so excluding the large field is the fix when the materialization fails with MessageSizeTooLarge.
7. Publish. Click Save and publish. Estuary creates missing topics and produces the collection's stored documents, then each new one.
8. Verify that events are landing.
   • Send a test event from the provider. On the capture details page the Shard Information card reads PRIMARY; open the collection from the Collections list to see it in Data Preview.
   • The materialization details page charts Data Read.
   • Consume the topic and print keys (for example kcat -C -t webhooks.shopify -f '%k\n' plus your broker and SASL settings). Resend the event from the provider and confirm both messages carry the same key.

How Estuary handles acknowledgements, duplicates and extra destinations

Every accepted request lands first in a collection, JSON documents kept durably in cloud storage. The materialization reads that collection independently, so the provider's response never waits on your brokers.

A Kafka outage doesn't bounce webhooks

A receiver that answers after Kafka acknowledges has to return 5xx while Kafka is down, and each provider retry spends its window. Here the 200 depends only on the collection commit. If the brokers are unreachable or reject a message, the materialization fails while the capture keeps accepting requests; once Kafka accepts writes, the materialization resumes from its last checkpoint and produces the backlog. Delivery is at least once, so a message from the interrupted transaction can appear twice.

What a delivery-ID key removes, and what it doesn't

The Kafka materialization always uses delta updates: it combines documents with the same key within a transaction and produces one message per key, without reading what it produced before.

Kafka and a warehouse from one endpoint

To send the same events to Kafka and a warehouse, add a second materialization on the collection: providers keep one URL, and the new destination starts from the collection's stored history. The same history can feed a new topic after a format change without providers resending, as long as the collection still holds the documents.

When another approach may make more sense

• Kafka consumers, but no cluster of your own:Dekaf serves Estuary collections over the Kafka protocol, with Estuary as the broker and schema registry. Consumers connect to dekaf.estuary-data.com:9092 with SASL_SSL and PLAIN and read each collection as a topic. It is read-only and Avro-only and puts nothing in your cluster.
• Stripe or GitHub signatures checked at the edge: run your own receiver, since Estuary's custom setting can't read those header formats. You then own the endpoint's availability.
• One provider, low volume, all on AWS: API Gateway and a small Lambda function, with your own producer and verification code to maintain.
• Providers that already deliver to your event bus: Shopify and Stripe can skip the public endpoint (see the provider requirements), but the bridge from the bus into Kafka is yours to build.
• Kafka Connect shops that accept unsigned requests: Apache Camel's webhook-source connector forwards requests as they arrive; its docs exist only in the pre-release line.
• The reverse, Kafka events out as webhooks: an HTTP sink connector or outbound webhook gateway. Estuary's HTTP Webhook materialization does this from a collection.

Related resources

• Estuary HTTP Webhook capture (signature verification, webhook IDs), Apache Kafka materialization, Dekaf and reading collections with Dekaf
• Estuary collections, delta updates, customizing materialized fields, and the HTTP Webhook to Apache Kafka integration
• Providers: Stripe webhooks, GitHub: validating deliveries, GitHub: failed deliveries, Shopify: verifying deliveries
• Kafka and gateways: Kafka producer configs, Confluent REST Proxy API, Strimzi HTTP Bridge, Redpanda Connect http_server, Apache Camel webhook source, API Gateway quotas

To try the route, create an HTTP Webhook capture in the Estuary dashboard and materialize it to Apache Kafka. Pricing: estuary.dev/pricing.