A decade of Gazette: a streaming broker that doesn’t store data

There’s something even more scary than live traffic in a streaming system: backfills.

A new application almost never wants just the events arriving from that moment forward, it wants the history too, months or years of it, sometimes petabytes. It has to work through all of that as fast as possible, catch up to the present, and then keep processing new events at millisecond latency.

So, you get two very different workloads fighting over the same hardware: the live path wants predictable latency and lots of small, frequent writes, and the historical path wants to scan enormous contiguous ranges as fast as the disks and network will let it. If you run both through one broker fleet, then every new consumer is a threat to production stability.

This can usually be fixed by exporting the broker log to object storage, but that's not really a fix, is it. Now you have another pipeline, another copy of the data, another retention policy, and another place where the "historical" view disagrees with the "live" one. Separate pipelines derive slightly different versions of what should be the same dataset, and engineers burn absurd amounts of time reconciling them.

Gazette was built specifically to avoid opening that gap and the distilled constraint we aimed for sounds like this:

“The live stream and its historical archive should be the same ordered dataset.”

Gazette is an open-source streaming broker designed by Johnny Graettinger at Arbor. It has run production workloads for about a decade and later became the streaming engine underneath Estuary.

This is a deep dive into its architecture.

Why object storage alone isn’t a low-latency event log

Object storage is the obvious place to keep a lot of data, right? It's elastic, durable, cheap next to replicated broker disks, and pretty much every batch and analytical system already knows how to read data from it.

But an object store, by itself, is a terrible low-latency event log. If every producer had to create an object and wait for it to become visible, the live path inherits the object store's latency and request model. Small appends get inefficient, concurrent writers need coordination from somewhere, and readers still need a consistent way to follow the head of the stream.

Local broker storage looks the opposite: it's great at receiving small appends, sequencing concurrent writers, and waking readers immediately, but, it's a lousy place to keep years of history: capacity has to be provisioned up front, disks get rebalanced, and historical scans compete with the live workload.

We use both, they just have different roles.

Brokers own the hot path:

• Establishing the order of concurrent appends
• Replicating recent writes across failure zones
• Making committed bytes immediately visible
• Serving readers who follow the live head

Object storage owns history:

• Sealed, immutable ranges of the log
• Large replays and backfills
• Historical read capacity that scales independently of the brokers
• The same data exposed as ordinary files

Short-term durability comes from synchronous broker replication and long-term durability comes from the object store, which is deliberately kept out of the latency-sensitive append path.

It’s important to keep in mind that these aren't two datasets, rather, it's one logical log with two representations, “stitched together” by byte offsets, as the diagram below illustrates:

Gazette journals: a file you can tail

Gazette's basic unit is a journal: an append-only byte stream that can be read from any byte offset. Once a reader catches up to the write head, it blocks and receives new bytes as they commit.

The closest Unix analogy is literally:

bash
tail -c "$OFFSET" -f journal

Unlike a regular file, a journal can have many concurrent writers, so the broker has to establish one serial order for their appends. Appends are serializable: an individual append either arrives as one contiguous range or it doesn't arrive at all, appended spans have a total ordering, bytes from two appends never interleave, and readers never observe a partial append that later gets rolled back.

Conceptually, the API looks like this:

bash
append(journal, bytes) -> [begin_offset, end_offset)

read(journal, offset) -> committed bytes from offset onward

Bytes, not records

This decision had a huge impact on pretty much everything downstream of it.

Kafka and most other streaming systems make records a first-class broker concept, Gazette doesn't. The broker has no idea whether a journal holds JSON, Protobuf, CSV, Avro, or database rows. It understands ordered, additive byte ranges, and that's it.

That keeps the broker's job well defined. It assigns a total order within each journal, atomically commits or rolls back an append, replicates bytes, validates offsets and content hashes, and serves committed ranges. That's the whole list.

Everything else — representation, framing, packing, parsing — belongs to the client. An append might be one newline-delimited JSON document or ten thousand Protobuf messages. Gazette treats it as one atomic byte span either way.

This removes a huge amount of policy from the broker. Gazette can guarantee that a connection failure never leaves half an append visible, but it cannot stop an application from successfully committing invalid data. It cannot stop an application from successfully committing garbage. If a client correctly writes an invalid JSON line, the broker will faithfully preserve that invalid JSON line forever.

People occasionally read that as a missing integrity check, and it isn't: content chunks are SHA-summed as they stream, every replica computes the sum independently, and each sealed fragment is defined in part by a SHA sum over its byte range. (SHA1, for the record, it's there to catch corruption and to make fragments content-addressable, not as a security boundary.) What's missing is any notion of what a record is, but that's the price of keeping record semantics out of the storage layer, and we'd definitely pay it again.

What happens during an append

From the client's side, an append is one call, but internally, it's a small distributed transaction over the brokers currently assigned to the journal. Seven steps, of which the interesting ones are 3, 4, and 6, so if you skim, skim to those!

Every journal has a route: one primary broker plus its replicas, recorded in Gazette's etcd-backed topology with replication factor being configurable per journal, and in Estuary it's set so that a journal's peers span at least two availability zones.

The common scenario goes roughly like this:

1. Resolve the current route

The receiving broker resolves the journal to its current primary and replicas.

Topology can change while an append is waiting as brokers compare etcd revisions to figure out which route is newer, and a broker that discovers its local view is behind waits for its watched etcd state to catch up before proceeding. Otherwise an old topology can go unnoticed and simply win a race against a newer one, which is exactly the kind of bug that pages your ops team at 3am, so we should do everything to avoid it.

2. Acquire the journal's replication pipeline

The primary keeps long-lived bidirectional replication streams open to the journal's replica brokers. These get reused across many appends, because paying connection setup and synchronization costs on every small write would be silly. The pipeline can have multiple ordered appends in flight at once, with separate send and receive ownership, while acknowledgements stay in order.

Underneath the pipeline sits the spool. Each replica keeps one per journal: its transactional memory of the currently open fragment.

The spool tracks the previously adopted fragment, new content proposed after it, byte offsets, content hashes, and a small set of journal registers. Only one mutating RPC owns a spool at a time, which makes it a small explicit state machine rather than shared mutable storage with ambiguous writers.

3. Synchronize the replicas

Before a newly built pipeline gets used, the primary proposes its view of the current fragment and route to every replica, and each replica checks that view on its own. A replica might know about a later route, might have seen a larger journal offset, might be unable to continue the proposed fragment. In those cases the primary re-resolves the topology or closes the old fragment before moving on.

So a broker joining a journal's topology doesn't just take the primary's word about the log head, it also verifies against its own spool and known fragments. This has a useful implication that shows up again later: the current fragment is always closed and persisted whenever a new broker joins the topology, so the fragment a new peer starts from is always a fresh one at the current write head.

4. Check append preconditions

Before streaming any content, the primary validates requested offset or register conditions. Most appends don't set these; they just append at the current head. But preconditions are how fencing, conditional writes, and explicit recovery after a bad consistency loss get built. If a register doesn't hold the expected value, the append fails instead of applying against state nobody expected.

The primary also validates the offset itself: content must arrive at the furthest known extent of the journal. Usually that's the offset of the pipeline's spool, but if consistency was lost through enough broker or etcd faults, the fragment index may already know about a larger one. Gazette deliberately refuses to resolve that automatically. There may be a partitioned broker still holding local fragments it hasn't persisted, and guessing risks writing an offset that broker already committed. Instead the append fails with INDEX_HAS_GREATER_OFFSET and an operator has to explicitly declare the new maximum offset with gazctl journals reset-head. Failing to a human has proven to be the right call when the alternative is double-writing an offset.

5. Stream bytes to every replica

The client streams content to the primary, which forwards each chunk through the pipeline. Every replica independently tracks the resulting offset and hash. The primary can't force a replica to adopt a fragment whose range or content hash doesn't match the bytes that replica actually received. A mismatch is an invalid state transition and aborts the replication stream.

6. Commit or roll back

When the client finishes cleanly, the primary sends a fragment proposal that extends the previously committed fragment over the new bytes. Adopting that proposal is the commit.

If the client disconnects or the stream dies partway, the primary sends a rollback proposal instead. The spool returns to the prior adopted fragment, and readers never see the abandoned bytes.

7. Wait for replica acknowledgements

The append succeeds only after the replication peers explicitly acknowledge the commit proposal. In the common case, the long-lived full-duplex pipeline gets the internal commit protocol down to one acknowledgement round trip, and while one append waits on its acks, later ordered appends are already moving through the pipeline.

A synchronized pipeline is effectively a distributed exclusive lock on appending to the journal, which is also why brokers can't enter or leave a topology without synchronizing first. Client appends don't always arrive often enough to drive that, so the primary runs a "pulse" daemon that synchronizes the pipeline proactively even on an idle journal.

A fragment is not an export of the stream

The open spool can't grow forever as eventually, Gazette closes it and starts another. We call the sealed range a fragment, and it's identified by the journal name, an inclusive begin offset, an exclusive end offset, and a SHA sum of the bytes in between.

Fragment boundaries never split a client append, so if clients append complete newline-delimited records, every fragment holds complete newline-delimited records. It’s the same story for length-delimited Protobuf, CSV, or whatever framing the client picked.

Once closed, a fragment gets uploaded to the configured object store as an immutable file whose name encodes the offset range and content hash. These files may be compressed but they hold the journal's raw content rather than some proprietary broker format.

Worth emphasizing that nothing "exports" the journal into a data lake on a schedule. The sealed fragments are the journal's history (everything older than the open spool, anyway).

For sealed history, the object-store layout is considered authoritative, meaning that fragment names are structured so that listing the journal's prefix tells you which ranges exist and where they sit in the stream. Brokers periodically list the store to build an in-memory fragment index, merge it with fragments that are still local, open, or mid-persist, and use the combined view to find the fragment covering any requested offset.

This turned out to be a really pleasant simplification: there's no separate database holding an indispensable manifest of every historical segment, rather, the immutable objects and their names already describe the sealed log.

Historical reads shouldn't go through production brokers

Once fragments were regular objects, we had to decide whether brokers should proxy every historical byte. They can. But they don't have to.

For a requested offset, a broker can serve a fragment it still has locally, proxy the content from the object store, or return a signed object-store URL so the client reads the fragment directly and the broker gets out of the way.

That third option changes the scaling model as a new consumer can read historical fragments at the aggregate throughput of the object store and its own workers, instead of dragging every byte through the same broker processes that are coordinating current writes.

In Estuary, a new materialization or derivation starts at its required journal offsets and fetches historical fragments from the configured bucket. As it approaches the present, it transitions automatically from object-backed reads to the broker-served live tail. The offset space never changes across that transition.

So this is where "can we add this application and replay everything" stops being a scary question because the backfill can be enormous without being an enormous broker workload.

The specific fear is familiar to anyone who's operated a shared streaming cluster: a genuinely useful new application needs a petabyte-scale replay, and that replay might knock over the production path. Serving history straight from object-storage files removed the coupling.

How Gazette differs from tiered storage

At the level of a simplified block diagram, sure. It looks like Kafka with KIP-405 tiered storage, or Pulsar over BookKeeper, or Confluent's Freight and WarpStream. The difference is less about whether object storage shows up somewhere and more about where authority and operational responsibility live.

Kafka's tiered storage moves cold segments to a bucket, but the broker's log remains the system of record and the tier is a retention optimization underneath it. Pulsar splits serving from storage but keeps the storage tier as its own stateful cluster to operate. Gazette inverts the authority: the bucket listing is the fragment index, and the broker holds nothing it can't lose.

In Gazette, sealed fragments in storage are the durable historical representation, full stop. A broker doesn't need its old host disk to come back, a cold broker can start serving without first copying a journal's complete history, reassigning a journal doesn't mean migrating its retained log data, and historical clients can be delegated straight to the stored files.

The local disk is scratch space for recent content. Gazette still makes great use of local SSDs, it just assumes the machine and its disk can disappear at any moment. That boundary is what the system was designed around, not a cost optimization bolted on afterward.

How Gazette makes streaming brokers disposable

A Gazette cluster coordinates metadata through etcd: journal specifications, broker membership, routes, and assignments. Journal payloads never touch etcd.

Each broker advertises an ephemeral lease describing its endpoint, failure zone, and available capacity. Assignment records say which brokers are the current primary and replicas for each journal. When a broker exits, or its lease expires, the allocator recomputes the assignments.

And because old fragments already live in shared object storage, moving an assignment doesn't require dragging a journal's full history from one broker disk to another. The newly assigned broker synchronizes the active edge of the log with the existing route, and that's it.

Short-term durability is still important as we can't upload every individual append as its own object, and the newest fragment may not be sealed yet, so Gazette synchronously replicates current appends across brokers placed in different failure zones. The object store takes over as the durable copy once a fragment closes.

This separation also changed how we do deployments. On graceful shutdown, a broker declares it wants no more assignments, hands off its journals, makes sure local content has been persisted, drains its RPCs, and exits. No carefully reattached persistent volume full of irreplaceable history.

There's a tradeoff here: a cold broker occasionally has to wait until sufficiently recent content shows up in the object store before it can serve a particular read. We took that deal happily in exchange for deleting permanent machine identity from the architecture.

Exactly-once semantics in Gazette

"Exactly once" is one of those phrases that gets less useful every time it's repeated without defining a boundary around it.

Gazette's append API is at-least-once: suppose a producer commits an append, the replicas acknowledge it, and the response gets lost on the way back, so the producer can't know if the append is committed or not. If it retries, the same serialized message can land twice. No broker protocol erases that uncertainty without a higher-level notion of producer identity and sequence.

Gazette adds that notion in its message and consumer layers. Messages are sequenced with UUIDs, which carry a node ID that Gazette uses as a ProducerID plus a timestamp and clock sequence drawn from a strictly monotonic clock that ticks with every UUID. A reader tracks the largest clock seen per ProducerID and presumes anything smaller is a duplicate. Note that each publisher draws a new random ProducerID rather than carrying a stable application identity, which is part of why consumers age out producer state over time.

Gazette also repurposes some clock-sequence bits as flags, which is how transactions are expressed: pending messages are published with CONTINUE_TXN, and an ACK_TXN message commits every pending message with a smaller clock while rolling back those with a larger one.

That gives you the primitives for exactly-once effects, but only combined with transactional consumer state. Inside a transaction, a consumer shard does the following:

1. Reads one or more source messages.
2. Applies changes to its state store.
3. Publishes downstream messages in a pending state.
4. Builds a checkpoint holding source offsets, producer sequencing state, and acknowledgements for the pending outputs.
5. Commits that checkpoint with the state-store transaction.
6. Publishes the acknowledgements only after the store commit succeeds.

Die before the store commits, and the state changes vanish while the pending outputs stay uncommitted. Die after the commit but before all the acks go out, and the recovering process reads the committed checkpoint and republishes them. Duplicate acks are harmless.

The framework never promises that a handler runs only once. A handler may run several times, state mutations may be staged several times, and duplicate read-uncommitted messages may be published.

What it promises is more useful: a message is processed in exactly one completed transaction, store updates derived from it commit exactly once, and messages derived from it are read exactly once by any read-committed reader.

Since an uncommitted reader will see duplicates and pending messages that may never commit, the read-committed qualifier is especially important, and because the guarantee is stated in terms of committed effects, it cascades: chain five consumers together and the effect of a source message still lands once at the end.

Zombie processes

Now that the baseline is established, let’s take a look at scenarios where things go wrong. The nastiest failure mode is an old process that everyone believes is dead.

Say a shard moves from process Old to process New:

1. Old commits transaction T1.
2. Old pauses because of a network partition or a long runtime stall.
3. The cluster assigns the shard to New.
4. New recovers T1 and starts processing.
5. Old wakes up.
6. Old commits T2, writes its ACK intents, and exits.

Without a write fence, Old and New both commit. Derived messages get written twice and store mutations get applied twice, from two owners with two different views of state.

For remote transactional stores, Gazette can lean on a fence maintained by the store itself, but for local embedded stores backed by a recovery log, it uses journal registers. Registers are small key-value entries attached to a journal and updated through the same transactional append machinery, and an append can require that a register still hold an expected value.

1. When New takes ownership it finishes replaying the recovery log and injects a "handoff" that takes over sequencing of recorded operations, placing an updated "author" register as part of that. Its recorder then verifies that fence on every operation it records.
2. Old wakes up, tries its next recovery-log append carrying the fence it placed back when it was primary, and gets a register mismatch. One important constraint is that an append can only update registers if it writes at least one byte.

We like this mechanism a lot, because "I am still the owner" essentially transforms from being an assumption to an append precondition.

Replicating local state without teaching the broker compaction

Byte-oriented journals create one apparent hole: there's no broker-level keyed log compaction. As Gazette can't look inside arbitrary journal bytes, find the latest value for a key, and drop older records, it doesn't know where the keys are, plus, it doesn't know where the records are.

We could have taught the broker record semantics and compaction, but instead we leaned on a property of embedded databases like RocksDB: they already organize their durable state as immutable files, and they already know how to compact those files correctly.

Gazette consumers can record the embedded database's file operations into a recovery-log journal. A hot standby tails those operations and reproduces them on its own local disk. After a failure, a newly assigned process replays the recovery log and reconstructs the state store.

So a stateful processor gets local embedded-database performance without marrying its state to one machine. The running state is local, the ability to recover it is external.

Recovery logs are also why retention takes more thought than "delete objects older than 30 days." An old recovery-log fragment can still hold file content the current database needs. Gazette ships purpose-built pruning to figure out which recovery-log fragments are actually obsolete.

Impacts of the architecture

Before this design, adding a new historical consumer was a capacity-planning event: work out how much broker bandwidth it eats, whether it evicts useful pages from cache, whether the disks survive, whether we need an export first, and how stale that export will run.

Afterward, history and the live head had separate dials. More concurrently active journals or more live traffic: add brokers. More historical reads: consumers pull more fragments from the bucket. More retention: change bucket lifecycle policy instead of resizing every broker volume. Trimming works in both directions too, since content can be removed from the beginning or even the middle of a journal, and a file removed from the bucket is removed from the log.

That's the operational meaning of separating sequencing from storage and as you can see it was never just about the storage bill. It deleted several forms of coordination between workloads that had no business being coupled. Gazette's own design docs describe this as scaling write capacity and historical-read capacity independently.

What Gazette doesn't give you

Gazette gave us the primitives we wanted: ordered journals, transactional appends, replicated hot fragments, immutable object-store history, consumer shards, checkpoints, recovery logs, fencing.

What it didn't do was make those primitives accessible to data teams. A Gazette user still had to decide how records were encoded, how journals were named and partitioned, how schemas were managed, how source systems got captured, how destination transactions worked, and how processing shards were operated.

Estuary is the layer that answers those questions: collections of schema-bound JSON documents over groups of journals, with captures, derivations, and materializations as managed tasks. The storage model stays visible where it's useful — collection history is regular files under a prefix in your own bucket — and invisible everywhere else. Gazette made the architecture possible without making it easy to adopt.

A few more important decisions

A lot has changed in streaming infrastructure since we started. Object storage is faster and cheaper, separating compute from storage is the norm, tiered logs stopped being an exotic idea. If we rebuilt Gazette today we'd still put history in object storage, but so would everyone else, which is exactly why that choice no longer says much on its own. Some of the decisions doing the real work are the stricter ones, and each of them describes the system as it runs today:

• Sealed history is independently readable.
• Historical scans stay off the live broker path.
• Broker machines and their disks are replaceable.
• Hot and historical data share one offset space.
• Commit and rollback are explicit at the append layer.
• Exactly-once is defined in terms of committed effects.
• Ownership is fenced rather than inferred from failure detection.
• Application record semantics live above the byte log.

These decisions affect all aspects of the system. External history is what lets brokers be disposable and disposable brokers are what let assignment changes skip moving years of data. Direct fragment reads are what keep backfills from threatening the live path and having one offset space is what lets a consumer walk off an old file onto the live tail without switching datasets. And a transactional append layer is the floor that consumer transactions and fencing stand on.

It has to be said that Gazette isn't the right abstraction for every messaging workload. A team that mostly wants a familiar record broker, a big off-the-shelf ecosystem, or broker-managed keyed compaction should probably pick something else, and that's fine.

Our recurring problem was different as we needed to ingest continuously, retain complete history, add stateful consumers, and replay enormous ranges without treating every backfill as an incident waiting to happen.

Everybody puts files in object storage now. The commitment was making those files the durable history of the stream, and making the brokers forgetful on purpose.