How Together AI Uses Near-Real-Time Data to Understand Inference Economics

From distributed inference events to fresh analytics

A single backend engineer used Estuary to stream inference events to provide product leaders, executives, and engineers fresh data for business analysis and debugging.

Together AI is a cloud platform focused on infrastructure for training and running open-source AI models. The platform provides a full-stack product suite with compute and software components, supporting over 50 open-source models that users can customize with their data, retaining ownership.

Together AI runs a full-stack AI platform spanning inference, compute, and model development. Its inference platform is a distributed system: requests pass through multiple services before reaching an inference engine. Each stage produces useful signals about request behavior, latency, system health, utilization, and the service delivered to the customer.

When Julia Zhang joined Together AI's inference platform team, those signals did not yet form a durable analytics dataset. The company had Prometheus for metrics and Loki for logs, but no structured analytics path for request-level events. Prometheus data aged out after roughly two weeks, which limited historical analysis. Teams could see the system's current condition, but answering questions about long-term behavior, platform economics, or customer commitments was much harder.

Together needed to make event data queryable quickly, retain it for historical analysis, and avoid asking an inference engineer to build and operate a custom data platform first.

“It’s more just: we didn’t have any visibility, and now we do.”
— Julia Zhang, Software Engineer, Together AI

Operational monitoring covered the present, not the historical questions

Prometheus and Loki already served an important purpose. Together's engineers used them to monitor services, alert on active problems, and investigate recent behavior. The gap appeared when teams asked questions across a longer window or needed to connect infrastructure behavior with business outcomes:

• How is platform utilization changing over time?
• Which workloads affect serving efficiency and economics?
• How consistently is the platform meeting customer promises and SLAs?
• What request and system conditions preceded a difficult issue?
• Can product managers and executives use the same fresh data to guide priorities?

These questions required a persistent, queryable event history. Together kept Prometheus as its operational alerting system and built a separate analytics path for deeper analysis. That distinction is central to the architecture: monitoring remains optimized for rapid reaction, while Estuary delivers fresh operational data into a system designed for analytical queries.

Wide events created one flexible stream for analysis

Julia and the inference platform team instrumented services with an SDK that publishes events into AWS. They treated the records as wide events: large, sparse, composable records that can carry request metadata, latency measurements from important code paths, system-health signals, and other attributes that may become useful later.

“There’s no such thing as logs, metrics, and traces. There are just wide events.”— Julia Zhang

This approach kept instrumentation accessible to service owners. A team could publish relevant context without first designing a specialized pipeline for every metric, trace, or log type. The records could land in raw tables immediately, and the data team could add curated models and transformations as the analytical requirements became clearer.

The design also supported growth. AWS supplied the shared streaming backbone and the raw storage layer. Other internal consumers could read the streams for their own use cases without coupling every downstream system to the original services.

Estuary provided the shortest reliable path to Analytics

The ingestion path into the data warehouse was the open problem. Building and operating a custom connector would have slowed the initial project, and the other ETL options Together evaluated required more setup or more data-platform knowledge than the team wanted to take on.

Estuary matched the immediate requirements:

• A direct capture path from the event streams
• Continuous materialization into warehouse tables
• Low setup and operating overhead for an engineer outside the data team
• The ability to land raw data first and introduce transformations later
• Fresh data for analytics and debugging without waiting for an hourly or daily batch window

“At that time, we just wanted to go from A to B. We didn't need to transform the data. We just wanted to have data that we could analyze.”
— Julia Zhang

Estuary's role was deliberately focused. It captured the structured events and materialized them into the analytics warehouse, where teams could query the raw tables. Together gained a useful analytics system without committing to a large internal platform design before it understood the long-term requirements.

One engineer delivered the first iteration in about a week

The first implementation followed a short path:

1. Together instrumented inference-platform services to publish structured wide events into AWS.
2. Together maintained a separate raw copy..
3. Estuary captured the JSON event streams into collections.
4. Estuary materialized those collections into tables in the data warehouse.
5. Teams queried the raw tables and began building dashboards without waiting for an initial transformation layer.

Julia completed that first iteration by herself, before Together had dedicated data engineers available for the project.

“In a week, I was able to instrument my services, get data into the warehouse, and start using it. I don't think that would have been possible without Estuary at the time.”
— Julia Zhang

That week included more than connector configuration. It covered service instrumentation, event publication, delivery into the warehouse, and the first useful queries. This is the clearest measure of the project's impact: Together moved from having no durable analytics path for these events to using the data in roughly one week.

Fresh data changed both business analysis and engineering investigations

Once the event stream was available in the data warehouse, the same underlying data supported two groups with different questions.

Product managers and executives use high-level dashboards to understand:

• Platform utilization
• Serving efficiency and economics
• Performance against customer promises and SLAs
• Business performance and strategic priorities

Engineers use the event history to investigate difficult issues with current data already available. They can analyze request metadata, code-path latency, and system-health context without waiting for the next batch load or relying only on a short observability window.

Freshness matters here because partially completed batch windows create ambiguity. When a dashboard looks unusual, teams can lose time asking whether the underlying behavior changed or whether the latest load is incomplete. Together's streaming path removes much of that uncertainty for this dataset.

“We never have to question if our data is there when we go do analytics.”
— Julia Zhang

Together still uses Prometheus for service monitoring and alerting. Estuary supports the analytical and investigative path, where near-real-time delivery means teams can begin querying soon after an event occurs. This is the practical meaning of right-time data for Together: the data is ready when a decision or investigation begins.

The pipeline is observable too

Together uses Estuary's OpenMetrics API with Grafana to view pipeline metrics and configure alerts. The API exposes activity across captures, derivations, and materializations, including document and byte counts, errors, failures, and the publication time of the latest materialized source document.

This gives the team a direct view into the delivery layer. If a dashboard appears stale, engineers can inspect pipeline activity instead of guessing whether the issue is in the source, the query, or data movement.

A fast start that left room for a larger data organization

The project began as a single-engineer initiative. Together has since built a dedicated data team and additional custom processing pipelines for other internal and customer-facing needs. Kinesis remains useful as a shared stream because several consumers can read the same events, while the data copy preserves raw data independently.

Estuary continues to serve the path where it has the clearest fit: moving streaming data into the warehouse with little operational work. The surrounding data platform can grow without forcing Together to rebuild a working ingestion path prematurely.

The result is a useful pattern for fast-growing engineering organizations. Start with a dependable path from source to destination, make the raw data available while it is still useful, and add governance and transformations as the use cases mature.

Results

• About one week to first value: One backend engineer instrumented services and delivered usable data into the data warehouse.
• Near-real-time analytical data: Teams can query recent events without waiting for an hourly or daily batch load.
• Longer analytical history: Request-level events are available beyond Prometheus's roughly two-week retention window.
• Business visibility: Product managers and executives can analyze utilization, efficiency, economics, and customer commitments.
• Faster investigations: Engineers have fresh, structured context for difficult cross-service issues.
• A low-overhead starting point: Together began with raw tables and expanded its data organization over time.

“We were able to get real-time streaming into our data warehouse, so all of our logs are available immediately.”
— Julia Zhang

Takeaway

Estuary gave Together AI a direct, dependable streaming path into the data warehouse. In about a week, one engineer turned distributed inference events into fresh, persistent data that now supports business dashboards and engineering investigations across the company.