We pushed 64.8 million events through Amazon MSK Serverless and Databricks Zerobus Ingest into matched Delta tables, then reconciled the measured usage and every cost component. Here is what each path actually costs, and where the classic answer still wins.

There's a pattern hiding in almost every streaming architecture we review: events that are produced in one place and exist for exactly one purpose, to land in one table in the Lakehouse. No second consumer. No replay. No fan-out. And yet, by reflex, they travel through a Kafka cluster on their way there, because "streaming means Kafka" has been true for a decade.

Databricks Zerobus Ingest, a direct-to-Delta ingestion API that went GA in February 2026 as part of Lakeflow Connect, exists to challenge that reflex. For events with a single destination, it removes the middle of the pipeline entirely.

Earlier this year, Hubert Dudek described the “complexity tax” of multi-hop ingestion architectures: every additional system adds cost, maintenance, and another failure surface. His article made the architectural case for removing intermediaries that serve no independent purpose. We wanted to quantify one specific version of that tax.

For a single-sink pipeline into Delta, is Zerobus actually cheaper than MSK Serverless plus a Spark consumer? By how much, and does the answer hold as throughput grows? We went looking for a reproducible end-to-end benchmark that priced both the broker and the consumer required to land events in Delta, and came up empty. So we built one, carefully enough that you can audit it.

The full benchmark will be public and reproducible soon. The Github repository will contain the producer, Terraform, Databricks Asset Bundles, normalized results, raw billing exports, and a CAVEATS.md documenting every allocation and judgment call. Every number in this article can be traced back to those artifacts.

TL;DR

For single-sink ingestion into Delta at 500 to 6,000 records per second, the Kafka path (MSK Serverless plus a Spark Structured Streaming consumer) cost 3.6× to 20.9× more per GiB than Zerobus at list prices, as measured over our campaign. The gap is mostly Kafka's fixed hourly floor, so it shrinks as utilization rises, but it never closes: even ignoring the fixed costs entirely, MSK Serverless's volumetric charges alone run about 2.3× Zerobus's entire ingest price. In exchange, Kafka delivered rows to the table roughly a second and a half sooner in our measurements. If that second and a half matters, it’s real. If dollars matter, it's not even close.

First, the contenders: Apache Kafka and Zerobus Ingest

Apache Kafka needs little introduction. It's the de facto event backbone of modern data platforms: a distributed, append-only log where producers write to topics and any number of consumers read independently, at their own pace. That decoupling is the whole point. It buys you fan-out to multiple consumers, replay from any offset, retention, and a buffer between systems that fail on different schedules. In this benchmark, we ran Kafka as Amazon MSK Serverless, AWS's managed flavor that removes broker sizing and management and bills per cluster-hour, per partition-hour, and per GB in and out. One thing Kafka does not do by itself is write to your lakehouse: getting events from a topic into a Delta table requires a consumer. Typically, and here, that's a Spark Structured Streaming job on Databricks.

Databricks Zerobus Ingest is much newer, part of Lakeflow Connect and now generally available. It's a push-based ingestion API: your application opens a gRPC stream through a lightweight SDK and writes records straight into a Unity Catalog Delta table. Databricks runs the ingestion service end to end, and its rate card ($0.064/GB on AWS at Enterprise tier, us-east) is footnoted "Includes underlying compute costs": There is no separate compute charge for the ingest service itself. You still run your producer and pay for Delta storage, as on any path, what disappears is the broker and the consumer.

The trade-off is scope. Zerobus is a pipe into a table, not an event backbone. There are no broker-style consumer offsets, no multi-consumer fan-out, and no user-addressable retained stream to rewind. Durable records land in Delta, while service-managed buffering exists only to handle delayed table flushes. Delivery is documented as at-least-once.

The benchmark question, then, is narrow and practical:

When all you need is the pipe, what does carrying the backbone anyway actually cost?

Two paths to the same table

Both paths start from the same producer codebase and end in the same Unity Catalog schema, landing rows that are equivalent at the application level. Only the middle differs.

Path A, Kafka: producer → Amazon MSK Serverless (6-partition topic, IAM auth) → an always-on Spark Structured Streaming job on classic Databricks job compute (fixed 2 × m5d.xlarge, no autoscaling) → Delta table.

Path B, Zerobus: producer → Zerobus Ingest SDK (gRPC) → the same kind of Delta table. That's the whole diagram.

The structural difference matters more than any rate card: the Kafka path has two components that bill by the hour whether data flows or not (the MSK Serverless cluster at $0.75/hour, plus the consumer's DBUs and EC2 at roughly $0.64/hour). Zerobus has no idle meter at all; it bills purely per byte ingested, compute included. Everything downstream follows from that.

How we kept it objective

A cost benchmark is only as good as its fairness rules, so these were hard constraints:

Identical logical workload. One producer codebase with two sink adapters, generating the same 13-field telemetry record, tuned to exactly 1 KiB as JSON. Kafka carried that JSON on the wire; Zerobus carried the same fields as protobuf (about 858 bytes per record), because protobuf is what its SDK speaks. Same schema, same generator, same rates, same volumes: three one-hour runs per path at 500, 2,500, and 6,000 records/second, all on 2026-07-31 in us-east-1. 32.4 million records per path, 64.8 million total. Every record was acknowledged: zero errors and zero duplicates on both paths.
Costs from the bills, not from estimates. Databricks-side costs come from system.billing tables; AWS-side costs from finalized Cost Explorer data, pulled after the billing lag settled. Every figure reconciles two ways, metered units × published rates versus what the platform actually billed, and the reconciliations agree to within rounding.
List prices in every headline. Zerobus launched with a 50% GA promotion; we report it strictly as a separate callout below, never folded into a headline number.
Where a judgment call favored one side, it favored Kafka. The Kafka producer enables lz4 compression, which we estimate took 15 to 20 percent off MSK's volumetric bill (Zerobus meters uncompressed protobuf and got no such discount), and an ambiguously attributable slice of MSK cluster-hours was excluded from Kafka's total rather than estimated in.

The full methodology, including allocation policies, unit definitions, and everything we'd want to know before trusting someone else's benchmark, will soon be available at the repo’s link.

The results

Cost per GiB landed in Delta, at list prices, including one month of Delta storage (near-identical on both paths):

A note on what's measured versus modeled here: each path's campaign total is measured straight off the bills. The split across load levels is an allocation, documented in the repo: volumetric charges by record share, fixed charges split equally across the path's three one-hour runs, storage by GiB share.

And here is where every Kafka dollar went, because a benchmark that won't show its components isn't auditable:

Zerobus, same campaign: $1.74 of ingest (27.0 metered GiB at list) plus the same $0.40 of Delta storage. Total: $2.14.

Two things jump out. First, Zerobus's cost per GiB is flat, $0.084 at every load level, because it's a purely volumetric price. Second, Kafka's cost per GiB falls steeply with utilization, because its fixed hourly floor is being spread over more bytes. Of the Kafka path's $11.13, roughly $6.62 was that floor: cluster hours, consumer DBUs, and consumer EC2 that tick at 3 a.m. whether events arrive or not.

Projected to continuous 24/7 operation at each level's throughput, excluding Delta storage (identical on both paths and volume-driven), the gap in absolute dollars looks like this:

These projections use a steady-state hourly floor (cluster, six partitions, consumer, all at their hourly rates), not the campaign's measured idle overhead.

One footnote required for intellectual honesty: the 20.9× figure charges Kafka for the cluster-hours the campaign actually consumed, including provisioning, one aborted first attempt, and drain time around the measured hours. That overhead was real money on our bill, but its size is specific to how this campaign stood infrastructure up and down; don't read it as a universal tax on short-lived workloads. Model a zero-idle steady state instead and L1 lands near 14× rather than 21×. The repo documents both models; the measured campaign is the range we can put on a bill, the zero-idle figure is a projection.

Why there's no crossover

The obvious rebuttal: "sure, at low utilization serverless-anything beats a cluster; push the throughput high enough and Kafka wins." We expected to find that crossover point. It doesn't exist on this architecture.

Strip away Kafka's entire fixed floor. Pretend the cluster and the consumer are free. What's left is MSK Serverless's per-byte tolls, $0.10/GB in and $0.05/GB out at list, and the broker meters both directions because the consumer has to read every byte back out to land it in Delta. Over our campaign, those tolls worked out, measured from the bill, to $0.159 per GiB landed in the table, and that's with lz4 compression already working in Kafka's favor. Zerobus's entire ingest cost over the same landed GiB is $0.069 (both figures ex-storage). The ratio asymptotes at roughly 2.3× as utilization approaches infinity; it never reaches 1×. On MSK Serverless, the broker's per-byte charges alone cost more than twice what Zerobus charges for the whole job, before you pay anyone to consume the topic.

A provisioned or self-managed Kafka cluster driven hard at high sustained utilization is the one configuration that could plausibly close this gap. We didn't measure that scenario, and we'd genuinely like to see someone do it. But note what it implies: recovering Kafka's economics requires taking on more operational surface, not less.

What Kafka bought for the money: about 1.5 seconds

Cost isn't the only axis, and here the trophy changes hands:

Kafka came in roughly a second and a half faster at the median, consistently across load levels. Two qualifications, because the two rows above are not the same measurement:

The clocks stop at different milestones. Kafka's figures are per-row, from producer event time to a timestamp the Spark consumer stamps while processing each micro-batch, which is before the Delta commit completes. Zerobus has no consumer to stamp anything, so we reconstructed its latency from the service's own commit records (system.lakeflow.zerobus_ingest): its clock stops at the Delta commit that makes rows queryable. Kafka's number therefore slightly understates time-to-queryable, and the true visibility gap is somewhat smaller than the raw 1.5 seconds.
Zerobus's latency is a batching interval, not transport. We observed the service committing to the table every ~5.0 seconds at all three rates (723, 723, and 722 commits in the three one-hour runs), so end-to-end latency is roughly half a commit interval plus overhead, and it stayed flat as load grew. The cadence itself is our observation, not a documented contract. Databricks' current docs target a median time-to-table of ≤ 5 seconds; our measured P50 of 2.9 to 3.1 s sits comfortably inside it.

If your consumers are dashboards and downstream jobs, ~3 seconds is indistinguishable from ~1.5. If you're doing operational alerting where every second counts, this is the line item where the Kafka path earns some of its premium back.

Reliability was a tie worth reporting: 64.8 million records, zero failed sends, zero duplicates, on both paths. One asterisk for the record: Zerobus documents at-least-once delivery, so our zero-duplicate count is an observed result for this campaign, not an exactly-once guarantee.

The cost you can't put in a CSV

The benchmark tracks dollars, but the asymmetry you'd live with daily is operational, and will be countable in the repo itself. The Kafka path deploys an MSK cluster plus ten more Kafka-only Terraform resources (a broker security group and its rules, IAM roles and policies for producer and consumer), an always-on Spark streaming job with checkpoint management, topic administration, and MSK IAM-auth client configuration on both ends. The Zerobus path deploys a Delta table, a service-principal grant, and an endpoint URL.

That's a real difference in deployment surface, in the permissions you maintain, in what you monitor, and in how many distinct components can fail on a Saturday. It is not zero-ops on the Zerobus side: your producer still owns acknowledgement handling, retries and reconnection, and keeping its protobuf schema in lockstep with the target table. Ours carries bounded-backoff reconnect logic for exactly that reason; over 32 million records it never fired, but you still have to write it. What you stop owning is the distributed system in the middle.

What this benchmark does not say

Benchmarks earn trust by stating their edges, so, clearly:

This is single-sink ingestion only. Multi-consumer fan-out, replay, long retention, stream processing between producer and sink, the workloads Kafka was actually built for, were not tested, and nothing here argues Zerobus replaces Kafka there. If two teams read the same topic, the calculus changes fundamentally.
MSK Serverless is not all of Kafka. Provisioned MSK or self-managed Kafka at high sustained utilization was not measured and is the most plausible path to closing the cost gap.
The top rate was bounded by our client, not the services. Our single-threaded Python producer topped out near 7,400 records/s, so the ladder stops at 6,000. Neither service showed errors or pushback at any tested rate; read no throughput ceiling into these numbers.
One region (us-east-1), one Databricks tier (Enterprise), measured 2026-07-31.

So, do you still need Kafka?

If events fan out to multiple consumers, need replay, or feed stream processing: yes, and this benchmark has nothing to say against it. That's the job Kafka is for.

But if you're maintaining a broker-plus-consumer pipeline whose only function is landing events in a Delta table, you're paying somewhere between 3.6× and 21× the going rate, depending on utilization and which cost model fits your deployment, for about a second and a half of headline latency. And you're operating a distributed system for the privilege. For that pattern, and our experience suggests it's a surprisingly large share of the streaming estate, a direct write to the lakehouse is no longer the exotic option. It's the default that the alternative should have to argue against.

Measured 2026-07-31 in AWS us-east-1 at Databricks Enterprise-tier list prices. Every vendor rate cited here was re-verified against the official AWS and Databricks pricing pages on 2026-08-04. Cloud pricing drifts, verify current rates before extrapolating to your own bill.

Databricks Zerobus vs. Kafka: Do you still need Kafka in front of your Lakehouse?