Kafka has dominated event streaming for a decade. It processes trillions of messages daily across thousands of companies. Its dominance created an ecosystem so large that “streaming” became synonymous with “Kafka.” Two challengers — Redpanda and Pulsar — have earned their place by making different bets on what Kafka gets wrong.

Redpanda thinks Kafka’s problem is operational complexity. Pulsar thinks Kafka’s problem is architectural rigility. Both are right about their specific complaint. Whether either is right for your workload depends on which kind of pain you are currently experiencing.

Kafka: The Standard

Kafka’s architecture is well-known: a distributed commit log partitioned across brokers, with producers writing to partitions and consumers reading from them. The design is simple in concept and complex in operation. ZooKeeper (now replaced by KRaft in recent versions) manages cluster metadata. Replication ensures durability. Consumer groups manage parallel consumption.

Kafka’s strength is its ecosystem. Kafka Connect provides hundreds of connectors to external systems. Kafka Streams offers stream processing without a separate framework. Schema Registry manages data contracts. ksqlDB provides SQL-based stream processing. The ecosystem around Kafka is larger than everything else combined.

The operational pain is equally well-documented. Managing a Kafka cluster requires understanding partition rebalancing, replication factors, ISR (in-sync replica) management, log compaction, and consumer group coordination. Upgrades require careful planning. Partition reassignment during scaling can cause performance degradation. The tuning parameters number in the hundreds, and the interactions between them are non-obvious.

Confluent Cloud and other managed Kafka offerings reduce this burden, but managed Kafka is expensive. At high throughput, Confluent Cloud bills can exceed the cost of self-managed Kafka by a factor of two or three. The convenience tax is real.

Kafka remains the right choice when you need the ecosystem, when your team already knows it, and when your throughput requirements are well within its capabilities. For most production streaming workloads, Kafka is the proven option.

Redpanda: Kafka-Compatible, C++-Native

Redpanda is a Kafka-compatible streaming platform written in C++. It speaks the Kafka protocol, so existing Kafka producers, consumers, and connectors work without modification. The compatibility is deep enough that migrating from Kafka to Redpanda is often as simple as changing a connection string.

The architectural difference is significant. Redpanda replaces ZooKeeper (and KRaft) with its own Raft-based consensus engine. It uses a thread-per-core architecture that eliminates the JVM garbage collection pauses that affect Kafka under high load. The result is more predictable latency — not necessarily faster in average-case benchmarks, but with tighter latency variance under load.

Redpanda’s operational model is simpler than Kafka’s. A single binary, no JVM tuning, no separate ZooKeeper cluster, and fewer configuration knobs. The reduced complexity translates directly to reduced on-call burden. Teams that migrate from self-managed Kafka to Redpanda consistently report spending less time on cluster operations.

The trade-off is ecosystem depth. Redpanda supports the Kafka protocol, so Kafka Connect connectors work. But Redpanda’s own ecosystem — its managed cloud, its stream processing capabilities, its tooling — is smaller than Kafka’s. If you rely on Confluent-specific features (ksqlDB, Confluent Schema Registry’s specific capabilities, Confluent Cloud’s managed connectors), the migration path is not as clean as the compatibility story suggests.

Redpanda’s single-binary architecture has a scaling ceiling. For most workloads — up to tens of gigabytes per second — Redpanda handles the load well. At the extreme end (hundreds of gigabytes per second, thousands of partitions), Kafka’s multi-process architecture and Confluent’s managed scaling may handle the load more gracefully.

Redpanda’s tiered storage feature (offloading older data to S3) reduces storage costs and simplifies retention management. This is a genuine advantage over self-managed Kafka, where tiered storage requires additional tooling.

Pulsar: Separated Compute and Storage

Pulsar takes a different architectural approach. Instead of coupling compute and storage on the same nodes (as Kafka and Redpanda do), Pulsar separates them. Brokers handle the serving layer — producing and consuming messages. BookKeeper handles the storage layer — writing and replicating message data. This separation means you can scale compute and storage independently.

The practical benefits of this separation are clearest in multi-tenant environments. If you run streaming for multiple teams with different throughput requirements, Pulsar’s architecture lets you allocate broker capacity per tenant without worrying about storage distribution. Kafka’s partition-based model ties compute and storage together, making multi-tenancy harder to manage.

Pulsar’s topic hierarchy and namespace model are more flexible than Kafka’s flat topic namespace. You can organize topics into namespaces and tenants, set retention and TTL policies at each level, and manage access control at the namespace level. For organizations that run a shared streaming platform for many teams, this organizational model is a real advantage.

Pulsar also offers features that Kafka lacks natively: geo-replication across data centers is built in (not a paid Confluent feature), tiered storage is first-class, and Pulsar Functions provide lightweight stream processing without a separate framework.

The cost of this architecture is operational complexity. Pulsar has more components to manage than Kafka or Redpanda: brokers, BookKeeper nodes, ZooKeeper (for metadata), and the proxy layer. Deploying Pulsar on Kubernetes is more involved because each component needs its own StatefulSet, configuration, and monitoring.

Pulsar’s ecosystem is the smallest of the three. The Kafka protocol compatibility layer (KoP) allows Kafka clients to connect to Pulsar, but the compatibility is not as deep as Redpanda’s native Kafka support. If you depend on Kafka Connect connectors, you may encounter edge cases where the compatibility layer does not behave identically to native Kafka.

Pulsar is the right choice for multi-tenant streaming platforms, geo-replicated deployments, and teams that need to scale compute and storage independently. It is the wrong choice for teams that want simplicity — Pulsar’s architecture trades operational simplicity for architectural flexibility.

Latency Profile

For applications where tail latency matters — real-time fraud detection, live bidding, interactive analytics — the latency profile of each tool is a key differentiator.

Kafka’s latency is good on average but has long tails due to JVM garbage collection, partition leader elections, and replication delays. P99 latency can be several times P50 latency under load. Tuning Kafka for low tail latency requires deep expertise.

Redpanda’s thread-per-core architecture eliminates GC pauses, producing the tightest tail latency of the three. P99 latency is much closer to P50 latency, making Redpanda the best choice for latency-sensitive applications.

Pulsar’s separated architecture introduces an additional network hop (broker to BookKeeper), which adds baseline latency. For throughput-oriented workloads where latency is measured in hundreds of milliseconds, this is irrelevant. For latency-sensitive workloads where every millisecond counts, the extra hop is a disadvantage.

Decision Framework

Use Kafka when your team already operates it, you rely on the Confluent ecosystem, and your throughput requirements are well within its capabilities. Managed Kafka (Confluent Cloud, Amazon MSK) reduces operational pain but increases cost. Kafka is the safe choice when the ecosystem matters.

Use Redpanda when you want Kafka compatibility with lower operational overhead and tighter latency profiles. Best for teams migrating from Kafka who are tired of JVM tuning and ZooKeeper management. Best for latency-sensitive workloads where tail latency matters.

Use Pulsar when you run multi-tenant streaming platforms, need built-in geo-replication, or must scale compute and storage independently. Best for platform teams that serve multiple internal consumers with different requirements. Wrong choice for teams that prioritize operational simplicity.

The most common mistake is choosing Pulsar for its architectural elegance when Kafka or Redpanda would handle the workload with less operational effort. The second most common mistake is staying on self-managed Kafka when Redpanda would cut your on-call burden in half with no functional regression. Evaluate based on your actual operational pain, not on architectural diagrams.