Apache Pulsar

Overview

Apache Pulsar is an open-source platform for messaging and event streaming. Applications publish messages to topics, and other applications subscribe to those topics to receive them. Pulsar supports both styles of messaging that systems usually need: streaming, where many readers consume an ordered, replayable log, and queuing, where a pool of workers shares the messages so each one is handled once.

Pulsar's defining design choice is to separate serving from storage. Brokers accept and deliver messages but keep no permanent data. Messages are stored in Apache BookKeeper, a distributed log storage system, on separate nodes called bookies. Because brokers are stateless, you can add or replace them quickly, and topics can move between brokers without copying data. Pulsar also builds in multi-tenancy, geo-replication between clusters and tiered storage that moves older data to object storage.

An analogy: a post office where the counter staff and the storage vaults are separate. Clerks at the counter (brokers) take in and hand out parcels but keep nothing behind the desk. Parcels go into a vault system (BookKeeper) where each is copied into several vaults. If a clerk goes home sick, another clerk takes over the queue at once, because the parcels never lived at that desk.

What it is

Topic names show the hierarchy, for example persistent://retail/orders/payments: tenant retail, namespace orders, topic payments.

Where it came from

Pulsar was developed at Yahoo, where it went into service around 2015 as a shared messaging platform for many internal applications. Yahoo open-sourced it in 2016, and the project entered the Apache Incubator in June 2017. The Apache Software Foundation announced Pulsar as a top-level project on 25 September 2018.

The project later introduced long-term support releases. Pulsar 3.0, in May 2023, was the first LTS version, and Pulsar 4.0, announced in October 2024, was the second. Version 4.0 made more components pluggable, added support for the Oxia metadata store, moved its container images to Java 21 and added OpenTelemetry metrics. Pulsar is released under the Apache License 2.0.

How it works

Publishing

1. A producer connects to any broker and looks up which broker owns the topic, or each partition of a partitioned topic.
2. The owning broker receives messages and appends them to the topic's current ledger in BookKeeper.
3. Each entry is written to several bookies in parallel. The write is acknowledged once a configured number of bookies (the ack quorum) confirm it.
4. The broker acknowledges the producer. Ledgers are closed and new ones opened as they grow or when ownership changes.

Consuming

A consumer subscribes to a topic with a subscription name and a type. Pulsar keeps a cursor per subscription that records which messages have been acknowledged. With an exclusive subscription only one consumer reads. With failover, one consumer reads and another takes over if it fails. With shared, messages are spread round-robin across consumers, like a work queue. With key-shared, messages with the same key always go to the same consumer, preserving per-key order while spreading load.

Consumers acknowledge messages individually or cumulatively. Unacknowledged messages are redelivered, and repeatedly failing messages can be sent to a dead-letter topic.

Storage and retention

Because storage is separate, a topic's data is spread across many bookies and is not tied to one broker's disk. Retention policies decide how long acknowledged messages are kept. Tiered storage can move older ledgers to object storage such as S3, so topics can hold long histories cheaply while brokers keep serving recent data quickly.

Multi-tenancy and replication

Tenants and namespaces carry policies for authentication, authorisation, quotas, retention and replication. Geo-replication copies messages between Pulsar clusters in different regions, configured per namespace or topic.

import pulsar
client = pulsar.Client("pulsar://localhost:6650")
producer = client.create_producer("persistent://public/default/orders")
producer.send(b'{"order_id": 42}')
consumer = client.subscribe("persistent://public/default/orders", "billing",
                            consumer_type=pulsar.ConsumerType.Shared)

Key concepts and terms

A worked example

This scenario is illustrative. A software company runs a platform used by 40 business customers. Each customer's events must be isolated, some customers need data replicated between two regions and the company wants one shared messaging system.

1. Each customer becomes a tenant, with namespaces for events and commands. Quotas limit each tenant's publish rate, so one noisy customer cannot starve the others.
2. The events namespaces use partitioned topics with eight partitions. Analytics jobs read them through exclusive or failover subscriptions as an ordered stream.
3. The commands namespaces use shared subscriptions. A pool of 20 workers processes commands, and adding workers increases throughput immediately.
4. For a payment customer, a key-shared subscription keyed by account ID keeps each account's commands in order while spreading accounts across workers.
5. Retention keeps 30 days of events. Tiered storage moves data older than two days to object storage, reducing the bookie disk needed.
6. Two premium customers have geo-replication enabled between European and North American clusters for their namespaces only.
7. During a broker upgrade, topics move to other brokers within seconds because no data has to be copied.

Strengths and limitations

Strengths:

• Separated, stateless brokers make scaling and recovery quick.
• Built-in multi-tenancy with per-tenant and per-namespace policies.
• Both streaming and queuing semantics, including key-shared ordering.
• Native geo-replication and tiered storage.
• Supports very large numbers of topics.

Limitations:

• More moving parts: brokers, bookies and a metadata store to run and monitor.
• Smaller ecosystem and community than Kafka, with fewer third-party tools.
• Operational expertise is less widespread.
• Tuning BookKeeper quorums, retention and offloading requires care.
• Client libraries differ in feature coverage between languages.

How it compares with alternatives

Practical guidance and best practices

1. Design the tenant and namespace layout first; policies are applied at these levels.
2. Choose the subscription type for each use case: exclusive or failover for ordered streams, shared for work queues, key-shared for ordered work by key.
3. Use partitioned topics for high-throughput streams and choose message keys that spread load.
4. Set retention and backlog quotas explicitly so storage does not grow without bound.
5. Configure write and ack quorums to match your durability needs and failure domains.
6. Use tiered storage for long retention instead of large bookie disks.
7. Configure dead-letter topics and redelivery limits for consumers that can fail.
8. Monitor backlog, publish and dispatch rates, and bookie disk use, and alert on growing backlogs.

Common mistakes

• Treating Pulsar exactly like Kafka and ignoring subscription types and tenants.
• Never acknowledging messages, so backlogs grow until quotas block producers.
• Using a shared subscription where strict ordering is required.
• Under-provisioning bookies or the metadata store, which then limit the whole cluster.
• Turning on geo-replication everywhere without considering cost and conflict scenarios.
• Ignoring client version differences when adopting newer features.

Frequently asked questions

How is Pulsar different from Kafka?

In Kafka, brokers store the partitions they serve. In Pulsar, brokers are stateless and data lives in BookKeeper, so brokers scale and recover independently of storage. Pulsar also builds in multi-tenancy, queue-style shared subscriptions and geo-replication. Kafka has a larger ecosystem and more widespread operational knowledge.

Can Pulsar act as a message queue?

Yes. Shared and key-shared subscriptions distribute messages among many consumers, with individual acknowledgements, redelivery and dead-letter topics, which covers typical work-queue patterns.

What is BookKeeper?

Apache BookKeeper is a distributed, replicated log storage system. Pulsar uses it to store topic data in ledgers spread across bookie nodes, writing each entry to several bookies for durability.

Does Pulsar still need ZooKeeper?

ZooKeeper has been the traditional metadata store, but Pulsar's metadata layer is pluggable. Newer releases support alternatives, including Oxia, which Pulsar 4.0 presented as a more scalable option. Check the documentation for your version.

Can Kafka clients talk to Pulsar?

Protocol handlers exist that let Pulsar accept the Kafka protocol, but they are separate components with their own compatibility limits. Test carefully before relying on them.

What are Pulsar Functions?

Lightweight processing functions that consume from topics, apply simple logic and publish results, run inside the Pulsar ecosystem. For complex stateful processing, teams usually use an engine such as Apache Flink.

Glossary

• Ack quorum: number of bookies that must confirm a write.
• Backlog: unacknowledged messages for a subscription.
• Bundle: a group of topics that a broker owns together for load balancing.
• LTS: long-term support release line.
• Non-persistent topic: a topic whose messages are not stored.
• Oxia: a scalable metadata store supported by Pulsar.
• Pulsar IO: framework of source and sink connectors.
• Write quorum: number of bookies each entry is written to.

Summary

Apache Pulsar is a messaging and streaming platform that separates stateless brokers from BookKeeper storage. That design, together with built-in tenants, flexible subscription types, tiered storage and geo-replication, makes it a strong fit for shared, multi-tenant and multi-region messaging. The cost is more components to run and a smaller ecosystem than Apache Kafka. Compare it with Kafka, Amazon Kinesis and RabbitMQ against your needs for ordering, replay, queuing and operations, and see event-driven architecture for how it fits a wider design.

Related

It belongs to Streaming in my stack, under Big data. Others in the same group: