This document provides an overview of Apache Pulsar: - It introduces Apache Pulsar and shares some stats on its adoption and contributors. - It describes Pulsar's architecture including brokers, Zookeeper, BookKeeper, topics, and subscribers. - It explains how Pulsar stores data across tenants, namespaces, bundles, ledgers and topics to enable features like multi-tenancy, load balancing, and geo-replication.

2. Others?

3. •
•
•
•
•
•
•
•
•
linkedin.com/in/shivjijha/
youtube.com/@ShivjiKumarJha
slideshare.net/shiv4289

4. Introduce Apache Pulsar
Some numbers running 4 years in
Production
Components under the hood
Read & Write Paths

6. SUBSCRIBER
TOPIC
SUBSCRIBER
SUBSCRIBER
PUBLISHER

7. 600+
contributors
9800+ Slack
members
1000+ orgs
using Pulsar
2 BLR
meetups*

8. Open Source
Apache 2.0
License
Unified
Streaming &
Queuing
Multi-
tenancy
Built for the
cloud
Isolation
Options
Tiered
Storage
Geo
Replication
Auto Load
Balancing

10. LAST 2 DAYS

11. LAST 7 DAYS

13. APP 1 APP 2

14. PUBLISH CONSUME
BROKER

15. PUBLISH CONSUME
BROKER
ZOOKEEPER

16. PULSAR
PUBLISH CONSUME
BROKER
ZOOKEEPER
BOOKKEEPER

17. PULSAR
PUBLISH CONSUME
BROKER
ZOOKEEPER
BOOKKEEPER

18. PULSAR
PUBLISH CONSUME
BROKER (TOPIC CACHE)
ZOOKEEPER (COORDINATION)
BOOKKEEPER (DATA)

20. https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works

21. BOOKKEEPER
CLIENT

22. BOOKKEEPER
CLIENT
JOURNAL
ENTRY <L1
E1>
ENTRY <L1 E2>
ENTRY<L2 EO>
ENTRY<L2 E1>
ENTRY <L1 E3>
ENTRY <L2 E2>
ENTRY <L3
EO>
JOURNAL
Synchronous write
Legend
WRITE CACHE

23. BOOKKEEPER
CLIENT
JOURNAL
LEDGER
ENTRY <L1
E1>
ENTRY <L1 E2>
ENTRY<L2 EO>
ENTRY<L2 E1>
ENTRY <L1 E3>
ENTRY <L2 E2>
ENTRY <L3
EO>
WRITE CACHE
JOURNAL
LEDGER
Synchronous write Asynchronous Write
Legend

24. https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works

25. APP
(PULSAR CLIENT)
JOURNAL
READ
CACHE
LEDGER
INDEX

26. Open Source
Apache 2.0
License
Unified
Streaming &
Queuing
Multi-
tenancy
Built for the
cloud
Isolation
Options
Tiered
Storage
Geo
Replication
Auto Load
Balancing

27. 1. UNDERSTANDING HOW APACHE PULSAR WORKS BY JACK VANLIGHTLY
2. PULSAR ARCHITECTURE PERFORMANCE TUNING - STREAMNATIVE BLOG
3. HOW PULSAR STORES YOUR DATA – PULSAR SUMMIT NA 2021
4. BOOKKEEPER AS A KEY VALUE STORE AND ITS USE CASES – APACHE FOUNDATION
5. KEEPING ON TOP OF HYBRID CLOUD USE CASES – PULSAR USAGE AT NUTANIX
6. PULSAR VS KAFKA, NATS AND KINESIS – PULSAR SUMMIT ASIA 2022
7. APACHE PULSAR DESIGN CHOICES AND USE CASE IMPLICATIONS
8. TGI Pulsar Playlist by Sijie Guo
9. https://pulsar.apache.org/docs

28. • PULSAR COMMUNITY
• HTTPS://APACHE-PULSAR.SLACK.COM/
• USERS@PULSAR.APACHE.ORG
• DEV@PULSAR.APACHE.ORG
•
linkedin.com/in/shivjijha/
youtube.com/@ShivjiKumarJha
slideshare.net/shiv4289
Drop me a message on

30. Streaming features
Data processing in order
Ingestion of large amounts of data
Data retention
Handle big catch-up reads
Message queuing features
Add/remove consumers dynamically
Do not block the queue when a consumer is busy or fails
to consume one message
Redeliver ‘failed’ messages
Scheduled or delayed delivery
is the best of both worlds!

31. PULSAR DATA STORES :
DISTRIBUTION
tenant
namespace
bundle1 bundle2 bundle3
T
O
P
I
C
1
T
O
P
I
C
2
T
O
P
I
C
3
T
O
P
I
C
4
T
O
P
I
C
5
T
O
P
I
C
6
Shards (aka namespace Bundles) for load balancing
31

32. tenant
namespace
bundles
Shards (aka namespace Bundles) for load balancing
ledgers
topic
cursors
managed ledger
32
PULSAR DATA STORES :
DISTRIBUTION

33. tenant
namespace
bundles
Shards (aka namespace Bundles) for load balancing
ledgers
topic
cursors
managed ledger
33
PULSAR DATA STORES :
DISTRIBUTION