In the Israeli Java Community meetup, Addison Higham introduced a fantastic project written in Java called Apache Pulsar - a high throughput distributed messaging system. He also walked through the basic design principles under the hood and several challenges he faced in the Java eco-system.

1. Apache Pulsar

2. ●
●
●
●
Hi!

3. Today’s Agenda
●
○
○
○
○
●
○
○
○
○

4. An overview of
Pulsar

5. What is it?
●
○
○
○
○
●

6. History
●
●
●
●

7. Feature Overview
●
●
●
●
●
●
●

8. Core Concepts
● Producers consumers
broker
● Topics
persisitent://tenant/namespace/topic
● Messages
partition key
● Partitioned Topics
● Tenants namespaces

9. Subscriptions
●
subscription
●
○
○
○

10. Creating a client

11. Sending a message

12. Receiving a message

13. Pulsar’s
Architecture

14. ●
●
●
●
●
●
●
●
●
●
●
●
●
○
○
●

15. Bookie 0 Bookie 1 Bookie 2
Broker 0 Broker 1 Broker 2
ZooKeeper
ZooKeeper
ZooKeeper
Producer Consumer

16. Bookkeeper
Details

17. New Broker Added
p
0
p
1
p
3
p
4
s
ss
s
ss
s
ss
s
ss
s
ss
Broker 4
p
5

18. Broker Failure
p
0
p
1
p
3
p
4
s
ss
s
ss
s
ss
s
ss
s
ss
p
5

19. New Bookie Added
p
0
p
1
p
3
p
4
p
5
s
ss
s
ss
s
ss
s
ss
s
ss
Bookie
6
s
ss

20. Bookie Failure
p
0
p
1
p
3
p
4
p
5
s
ss
s
ss
s
ss
s
ss
s
ss

21. Pulsar’s advantages
and unique features

22. Architectural advantages
●
●
●

23. Tiered Storage
●
●
●
●
p
0
p
1
p
3
p
4
p
5
s
ss
s
ss
s
ss
s
s
s
ss
offloa
ding

24. Pulsar SQL
●
●
●

25. Lessons from
Bookkeeper and
Pulsar

26. Bookkeeper disk
and memory
management

27. Java Disk IO Methods
●
○
●
○
●
○

28. Write Performance Comparison
https://github.com/romromov/java-io-benchmark

29. Problems with MappedByteBuffer
● .get()/.put() is potentially blocking
● Flushes to disk may happen in the background (but can be forced)
● Caches are controlled entirely by OS, which may not be desired
when we know we would be thrashing the cache or for precisely
controlling what is hot in memory

30. Making efficient
use of disk
●
○
○
○
●
●
https://bookkeeper.apache.org/docs/4.9.2/getting-started/concepts/#data-management-in-bookies

31. Using Caches
write cache

32. Takeaways
●
●
●
●
●

33. High throughput and
low latency in
Pulsar

34. Low latency + throughput
tailing reads
●
●
●
●

35. Avoiding the slowest worker
Takeaway

36. Streaming-pull
Takeaway

37. Broker caching
Takeaway

38. Async
Takeaway

39. Pulsar’s challenges

40. Async is still relatively new

41. Cloud Native

42. Network throttling/isolation

43. Interested in
Pulsar?

44. Join the Pulsar Community
https://pulsar-summit.org/en/event/virtual-conference-2020

45. Hosted Pulsar with StreamNative Cloud

46. Questions?