Kafka is evolving to remove its dependency on Zookeeper. The Kafka Improvement Proposal 500 (KIP-500) aims to manage Kafka's metadata log with a self-managed Raft consensus algorithm and controller quorum rather than relying on Zookeeper. This will improve scalability, robustness, and make deployment easier. It will take multiple releases to fully implement KIP-500, beginning with removing Zookeeper from clients and ending with a release where Zookeeper is no longer required.

1. 1
Kafka Needs No Keeper
Colin McCabe

2. InfoQ.com: News & Community Site
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
kafka-zookeeper/

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. 2
● Kafka has gotten its mileage out of Zookeeper
● But it is still a second system
● KIP-500 has been adopted by the community
● This is not a 1-1 replacement
● We’ve been headed this direction for years
Introduction

5. 3
Evolution of Apache
Kafka Clients

6. 4
Producer
Consumer
Admin Tools

7. 5
write to
topics
Producer
Consumer
Admin Tools

8. 6
write to
topics
read from
topics
Producer
Consumer
Admin Tools

9. 7
write to
topics
read from
topics
offset
fetch/commit
group partition
assignment
Producer
Consumer
Admin Tools

10. 8
write to
topics
read from
topics
offset
fetch/commit
group partition
assignment
topic
create/delete
Producer
Consumer
Admin Tools

11. 9
Consumer Group Coordinator

12. 10
Consumer
offset
fetch/commit
group partition
assignment
read from
topics

13. 11
Consumer
offset
fetch/commit
group partition
assignment
read from
topics
Consumer APIs
● Fetch

14. 12
Consumer
offset
fetch/commit
group partition
assignment
read from
topics
Consumer APIs
● Fetch

15. 13
Consumer
Consumer APIs
● Fetchoffset
fetch/commit
group partition
assignment
read from
topics
__offsets

16. 14
offset
fetch/commit
Consumer
group partition
assignment
read from
topics
Consumer APIs
● Fetch
● OffsetCommit
● OffsetFetch
__offsets

17. 15
Consumer
group partition
assignment
read from
topics
offset
fetch/commit Consumer APIs
● Fetch
● OffsetCommit
● OffsetFetch
__offsets

18. 16
Consumer
group partition
assignment
read from
topics
offset
fetch/commit Consumer APIs
● Fetch
● OffsetCommit
● OffsetFetch
__offsets

19. 17
group partition
assignment
Consumer
read from
topics
offset
fetch/commit Consumer APIs
● Fetch
● OffsetCommit
● OffsetFetch
● JoinGroup
● SyncGroup
● Heartbeat
__offsets

20. 18
Consumer
read from
topics
offset
fetch/commit
group partition
assignment
Consumer APIs
● Fetch
● OffsetCommit
● OffsetFetch
● JoinGroup
● SyncGroup
● Heartbeat
__offsets

21. 19
Consumer
read from
topics
offset
fetch/commit
group partition
assignment
Consumer APIs
● Fetch
● OffsetCommit
● OffsetFetch
● JoinGroup
● SyncGroup
● Heartbeat
__offsets

22. 20
read from
topics
offset
fetch/commit
group partition
assignment
Consumer
Consumer APIs
● Fetch
● OffsetCommit
● OffsetFetch
● JoinGroup
● SyncGroup
● Heartbeat
__offsets

23. 21
Consumer
Producer
Admin Tools
create/delete
topics

24. 22
Kafka Security
and the
Admin Client

25. 23
Consumer
Producer
create/delete
topics
Admin Tools

26. 24
ACL Enforcement
create/delete
topics
Admin Tools
Consumer
Producer

27. 25
create/delete
topics
ACL Enforcement
Admin Tools
Consumer
Producer

28. 26
create/delete
topics
ACL Enforcement
Admin Tools

29. 27
AdminClient
Admin Tools
ACL Enforcement
create/delete
topics

30. 28
AdminClient
Admin Tools
ACL Enforcement
create/delete
topics
Admin APIs:
● CreateTopics
● DeleteTopics
● AlterConfigs
● ...

31. 29
Admin APIs:
● CreateTopics
● DeleteTopics
● AlterConfigs
● ...
AdminClient
Admin Tools
ACL Enforcement

32. 30
Producer
Consumer
AdminClient
Client APIs:
● Produce
● Fetch
● Metadata
● CreateTopics
● DeleteTopics
● ...

33. 31
Producer
Consumer
AdminClient
Client APIs:
● Produce
● Fetch
● Metadata
● CreateTopics
● DeleteTopics
● ...
● Encapsulation
● Security
● Validation
● Compatibility

34. 32
Inter Broker
Communication

35. 33

36. 34
Broker Registration
ACL Management
Dynamic Configuration
ISR Management

37. 35
Controller
Broker Registration
ACL Management
Dynamic Configuration
ISR Management

38. 36
Controller
Broker Registration
ACL Management
Dynamic Configuration
ISR Management
Controller Election

39. 37
Controller
Broker Registration
ACL Management
Dynamic Configuration
ISR Management
Controller Election

40. 38
Controller
Broker Registration
ACL Management
Dynamic Configuration
ISR Management
Controller Election

41. 39
Controller Controller APIs:
● LeaderAndIsr
● UpdateMetadata
● StopReplica
Leader/ISR Push
Update Metadata
Stop/Delete Replica
Broker Registration
ACL Management
Dynamic Configuration
ISR Management
Controller Election

42. 40
Controller Controller APIs:
● LeaderAndIsr
● UpdateMetadata
● StopReplica
Leader/ISR Push
Update Metadata
Stop/Delete Replica
Broker Registration
ACL Management
Dynamic Configuration
ISR Management
Controller Election

43. 41
Controller Controller APIs:
● LeaderAndIsr
● UpdateMetadata
● StopReplica
Leader/ISR Push
Update Metadata
Stop/Delete Replica
Broker Registration
ACL Management
Dynamic Configuration
ISR Management
Controller Election

44. 42
Controller Controller APIs:
● LeaderAndIsr
● UpdateMetadata
● StopReplica
● AlterIsr
Broker Registration
ACL Management
Dynamic Configuration
ISR Management
Controller Election
Leader/ISR Push
Update Metadata
Stop/Delete Replica

45. 43
Controller
Leader/ISR Push
Update Metadata
Stop/Delete Replica
ISR Management
Controller APIs:
● LeaderAndIsr
● UpdateMetadata
● StopReplica
● AlterIsr
Broker Registration
ACL Management
Dynamic Configuration
ISR Management
Controller Election

46. 44

47. 45
● Encapsulation
● Compatibility
● Ownership

48. 46
Broker Liveness

49. 47
Zk Session

50. 48
/brokers/1 -> {
host: 10.10.10.1:9092
rack: rack-1
}

51. 49
/brokers/1 -> {
host: 10.10.10.1:9092
rack: rack-1
}

52. 50

53. 51
Watch trigger

54. 52
Watch trigger
Broker 1 is offline

55. 53
Network Partition
Resilience

56. 54

57. 55
Case 1: Total partition

58. 56
Case 2: Broker partition

59. 57
Case 3: Zk Partition

60. 58
Case 4: Controller partition

61. 59
Metadata Inconsistency

62. 60

63. 61
Metadata Source
of Truth

64. 62
Metadata Source
of Truth
Metadata Cache
- sync writes
- async updates

65. 63
Metadata Source
of Truth
Metadata Cache
- async update
Metadata Cache
- sync writes
- async updates
Metadata Cache
- async update

66. 64

67. 65

68. 66

69. 67
Last Resort:
> rmr /controller

70. 68
Last Resort:
> rmr /controller
New controller!

71. 69
Last Resort:
> rmr /controller
Load ALL
Metadata

72. 70
Last Resort:
> rmr /controller
Load ALL
Metadata

73. 71
Last Resort:
> rmr /controller
Push ALL
Metadata

74. 72
Last Resort:
> rmr /controller
Push ALL
Metadata

75. 73
Last Resort:
> rmr /controller
Push ALL
Metadata
How do you know the metadata
has diverged?

76. 74
Performance of Controller
Initialization

77. 75

78. 76

79. 77
New controller!

80. 78
Load ALL
Metadata

81. 79
Load ALL
Metadata
Complexity: O(N)
N = number of partitions

82. 80

83. 81
Push ALL
Metadata

84. 82
Push ALL
Metadata
Complexity: O(N*M)
N = number of partitions
M = number of brokers

85. 83
Metadata as an
Event Log

86. 8484
Metadata as an
Event Log
- Each change becomes a
message
- Changes are propagated
to all brokers
...
924 Create topic ”foo”
925 Delete topic “bar”
926 Add node 4 to the cluster
927 Create topic “baz”
928 Alter ISR for “foo-0”
929 Add node 5 to the cluster

87. 8585
Metadata as an
Event Log
- Clear ordering
- Can send deltas
- Offset tracks consumer
position
- Easy to measure lag
...
924 Create topic ”foo”
925 Delete topic “bar”
926 Add node 4 to the cluster
927 Create topic “baz”
928 Alter ISR for “foo-0”
929 Add node 5 to the cluster

88. 86
Consumer
Consumer
Consumer
offset=3
offset=1
offset=2

89. 87
offset=3
offset=1
offset=2
Broker
Broker
Broker
?

90. 88
offset=3
offset=1
offset=2
Broker
Broker
Broker
Controller

91. 89
Can we use the existing Kafka log
replication protocol?
- How do we elect the leader?
We need a self-managed quorum.
Implementing
the Controller
Log

92. 90
Can we use the existing Kafka log
replication protocol?
- How do we elect the leader?
We need a self-managed quorum.
Implementing
the Controller
Log
Enter Raft.
Leader election
is by simple
majority.

93. 91
Kafka Raft
Writes Single Leader Single Leader
Fencing Monotonically increasing
epoch
Monotonically increasing
term
Log reconciliation Offset and epoch Term and index
Push/Pull Pull Push
Commit Semantics ISR Majority
Leader Election From ISR through
Zookeeper
Majority

94. 92
The Controller Quorum

95. 93
offset=1
offset=2
Broker
Broker
Controller
Controller
Controller
The Controller Raft Quorum
- The leader is the active controller
- Controls reads / writes to the log
- Typically 3 or 5 nodes, like ZK

96. 94
offset=1
offset=2
Broker
Broker
Controller
Controller
Controller
Instant Failover
- Low-latency failover via Raft election
- Standbys contain all data in memory
- Brokers do not need to re-fetch

97. 95
/mnt/logs/kafka/metadata
offset=1
Broker
Broker
Controller
Controller
Controller
Metadata Caching
- Brokers can persist metadata to disk
- Only fetch what they need
- Use snapshots if we’re too far behind
/mnt/logs/kafka/metadata
offset=2

98. 96
Broker Registration
- Building a map of the
cluster
- What brokers exist in
the cluster?
- How can they be
reached?
Controller

99. 97
Broker Registration
- Brokers send
heartbeats to the active
controller
- The controller uses this
to build a map of the
cluster
Controller

100. 98
Controller
Broker Registration
- Brokers send
heartbeats to the active
controller
- The controller uses this
to build a map of the
cluster
- The controller also tells
brokers if they should
be fenced or shut down

101. 99
Controller
Fencing
- Brokers need to be
fenced if they’re
partitioned from the
controller, or can’t keep
up
- Brokers self-fence if
they can’t talk to the
controller

102. 100
Handling network
partitions

103. 101
Case 1: Total partition

104. 102
Case 1: Total partition

105. 103
Case 2: Broker
partition

106. 104
Case 3:
Controller
partition

107. 105
Case 3:
Controller
partition

108. 106
Deployment
Current KIP-500
Configuration File Kafka and
ZooKeeper
Kafka
Metrics Kafka and ZK Kafka
Administrative
Tools
ZK Shell, Four
letter words,
Kafka tools
Kafka tools
Security Kafka and ZK Kafka

109. 107
Shared
Controller
Nodes
- Fewer resources
used
- Single node
clusters
(eventually)

110. 108
Separate
Controller
Nodes
- Better resource
isolation
- Good for big
clusters

111. 109
Roadmap

112. 110
Remove
Client-side ZK
dependencies
Remove
Broker-side ZK
dependencies
Controller
Quorum

113. 111
Remove
Client-side ZK
dependencies
Remove
Broker-side ZK
dependencies
Controller
Quorum
Incremental KIP-4
Improvements
- Create new APIs
- Deprecate direct ZK
access

114. 112
Remove
Client-side ZK
dependencies
Remove
Broker-side ZK
dependencies
Controller
Quorum
Broker-Side Fixes
- Remove deprecated
direct ZK access for
tools
- Create broker-side
APIs
- Centralize ZK access
in the controller

115. 113
Remove
Client-side ZK
dependencies
Remove
Broker-side ZK
dependencies
Controller
Quorum
First Release
without ZooKeeper
- Raft
- Controller quorum

116. 114
Upgrade
Issues
- Tools using ZK
- Brokers
accessing ZK
- State in ZK
KIP-500 Release
Older Kafka Release

117. 115
Bridge Release
KIP-500 Release
Older Kafka Release
Bridge
Release
- No ZK access
from tools,
brokers
(except
controller)

118. 116
Upgrading
- Starting from the
bridge release

119. 117
Upgrading
- Start new controller
nodes (possibly
combined)
- Quorum elects leader
- Claims leadership in
ZK

120. 118
Upgrading
- Roll nodes one by
one as usual
- Controller continues
sending
LeaderAndIsr, etc. to
old nodes

121. 119
Upgrading
- When all brokers
have been rolled,
decommission ZK
nodes

122. 120
Conclusion

123. 121
Apache ZooKeeper has served us well
- KIP-500 is not a 1:1 replacement, but a different
paradigm
We have already started removing ZK from clients
- Consumer, AdminClient
- Improved encapsulation, security, upgradability

124. 122
Metadata should be managed as a log
- Deltas, ordering, caching
- Controller Failover, Fencing
- Improved scalability, robustness, easier deployment
The metadata log must be self-managed
- Raft
- Controller quorum

125. 123
It will take a few releases to implement KIP-500
- Additional KIPs for APIs, Raft, Metadata, etc.
Rolling upgrades will be supported
- Bridge release
- Post-ZK release
Kafka needs no Keeper

126. 124
cnfl.io/meetups cnfl.io/blog cnfl.io/slack
THANK YOU
Colin McCabe
cmccabe@confluent.io

127. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
kafka-zookeeper/