Improving Streams Scalability with Transactional StateStores (KIP-892)
•
0 likes•70 views
"Kafka Streams provides a horizontally scalable framework for exactly-once stream processing. But the more data a Kafka Streams app stores, the longer the state restore times become, which can grow into massive production incidents. This talk will look at a key problem with the scalability of Kafka Streams, how it’s being resolved with KIP-892, Transactional StateStores, and future improvements that can be made to further improve the situation."
Recommended
Recommended
More Related Content
Similar to Improving Streams Scalability with Transactional StateStores (KIP-892)
Similar to Improving Streams Scalability with Transactional StateStores (KIP-892) (20)
More from HostedbyConfluent
More from HostedbyConfluent (20)
Recently uploaded
Recently uploaded (20)
Improving Streams Scalability with Transactional StateStores (KIP-892)
- 1. Improving Streams Scalability with Transactional StateStores (KIP-892) Nick Telford
- 9. 46 sub-topologies 579 Processors 45 StateStores
- 10. 1. The Problem
- 11. 1. The Problem 2. The Solution
- 12. 1. The Problem 2. The Solution 3. Limitations
- 13. 1. The Problem 2. The Solution 3. Limitations 4. Future Work
- 19. Transaction
- 29. Commit Log Structure Random Access Structure
- 30. Commit Log Structure Random Access Structure
- 31. Commit Log Structure Random Access Structure
- 32. Random Access Structure Commit Log Structure
- 34. StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic
- 35. StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic
- 36. StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic
- 37. StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic
- 38. StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic
- 39. StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic
- 40. StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic
- 41. Restore Time
- 42. Changelog Size
- 46. Transient Network Problems Misconfigured Clients
- 47. Transient Network Problems Misconfigured Clients Bugs
- 48. Restore Time
- 49. Restore Time
- 51. DISCLAIMER
- 52. StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic
- 53. StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic
- 54. StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic Transaction Buffer
- 55. commit StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic Transaction Buffer
- 56. StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic Transaction Buffer
- 58. StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic Transaction Buffer
- 59. StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic Transaction Buffer Interactive Query Interactive Query Interactive Query
- 61. READ_UNCOMMITTED StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic Interactive Query
- 62. READ_UNCOMMITTED StateStore Kafka Streams Thread Source Topic Output Topic Interactive Query Changelog Topic
- 63. StateStore Kafka Streams Thread Source Topic Output Topic Transaction Buffer Interactive Query READ_COMMITTED Changelog Topic
- 64. StateStore Kafka Streams Thread Source Topic Output Topic Interactive Query READ_COMMITTED commit Transaction Buffer Changelog Topic
- 65. processing.guarantee READ_UNCOMMITTED READ_COMMITTED at-least-once O(1) O(1) Restore time complexity
- 66. processing.guarantee READ_UNCOMMITTED READ_COMMITTED at-least-once O(1) O(1) exactly-once(-v2|-beta ) O(n) O(1) Restore time complexity
- 67. processing.guarantee READ_UNCOMMITTED READ_COMMITTED at-least-once O(1) O(1) exactly-once(-v2|-beta ) O(n) O(1) Restore time complexity
- 68. processing.guarantee commit.interval.ms default.state.isolation.level at-least-once 30 seconds READ_UNCOMMITTED Default Configuration
- 69. processing.guarantee commit.interval.ms default.state.isolation.level at-least-once 30 seconds READ_UNCOMMITTED exactly-once(-v2|-beta ) 100 milliseconds READ_COMMITTED Default Configuration
- 76. RocksDB Changelog Topic Kafka Streams Thread Source Topic Output Topic WriteBatchWithIndex Interactive Query READ_COMMITTED
- 77. RocksDB Changelog Topic Kafka Streams Thread Source Topic Output Topic WriteBatchWithIndex Interactive Query READ_COMMITTED commit
- 78. RocksDB Changelog Topic Kafka Streams Thread Source Topic Output Topic WriteBatchWithIndex Interactive Query READ_COMMITTED
- 79. RocksDB Changelog Topic Kafka Streams Thread Source Topic Output Topic WriteBatchWithIndex Interactive Query READ_COMMITTED
- 80. Performance vs direct put() Benchmark by Adam Retter
- 85. public interface StateStore { }
- 86. public interface StateStore { default long approximateNumUncommittedBytes() { return 0; } }
- 90. 30
- 97. StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic
- 98. StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic
- 99. StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic commit
- 100. StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic .checkpoint checkpoint
- 101. StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic .checkpoint checkpoint
- 102. StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic .checkpoint changelog offsets
- 103. StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic .checkpoint
- 107. public interface StateStore { void flush(); }
- 109. -rw-r--r-- 1 root root 36 Sep 24 19:31 IDENTITY -rw-r--r-- 1 root root 68779002 Sep 24 20:19 000106.sst -rw-r--r-- 1 root root 68795231 Sep 24 20:19 000107.sst -rw-r--r-- 1 root root 68804429 Sep 24 20:19 000108.sst -rw-r--r-- 1 root root 68803477 Sep 24 20:19 000109.sst -rw-r--r-- 1 root root 68800584 Sep 24 20:19 000110.sst -rw-r--r-- 1 root root 22270913 Sep 24 20:19 000113.sst -rw-r--r-- 1 root root 14442 Sep 28 12:51 OPTIONS-000283 -rw-r--r-- 1 root root 16 Oct 5 09:13 CURRENT -rw-r--r-- 1 root root 14442 Oct 5 09:13 OPTIONS-000303 -rw-r--r-- 1 root root 0 Oct 5 09:13 000301.log -rw-r--r-- 1 root root 68787908 Oct 5 09:14 000306.sst -rw-r--r-- 1 root root 68815184 Oct 5 09:14 000307.sst -rw-r--r-- 1 root root 68810941 Oct 5 09:14 000308.sst -rw-r--r-- 1 root root 68805994 Oct 5 09:14 000309.sst -rw-r--r-- 1 root root 24334112 Oct 5 09:14 000310.sst -rw-r--r-- 1 root root 2386 Oct 7 20:28 000312.sst -rw-r--r-- 1 root root 2312 Oct 7 20:28 000313.sst -rw-r--r-- 1 root root 2177 Oct 7 20:28 000314.sst -rw-r--r-- 1 root root 2428 Oct 7 20:28 000315.sst -rw-r--r-- 1 root root 2311 Oct 7 20:28 000316.sst -rw-r--r-- 1 root root 2375 Oct 7 20:28 000317.sst
- 110. database created full compaction 5 10 15 20 25 days 0 30 periodic_compaction_seconds: 2592000
- 112. StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic .checkpoint checkpoint?
- 113. Changelog Topic Kafka Streams Thread Source Topic Output Topic commit StateStore + offsets
- 114. public interface StateStore { default boolean managesOffsets() { return false; } }
- 115. public interface StateStore { default boolean managesOffsets() { return false; } default void commit(Map<TopicPartition, Long> changelogOffsets) { flush(); } }
- 116. public interface StateStore { default boolean managesOffsets() { return false; } default void commit(Map<TopicPartition, Long> changelogOffsets) { flush(); } default Long getCommittedOffset(TopicPartition partition) { return null; } }
- 117. public interface StateStore { @Deprecated default void flush() {} }
- 118. --- MyCustomProcessor.java +++ MyCustomProcessor.java @@ -120 +120 @@ -store.flush(); +context.commit();
- 121. RocksDB column family A memtable column family B memtable
- 122. RocksDB column family A memtable .sst column family B memtable .sst
- 127. Restore Times Before/After KIP-892 > 3 days < 1 second
- 128. LIMITATIONS
- 130. READ_COMMITTED in-memory stores Thread-safe transaction buffer
- 131. READ_COMMITTED in-memory stores Thread-safe transaction buffer Per-query isolation levels
- 132. TL;DW
- 134. processing.guarantee default.state.isolation.level at-least-once READ_UNCOMMITTED READ_COMMITTED exactly-once(-v2|-beta) READ_UNCOMMITTED READ_COMMITTED full restore on error no restore on error
- 135. StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic Transaction Buffer
- 136. StateStore Changelog Topic Kafka Streams Thread Source Topic Output Topic Transaction Buffer commit
- 137. Q&A