A presentation in ApacheCon Asia 2021 from Yuchen He and Shuo Jia. Apache Pegasus is a horizontally scalable, strongly consistent and high-performance key-value store. Know more about Pegasus https://pegasus.apache.org, https://github.com/apache/incubator-pegasus

1. APACHE PEGASUS(INCUBATING) - A
DISTRIBUTED KEY-VALUE STORAGE
SYSTEM
Yuchen He & Shuo Jia
Software Engineer from XiaoMi, Apache Pegasus PPMC
Incubator

2. Speakers
Yuchen He
• Graduate from Renmin University of China
• Software engineer from XiaoMi
• Pegasus project leader in XiaoMi
• Apache Pegasus PPMC
Shuo Jia
• Graduate from Beijing Jiaotong University of China
• Software engineer from XiaoMi
• Apache Pegasus PPMC
• Participated in the development of Pegasus for 2
years

3. Outline
• Basic Introduction
– Architecture, Data Model, Dual WAL, Performance
• New Features
– Duplication, Bulk load, Access control, Partition split
• Surrounding Ecosystems
– Pegasus-Spark, Meta proxy, Disk Migration tools
• Community

4. Basic Introduction
4

5. Introduction
• Redis or HBase
– Non-Volatile vs Consistent
– Remote Access
• Pegasus
– C++
– Local persistent storage
– Strongly consistent
– High performance
– Horizontally scalable

6. Architecture
Meta server
• Cluster controller
• Configuration manager
Replica server
• Data node
• Hash partitioning
• PacificA (strongly consistent)
• RocksDB instance for each replica
Zookeeper
• Meta server election
• Metadata storage
ClientLib
• Cache data routing table
• Straightly access to replica server

7. Data Model

8. Dual WAL
Disk
Data
Log
Replica1
Data
Log
Replica2
Data
Log
Replica3
client
Traditional solution
• Data background compaction may strongly affect WAL sync performance

9. Dual WAL
Data Disk
Data
Private Log
Replica1
Data
Private Log
Replica2
Data
Private Log
Replica3
client
Shared Log
Log Disk
• Separate WAL and data, sync-write shared log, async-write private log

10. Performance
Read:Write Client*Thread --- QPS AvgLatency P99Latency(us)
0:1 3*15
read --- --- ---
write 46128 972 5591
1:0 3*50
read 282648 542 1674
write --- --- ---
1:1 3*30
read 36014 1068 15345
write 36016 1421 8197
1:3 3*15
read 11622 779 10417
write 34989 1021 5467
2.2.0 (Newest release) benchmark

11. New Features
11

12. Duplication
Region2
Table
Region1
Table
async-duplication
Basic introduction
• Design for cross-region online backup
• Transfer log, write asynchronously
• Supporting single-master and multi-master

13. Duplication
Case1: Online Migration
Target Cluster
Table
Source Cluster
Table
client
1. Reserve logs
Remote storage
2. cold backup
3. restore
4. duplication
5. switch

14. Duplication
Case2: Master-Slave cluster
client client
Slave region
Table
Master region
Table
duplication
Eventually-consistent
read
client client
Table
Region1 Region2

15. Duplication
Enhancement in future
• Master-master in practice
• More than two region duplication in practice
• Facility for supporting remote disaster-tolerant system
• auto-switch master slave
• better user experience
• Extension:
• supporting CDC on demand
• eg: ES, MQ…

16. Bulk Load
sst file
sst file
Table
Replica server
original data
File provider
sst file sst file
1. Generate Files
2. Download Files
3. Ingest Files
client
R/W Reject write(ingestion)
Fast import lots of data offline

17. Access Control
Authentication: Kerberos
Authorization: Whitelist based coarse-grained table-level access control
Cluster
KeytabA
X
TableA
KeytabB
TableB
KeytabA
client

18. Partition Split
• Replica divide into two replicas
• Replica[i] -> Replica[i], Replica[i+original_partition_count]
Basic introduction
Replica group0
Replica0 Replica4
Replica0
Replica group1
Replica1 Replica5
Replica1
Replica group2
Replica2 Replica6
Replica2
Replica group3
Replica3 Replica7
Replica3

19. Partition Split
Stage1: async-learn
client
Replica server
child
secondary
Replica server
child
primary
Replica server
child
secondary
copy data
copy data copy data
• parent(old replica), child(new replica)
• child replica copy data
• client only know parent replica

20. Partition Split
Stage2: register
client
Replica server
child
secondary
Replica server
child
primary
Replica server
child
secondary
meta server
register child X
• when child copy all parent data
• Reject R/W while registering

21. Partition Split
Partition split succeed
Replica server
secondary
secondary
Replica server
primary
primary
Replica server
secondary
secondary
client
• Merged in master, will be released in 2.3.0
• GC dup-data by compaction

22. Surrounding Ecosystem
22

23. Pegasus-Spark
Best practices
• Large offline data analysis (SQL)
• Large offline data load (BulkLoad)

24. Pegasus-Spark
Offline Analysis
• Convert into Hive(parquet)
• Use SparkSQL to analysis
HDFS
Replica server Replica server
Hive
Schema RDD

25. Pegasus-Spark
Convert to SST file for Bulk load
node
node
node
node
node
node
Transform(Pegasus-Spark)
HDFS
(sst file)
Distinct
Repartition
Sort
original
data
original
data

26. Meta Proxy
Basic introduction
• access unification
• primary and standby cluster manager
client client client
Cluster A
meta meta
Cluster B
meta meta
Cluster C
meta meta
client client client
Cluster A
meta meta
Cluster B
meta meta
Cluster C
meta meta
MetaProxy

27. Meta Proxy
Switch primary and standby cluster
client client client
Cluster primary
meta meta
Cluster secondary
meta meta
MetaProxy
duplication
client client client
Cluster secondary
meta meta
Cluster primary
meta meta
MetaProxy
duplication
switch

28. Disk migration tool
balance disk usage on replica server
Disk4
40%
Disk2
75%
Disk1
70%
Disk3
85%
Disk
migrator
Select Disk
Select
Replica
Migrate
Replica
balanced
Disk4
65%
Disk2
65%
Disk1
70%
Disk3
70%
Replica server Replica server
Loop
until balance

29. Community
29

30. Process
2016
Release 1.0.0
Join Apache
Release 2.0.0
Meet UP
2015
Start
Open GitHub
2017.9
2020.6
2020.9
2021.8

31. Tools
Start contribution from API and tools
Pegasus
core
user-cli
client
HTTP API
RPC API
monitoring
admin-cli
deploy tools
other tools …

32. In future
Issues, Roadmap, RFC
• New Features
• Cluster load balance
• Table Migrator Tools
• Read throughput throttling
• Support K8S
...
• Feature enhancement
• Duplication
• Bulk load
• Hot partition detection
…
• Tests
• Documents

33. Activities
• August 21st Beijing
First offline meetup will be coming soon

34. THANK YOU
QUESTIONS?
https://github.com/apache/incubator-pegasus
https://pegasus.apache.org/
Apache Pegasus