Yechao Chen Track 3: Applications https://open.mi.com/conference/hbasecon-asia-2019 THE COMMUNITY EVENT FOR APACHE HBASE™ July 20th, 2019 - Sheraton Hotel, Beijing, China https://hbase.apache.org/hbaseconasia-2019/

2. Phoenix Practice in China Life
Insurance Co., Ltd.
Leo Yuan
yuanliou2015@e-chinalife.com

3. 1 Scenarios
2 Designs
3 Optimizations
4 Problems
5 Future Work
Agenda

4. Scenarios - Overview
Main Processing
Cluster A
Application Procession
Cluster C
Readonly cluster B
Sync
basic
data
Sync result
Real Time Query
Cluster E

5. Scenarios - Overview
4 Cluste200+ Nodes（30+ Phoenix Nodes）
Clusters
Data
1300TB+ data
30TB+ biggest table
Querys
Ten million level perday
Processing
Hundreds of MR/Hive/Spark jobs per day
50TB+ Incremental data for update & insert

6. Scenarios - Overview
step 1 get some money from
company counter
step 2 query operation detail from app
This Is What We Do
Real Cust Rights View System !

7. Scenarios - Overview
30,000,000 + Incremental data/day 700,000 + users / day
10,000+ records/sec 8,000+ sqls / sec

8. 1 Scenarios
2 Designs
3 Optimizations
4 Problems
5 Future Work
Agenda

9. Designs - Data Architecture
ü Initialize Data
ü Build Phoenix Index
ü Sync Real Time Data
ü Provide Data Service
China Life Insurance APP
Business
System
SharePlex
Kafka
Spark
Streaming
Cust View System
Phoenix
HBase

10. Designs - Development Architecture
Data Integration
Data View
Cross-province Integration
Query Distribution Engine
V_constract table
(4+3)
Data Service
Ultimate Real Time Query Service
Data Exchange Initial Data Sync Program
Data Source Business System DataBase
JobSchedule
Monitor
Resourceschedule
monitor
Schedule
Monitor
Real Time Data Link
Incremental Data Sync Program
V_pay table
(4+3)
V_cust service table
(4+3)
V_claim table
(4+3) …
dim
Resourceaccess
control
UserAccess
Control
Privilege
Control

11. Designs - Physical Architecture
Phoenix Cluster
Configuration DB Weblogic Server Weblogic Server Weblogic Server
F5APP Server
Phoenix Gateway

12. Kafka
Designs - Real Time Data Sync
cid c_no type amount branch syssource updtime incr_flag commit_time ……
001 001 M 100 000000 V6 2019-06-11 16:37.322 1 2019-06-11 16:37.322 ……
02 002 R 100 000001 V7 2019-06-11 16:38.689 2 2019-06-11 16:38.689 ……
Contract（BeiJing）
cid c_no type amount
01 001 M 100
1、Partition By Global Primary Key
2、Shield Upstream System Table Structure Adjustment
3、No Effect on Normal Stream Process when Data
Supplement
…
… ……
SparkStreaming
Contract（ShangHai）
cid c_no type amount
02 002 R 100
SparkStreaming_compt

13. Designs - Ultimate Real Time Query Service
label name label type isHolder logic
paidBonus sum holder sql1,2,3
paidMoenyList list holder sql4,5
paidExpire sum Insured sql6
… … … …
label.properties
Analyze
Parameters
Package Sqls
Execue Sqls
Collect Results
Chinalife Insurance APP
Ultimate Real Time Query Service

14. 1 Scenarios
2 Designs
3 Optimizations
4 Problems
5 Future Work
Agenda

15. Optimizations - Sql Execution Process
DriverManager.getConnection("phoenixUrl")
con.prepareStatement(sql)
pstat.executeQuery()
rs.next()
1、SYSTEM.CATALOG
2、SYSTEM.STATS
3、SYSTEM.LOG
4、 SYSTEM.SEQUENCE
1、Query meta data of table/index
from phoenix server(SYSTEM.CATALOG)
2、Determine the table/index sql need to scan
3、Query statistics information of table/index
from phoenix server(SYSTEM.STATS)
4、Generate scans based on statistics information、meta data、sql
1、Parallelity decided by phoenix.query.threadPoolSize

16. Optimizations – Phoenix System Table
SYSTEM.CATALOG
SYSTEM.STATS
Describe table/index meta information,
such as
l TABLE NEME
l COLUMN NAME
l SALT_BUCKETS
l UPDATE_CACHE_FREQUENCY
l GUIDE_POST_WIDTH
Describe table/index accurate
statistics information, such as
l GUIDE_POST_KEY
l GUIDE_POSTS_WIDTH
PS:UPDATE STATISTICS TABLE_NEME

17. Optimizations - RS Group
Hmaster
SYSTEM:CATALOG
SYSTEM:MUTEX
SYSTEM:STATS
CONTRACT
INCOME
PERSON
RS Group 1 RS Group 2
rs1
SYSTEM:CATALOG SYSTEM:STATS
Hmaster
rs2 rs3 rs4 … rs1 rs3 rs4 …rs2
……
SYSTEM:STATSCONTRACTINCOME
PERSON ……
• Metadata Table Isolation to Decrease Impact on Business Table Query

18. Optimizations - UPDATE_CACHE_FREQUENCY
• Adjust this parameter to decrease hotspot in SYSTEM.CATALOG
p Decide the query frequency of SYSTEM.CATALOG
p Default value is “Always” and will Cause read/write pressure in SYSTEM.CATALOG
p Can be set per Cluster/Table
1. “phoenix.default.update.cache.frequency”: 86412345
2. create table test.test (a varchar not null primary key,b varchar )
SALT_BUCKETS = 10, UPDATE_CACHE_FREQUENCY=86400000;

19. Optimizations – Salt & Pre-Split
• Data table use salt_buckets, Index table use pre-split
CREATE TABLE …(
)SALT_BUCKETS = 60
CREATE INDEX …
ON …(
…
) INCLUDE (
…
) ASYNC SALT_BUCKETS
= 0
SPLIT ON (
…
)
1、Create index to
ensure that all of query
process stop at index
table
2、use pre split to reduce
the chunk number when
execute a query
3、use async index to
avoid OOM when
building the index table
1、Use
salted
table to
distribute
data evenly
Phoenix Client
write
Phoenix Client
query

20. Optimizations - Open Offheap
• Minimal read cost to improve query efficiency
BucketCache Configuration Properties
l hbase.bucketcache.combinedcache.enabled
l hbase.bucketcache.ioengine
l hfile.block.cache.size
l hbase.bucketcache.size
l hbase.bucketcache.bucket.sizes
l -XX:MaxDirectMemorySize

21. Optimizations - Other Configurations
• Region Balance By Table
• G1GC
• Manual MajorCompaction

22. 1 Scenarios
2 Designs
3 Optimizations
4 Problems
5 Future Work
Agenda

23. Problems - Current time function cause Query performance degradation
• This function will lead to client-cluster interaction frequently
SELECT …
FROM …
WHERE date <=
CURRENT_TIME()
SELECT …
FROM …
WHERE date <=
CURRENT_TIME()
SELECT …
FROM …
WHERE date <= NOW_DATE
replace CURRENT_TIME
In java code to NOW_DATE
①
②
③

24. Problems - HBase cluster balance abnormal
1.Turn on RSGroup
2.Set two RSGroup default 、my_group
3.Move rs to default and my_group
4.Restart one rs in default
process Problems Resolve
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer
2. balance_rsgroup 'default' abnormal
1. balancer abnormal

25. Problems - ACL abnormal when come together with RSGroup
1.Turn on RSGroup
2.Turn on ACL
process Problems
1. Non-hbase user can’t creat table
Resolve
1, Use hbase user to create table
2, grant this table 'RWX’ to Non-hbase user
<property>
<name>
hbase.coprocessor.master.classes
</name>
<value>
org.apache.hadoop.hbase.security.
access.AccessController,
org.apache.hadoop.
hbase.rsgroup.RSGroupAdminEndpoint
</value>
</property>

26. 1 Scenarios
2 Designs
3 Optimizations
4 Problems
5 Future Work
Agenda

27. Future Work - RPC Read/Write Isolation
total queue = hbase.ipc.server.callqueue.handler.factor * handler
read queue = total queue * hbase.ipc.server.callqueue.read.ratio
write queue = total queue * (1- hbase.ipc.server.callqueue.read.ratio )
scan queue = = total queue * hbase.ipc.server.callqueue.read.ratio *
hbase.ipc.server.callqueue.scan.ratio
total queue
write
queue
read queue
scan
queue

28. Future Work - Compaction Contral
Set Offpeak Time
Set Peak Time Throughput
l hbase.offpeak.end.hour
l hbase.offpeak.start.hour
l hbase.hstore.compaction.throughput.offpeak
l key:hbase.regionserver.throughput.controller
value:org.apache.hadoop.hbase.regionserver.compactions.
PressureAwareCompactionThroughputController
Open Compaction Controller
l hbase.hstore.compaction.throughput.higher.bound
l hbase.hstore.compaction.throughput.lower.bound

29. Future Work - Join Optimization
SELECT …
FROM ( a1
JOIN (SELECT...
FROM b1
WHERE …) a2
ON a1…. = a2…)
WHERE …
EXPLAIN

30. Thanks！
yuanliou2015@e-chinalife.com