Andrew Cheng
Track 3: Applications
https://open.mi.com/conference/hbasecon-asia-2019
THE COMMUNITY EVENT FOR APACHE HBASE™
July 20th, 2019 - Sheraton Hotel, Beijing, China
https://hbase.apache.org/hbaseconasia-2019/
5. HBase Story in Tencent
l Began using since 2013
l Used version
l 0.94.17 -> 0.98.6 -> 1.2.5 -> 2.2.0 (ing)
l Largest cluster more than 500 nodes
90+
Clusters
4000+
Nodes
10PB+
Data
3Tri+
RPD
13. Practices–Table
l Create table per day
l Large amount of data
l TTL is short
l Benefit
l Reduce the amount of data in compaction
l Easy to delete expired data
14. Optimization - Bandwidth
② RS2 and RS3 Wal data
① Input Data
③ RS2 and RS3 Flush data
⑤ RS2 and RS3 Large compact
④ RS2 and RS3 Small compact
RS1 RS2 RS3
Input Data
Wal
Flush
①
Small compact
Large compact
②
③
④
⑤
Input Data Input Data
15. Optimization - Bandwidth
l Enable compressing of CellBlocks
l Wal compressor
l Increase the size of memstore
l Reduce the number of threads about compaction
l Turn off major compaction
l create tables by day
16. Optimization - Online filtering of dirty data
l A large amount of data which have the same Rowkey
l How to find filter rowkeys?
l ResponseTooSlow
l How to set filter rowkeys?
l hbase.hregion.filter.rowkeys
l How to refresh filter rowkeys?
l update_config
Input Data
Filter
Enable
Write
Filter
Yes
Yes
No
No
17. Optimization - Prefix Bloom Filter(HBASE-20636)
l ROWPREFIX_FIXED_LENGTH
l ROWPREFIX_DELIMITER
uin ts action
Bloom Filter
Prefix
Create Table:
File info:
18. Optimization - Prefix Bloom Filter(HBASE-20636)
Scan
Not Filter StoreFile
Same
prefix?
{StartKey,EndKey}
Computer hash
value
Hit
BloomFilter?
Prefix length
>=
prefix_length
Yes
Yes
No
Filter StoreFile
No
No
Get prefix key by
prefix_length
Yes
Read
Rowkey
Get prefix key by prefix_length
Computer hash value
Set BloomFilter
Last line?
Input Data
Write BloomFilter information to StoreFile metadata
Yes
No
Write
21. Optimization - RestServer
l Only maintain one configuration
l use effectively resources
l User-friendly access
22. HBase Community
l 1 Committer, 2 Contributor
l Total commits: 80+
l Feature
l HBASE-20636 Introduce two bloom filter type : ROWPREFIX_FIXED_LENGTH and ROWPREFIX_DELIMITED
l HBASE-19799 Add web UI to rsgroup
l HBASE-20243 [Shell] Add shell command to create a new table by cloning the existent table
l HBASE-19483 Add proper privilege check for rsgroup commands
l ………