Junhong Xu of Xiaomi Track 2: Ecology and Solutions https://open.mi.com/conference/hbasecon-asia-2019 THE COMMUNITY EVENT FOR APACHE HBASE™ July 20th, 2019 - Sheraton Hotel, Beijing, China https://hbase.apache.org/hbaseconasia-2019/

2. OpenTSDB at Xiaomi
Junhong Xu
Software engineer@Xiaomi

3. Agenda
1. OpenTSDB
2. OpenTSDB on Kubernetes
3. Use Cases

4. Trend from DB-Engine

5. Time Series Data
1. Usually arrive in chronological order
2. Almost immutable, new data is new record
3. Usually no peak, no trough
4. Data similarity

6. What is OpenTSDB?
1. Stateless
2. Scalable: build on HBase
3. High performance
4. High level of granularity
5. Supported by Grafana
Compatible with HBase 2.x since OpenTSDB 2.3.1

7. OpenTSDB Overview
钟老板 机器环境下有安openjdk 11吗

8. OpenTSDB Internal

9. OpenTSDB API Path

10. OpenTSDB UID Table in HBase

11. OpenTSDB Row Key in HBase
row key = metric(uid) + base timestamp + tag(uid)

12. OpenTSDB Compaction

13. OpenTSDB Configuration
config name default description
tsd.core.auto_create_metrics False Whether or not a data point with a
new metric will assign a UID to
the metric
tsd.core.tag.allow_specialchars !~/ An optional list of ASCII
characters allowed in metric
names, tag names and tag keys
above those already allowed by
TSDB. Spaces are allowed.
tsd.http.request.enable_chunked False Whether or not to enable
incoming chunk support for the
HTTP RPC
tsd.http.request.max_chunk 4096 The maximum request body size
to support for incoming HTTP
requests when chunking is
enabled.
tsd.query.timeout 0 How long, in milliseconds, before
canceling a running query.

14. OpenTSDB Configuration
config name default description
tsd.storage.enable_compaction True Whether or not to enable
compactions
tsd.storage.fix_duplicates False Whether or not to accept the last
written value when parsing data
points with duplicate timestamps
tsd.storage.hbase.data_table tsdb Name of the HBase table where
data points are stored
tsd.storage.hbase.scanner.maxNumRows 128 The maximum number of rows to
fetch from HBase per call to the
scanner's nextRows() method.
tsd.storage.hbase.uid_table tsdb-uid Name of the HBase table where
UID information is stored
tsd.storage.hbase.zk_basedir /hbase Path under which the znode for the
-ROOT- region is located

15. OpenTSDB Configuration
config name default description
tsd.storage.hbase.zk_quorum localhost A comma-separated list of
ZooKeeper hosts to connect to,
with or without port specifiers.
tsd.storage.max_tags 8 The maximum number of tags
allowed per data point.
tsd.storage.salt.buckets 20 The number of salt buckets used to
distribute load across regions.
tsd.storage.uid.width.metric 3 The width, in bytes, of metric UIDs.
tsd.storage.uid.width.tagk 3 The width, in bytes, of tag name
UIDs.
tsd.storage.uid.width.tagv 3 The width, in bytes, of tag value
UIDs.
tsd.query.enable_fuzzy_filter True Whether or not to enable the
FuzzyRowFilter for HBase when
making queries using the
explicitTags flag.

16. OpenTSDB Kerberos Conf

17. OpenTSDB on
Kubernetes

18. What we concern when maintaining OpenTSDB?
1. Deployment
2. Config management
3. Auto-scaling
4. Fault tolerance
5. Multi-tenancy

19. What we concern when maintaining OpenTSDB?
1. Deployment
2. Config management
3. Auto-scaling
4. Fault tolerance-tenancy
5. Multi-tenancy

20. Kubernetes
1. Automating deployment
2. Automating scaling
3. Automating management
4. Containerized applications
So OpenTSDB benefits from Kubernetes!

21. Why OpenTSDB is ffit ffor Kubernetes?
1. Stateless
2. High Performance
3. Latency-insensitive

22. How to build OpenTSDB on Kubernetes?

23. ConfigMap: Decouple Conf from Image
attribute name description
apiVersion v1
kind ConfigMap
data
metadata metadata
metadata.name name
metadata.namespace default is ‘default’

24. Secret：Secure Conf
attribute name description
apiVersion v1
kind Secret
metadata metadata
metadaba.name name
metadata.namespace default is ‘default’
data

25. Deployment
• Pod
• Quota
• Replica Set
attribute name description
spec.template.metadata.labels Pod label
spec.template.spec.containers[].args Container start args
spec.template.spec.containers[].command Container start command
spec.template.spec.containers[].image Image address
spec.template.spec.containers[].imagePullPolicy Always
spec.template.spec.containers[].resources.limits.cpu Cpu limit
… …

26. Service
1. Entry for a set of pods
2. Load balancer
3. Service discovery
attribute name required description
version Y v1
kind Y Service
metadata Y metadata
metadaba.name Y name
metadata.namespace Y default is ‘default’
spec Y
spec.selector[] Y Label Selector
spec.type Y ClusterIP as default
spec.clusterIP Service virtual IP
spec.ports[] Service exposed ports
spec.ports[].targetPort backend port

27. Ingress
1. Entry from the internet
2. Logical, no entity
3. HTTP/HTTPS only
attribute name description
apiVersion extensions/v1beta1
kind Ingress
metadata metadata
metadaba.name name
metadata.namespace default is ‘default’
metadata.annotations
spec
spec.rules[]
spec.rules[].host url access from internet
spec.rules[].http.paths[].backend
spec.rules[].http.paths[].backend.serviceName Service name
spec.rules[].http.paths[].backend.servicePort service port

28. Ingress
1. Entry from the internet
2. Logical, no entity
3. HTTP/HTTPS only
Now we have a cluster! So?
attribute name description
apiVersion extensions/v1beta1
kind Ingress
metadata metadata
metadaba.name name
metadata.namespace default is ‘default’
metadata.annotations
spec
spec.rules[]
spec.rules[].host url access from internet
spec.rules[].http.paths[].backend
spec.rules[].http.paths[].backend.serviceName Service name
spec.rules[].http.paths[].backend.servicePort service port

29. OpenTSDB as a Service

30. OpenTSDB Manager

31. Implementation details
1. Running on Kubernetes
2. Web framework: SpringBoot
3. Kubernetes client: Fabric8
4. Build OpenTSDB cluster asynchronously
5. One HBase namespace per user
6. Add trace info in HTTP header

32. OpenTSDB Use Cases

33. Case 1: HBase monitoring

34. Stats
metric description
Read QPS ~100
Write QPS ~660k
Storage ~90T
Node 65
p95 ~228 ms
metric 10000+
tags 1000000+

35. Case 2: Kubernetes monitoring

36. Stats
metric description
Read QPS ~0
Write QPS ~60k
Storage ~6T
Node 10
p95 ~108 ms
metric 1000+
tags 100000+

37. Thanks！