Karthik Ranganathan, ehemaliger Lead-Ingenieur bei Facebook, und heute bei NUTANIX beschäftigt erklärt in seiner Präsentation moderne Datencenter anhand des Business Use Case Facebook.

1. 1
Kannan Muthukkaruppan & Karthik Ranganathan
Jun/20/2013
How Big Data Technologies
Power Facebook
How Big Data Technologies Power Facebook
Karthik Ranganathan
September, 2013

2. 2
Introduction
Email: karthik@nutanix.com
Twitter: @KarthikR
Current: Member of Technical Staff, Nutanix
Background: Technical Engineering Lead at Facebook. Co-
built Cassandra for Facebook Inbox Search and improved
performance and resiliency of Hbase for Facebook
Messages and Search Indexing.

3. 3
Agenda
 Big data at Facebook
 HBase use cases
• OLTP
• Analytics
 Operating at scale
 The Nutanix solution

4. 4
Big Data at Facebook
 OLTP
• User databases (MySQL)
• Photos (Haystack)
• Facebook Messages, Operational Data Store (HBase)
 Warehouse
• Hive Analytics
• Graph Search Indexing

5. 5
HBase in a nutshell
 Apache project, modeled after BigTable
 Distributed, large scale data store
 Built on top of Hadoop DFS (HDFS)
 Efficient at random reads and writes

6. 6
FB’s Largest Hbase Application
Facebook Messages

7. 7
The New Facebook Messages

8. 8
Why HBase?
 Evaluated a bunch of different options
• MySQL, Cassandra, building a custom storage system for
messages
 Horizontal Scalability
 Automatic failover and load balancing
 Optimized for write-heavy workloads
 HDFS already battle-tested at Facebook
 HBase’s strong consistency model

9. 9
Quick stats (as of Nov 2011)
 Traffic to HBase
• Billions of messages per day
• 75B+ rpc’s per day
 Usage pattern
• 55% reads, 45% writes
• Average write: 16 KV’s to multiple CF’s

10. 10
Data Sizes
 7PB+ online data
• ~21PB with replication
• LZO compressed
• Excludes backups
 Growth rate
• 500TB+ per month
• ~20PB of raw disk per year!

11. 11
Growing with size
 Constant need of features with growth
 Read and write path improvements
• Performance optimizations
• IOPS reduction
• New database file format
 Intelligent data and compute placement
• Shard level block placement
• Locality based load-balancing

12. 12
Other OLTP use cases of HBase
 Operational Data Store
 Multi-tenant KeyValue store
 Site integrity – fighting spam

13. 13
Warehouse use cases of HBase
 Graph Search Indexing
• Complex application logic
• Multiple verticals
 Hive over HBase
• Realtime data ingest
• Enables real-time analytics

14. 14
Real-time monitoring and anomaly detection
Operational Data Store

15. 15
ODS: Facebook’s #1 Debugging Tool
 Collects metrics from
production servers
 Supports complex
aggregations and
transformations
 Really well-designed UI

16. 16
Quick stats
 Traffic to HBase
• 150B+ ops per day
 Usage pattern
• Heavy reads of recent data
• Frequent MR jobs for rollups
• TTL to expire older data

17. 17
Real-time Analytics
Facebook Insights

18. 18
Real-time URL/Domain Insights
 Deep analytics for websites
• Facebook widgets
 Massive scale
• Billions of URL’s
• Millions of increments/sec

19. 19
Detailed Insights
 Tracks many metrics
• Clicks, likes, shares, impr
essions
• Referral traffic
 Detailed breakdown
• Age
buckets, gender, location

20. 20
Controlled Multi-tenancy
Generic KeyValue Store

21. 21
A Multi-tenant solution on HBase
 Generic Key-Value store
• Multiple apps on the same cluster
• Transparent schema design
• Simple API
put(appid, key, value)
value = get(appid, key)

22. 22
Architecture
HBase
put(appid, key, value)
Memcache
get(appid, key)
Read
Write

23. 23
Multi-tenancy Issues
 Not a self-service model
• Each app is reviewed
 Global and per-app metrics
• Monitor RPCs by type, latencies, errors
• Friendly names for apps
 If things went wrong
• Per-app kill switch

24. 24
Powering FB’s Semantic Search Engine
Graph Search Indexing

25. 25
Framework to build search indexes
 Multiple, independent input sources
 HBase stores document info
 Output is the search index image
rowKey = document id
value = terms, document data

26. 26
Architecture
HBase cluster
Document
source 2
Document
source 1
MR
cluster
…
Image files
…

27. 27
Do’s and Do-Not’s From Experience
Operating at Scale

28. 28
Design for failures(!)
 Architect for failures and manageability
 No single point of failure
• Killing any process is legit
 Minimize manual intervention
• Especially for frequent failures
 Uptime is important
• Rolling upgrades are the norm
• Need to survive rack failures

29. 29
Dashboard and Metrics
 Single place to graph/report everything
 RPC calls
 SLA misses
• Latencies, p99, Errors
• Per-request profiling
 Cluster and node health
 Network Utilization

30. 30
Health Checks
 Constantly monitor nodes
 Auto-exclude nodes on failure
• Machine not ssh-able
• Hardware failures (HDD failure, etc)
• Do NOT exclude on rack failures
 Auto-include nodes once repaired
 Rate limit remediation of nodes

31. 31
In a nutshell…
 Use commodity hardware
 Scaling out is #1
 Efficiency is #2
• though pretty close behind scale-out
 Design for failures
• Frequent failures must be auto handled
 Metrics, Metrics, Metrics!

32. 32
Overview through comparison
The Nutanix Solution

33. 33
Nutanix compared with HBase
 Evaluated a bunch of different options
• MySQL, Cassandra, building a custom storage system for
messages
 Horizontal Scalability
 Just add more nodes to scale out
 Automatic failover and load balancing
 When a node goes down, others take its place automatically
 Load of node that went down is distributed to many others

34. 34
Nutanix compared with HBase
philosophy
 Optimized for write-heavy workloads
 Optimized for virtualized environments
 Read and write heavy workloads
 Transparent use of flash to boost perf
 HDFS already battle-tested at Facebook
 Nutanix is also quite battle-tested
 HBase’s strong consistency model
 Nutanix is also strongly consistent

35. 35
Other aspects of Nutanix
 Architected for failures and manageability
 No single point of failure
 Minimal manual intervention for frequent failures
 Uptime is important
 Rolling upgrades are the norm
• Need to survive rack failures
 Single place to graph/report everything
 Prism UI to report and manage the entire cluster
 Constantly monitor nodes
 Auto-exclude nodes on failure

36. 36
In a nutshell about Nutanix…
 Runs on commodity hardware
 Scaling out is #1
 Drop in scale out for nodes
 Efficiency is #2
 Constant work on perf improvements
 Design for failures
 Frequent failures auto handled
 Alerts in UI for many other states
 Metrics, Metrics, Metrics!
 Prism UI gives insights into the cluster health

37. 37
Questions?

38. 38NUTANIX INC. – CONFIDENTIAL AND PROPRIETARY