1. Intorduction to Distributed Infrastructure
Intorduction to Distributed Infrastructure
Guang Feng
dudu@hunantv.com
April 29, 2010

2. Intorduction to Distributed Infrastructure
Topic
当今分布式技术主要有以下几个研究方向

3. Intorduction to Distributed Infrastructure
Topic
当今分布式技术主要有以下几个研究方向
分布式文件系统

4. Intorduction to Distributed Infrastructure
Topic
当今分布式技术主要有以下几个研究方向
分布式文件系统
分布式计算框架

5. Intorduction to Distributed Infrastructure
Topic
当今分布式技术主要有以下几个研究方向
分布式文件系统
分布式计算框架
分布式存储引擎

6. Intorduction to Distributed Infrastructure
Topic
当今分布式技术主要有以下几个研究方向
分布式文件系统
分布式计算框架
分布式存储引擎
数据库会议的一些个人思考

7. Intorduction to Distributed Infrastructure
Topic
当今分布式技术主要有以下几个研究方向
分布式文件系统
分布式计算框架
分布式存储引擎
数据库会议的一些个人思考
Language Tower

8. Intorduction to Distributed Infrastructure
Topic
当今分布式技术主要有以下几个研究方向
分布式文件系统
分布式计算框架
分布式存储引擎
数据库会议的一些个人思考
Language Tower
数据库性能监控与分析

9. Intorduction to Distributed Infrastructure
Topic
当今分布式技术主要有以下几个研究方向
分布式文件系统
分布式计算框架
分布式存储引擎
数据库会议的一些个人思考
Language Tower
数据库性能监控与分析
DBA团队建设与职责

10. Intorduction to Distributed Infrastructure
Distributed File System
解决的问题

11. Intorduction to Distributed Infrastructure
Distributed File System
解决的问题
大容量文件存储

12. Intorduction to Distributed Infrastructure
Distributed File System
解决的问题
大容量文件存储
本地文件系统的I/O瓶颈

13. Intorduction to Distributed Infrastructure
Distributed File System
解决的问题
大容量文件存储
本地文件系统的I/O瓶颈
灾难备份

14. Intorduction to Distributed Infrastructure
Distributed File System
解决的问题
大容量文件存储
本地文件系统的I/O瓶颈
灾难备份
主要产品

15. Intorduction to Distributed Infrastructure
Distributed File System
解决的问题
大容量文件存储
本地文件系统的I/O瓶颈
灾难备份
主要产品
面向大文件存储 Google GFS like...

16. Intorduction to Distributed Infrastructure
Distributed File System
解决的问题
大容量文件存储
本地文件系统的I/O瓶颈
灾难备份
主要产品
面向大文件存储 Google GFS like...
面向小文件存储 MogileFS

17. Intorduction to Distributed Infrastructure
Distributed Computing Framework
概念

18. Intorduction to Distributed Infrastructure
Distributed Computing Framework
概念
MapReduce

19. Intorduction to Distributed Infrastructure
Distributed Computing Framework
概念
MapReduce
并行处理的是数据,不是任务(计算能力)

20. Intorduction to Distributed Infrastructure
Distributed Computing Framework
概念
MapReduce
并行处理的是数据,不是任务(计算能力)
产品

21. Intorduction to Distributed Infrastructure
Distributed Computing Framework
概念
MapReduce
并行处理的是数据,不是任务(计算能力)
产品
Hadoop

22. Intorduction to Distributed Infrastructure
Distributed Computing Framework
概念
MapReduce
并行处理的是数据,不是任务(计算能力)
产品
Hadoop
*Gearman

23. Intorduction to Distributed Infrastructure
Distributed Computing Framework
Gearman
Figure: Gearman Arch

24. Intorduction to Distributed Infrastructure
Distributed Computing Framework
Gearman
work in parallel
Figure: Gearman Arch

25. Intorduction to Distributed Infrastructure
Distributed Computing Framework
Gearman
work in parallel
background task
Figure: Gearman Arch

26. Intorduction to Distributed Infrastructure
Distributed Storage Engine
Colunmn-Based Storage

27. Intorduction to Distributed Infrastructure
Distributed Storage Engine
Colunmn-Based Storage
Google BigTable

28. Intorduction to Distributed Infrastructure
Distributed Storage Engine
Colunmn-Based Storage
Google BigTable
Amazon Dynamo

29. Intorduction to Distributed Infrastructure
Distributed Storage Engine
Colunmn-Based Storage
Google BigTable
Amazon Dynamo
Not Only SQL

30. Intorduction to Distributed Infrastructure
Distributed Storage Engine
Colunmn-Based Storage
Google BigTable
Amazon Dynamo
Not Only SQL
Tokyo Cabinet

31. Intorduction to Distributed Infrastructure
Distributed Storage Engine
Colunmn-Based Storage
Google BigTable
Amazon Dynamo
Not Only SQL
Tokyo Cabinet
Relational Database

32. Intorduction to Distributed Infrastructure
Distributed Storage Engine
Colunmn-Based Storage
Google BigTable
Amazon Dynamo
Not Only SQL
Tokyo Cabinet
Relational Database
on-Proxy

33. Intorduction to Distributed Infrastructure
Distributed Storage Engine
Colunmn-Based Storage
Google BigTable
Amazon Dynamo
Not Only SQL
Tokyo Cabinet
Relational Database
on-Proxy
Cluster

34. Intorduction to Distributed Infrastructure
Colunmn-Based Storage
优势

35. Intorduction to Distributed Infrastructure
Colunmn-Based Storage
优势
动态增加存储列,水平切分伸缩性强

36. Intorduction to Distributed Infrastructure
Colunmn-Based Storage
优势
动态增加存储列,水平切分伸缩性强
每单元(Tracker)中记录统一数据的多个版本

37. Intorduction to Distributed Infrastructure
Colunmn-Based Storage
优势
动态增加存储列,水平切分伸缩性强
每单元(Tracker)中记录统一数据的多个版本
数据聚集性强

38. Intorduction to Distributed Infrastructure
Colunmn-Based Storage
优势
动态增加存储列,水平切分伸缩性强
每单元(Tracker)中记录统一数据的多个版本
数据聚集性强
缺点

39. Intorduction to Distributed Infrastructure
Colunmn-Based Storage
优势
动态增加存储列,水平切分伸缩性强
每单元(Tracker)中记录统一数据的多个版本
数据聚集性强
缺点
不适合频繁CRUD

40. Intorduction to Distributed Infrastructure
Colunmn-Based Storage
优势
动态增加存储列,水平切分伸缩性强
每单元(Tracker)中记录统一数据的多个版本
数据聚集性强
缺点
不适合频繁CRUD
不支持条件查询

41. Intorduction to Distributed Infrastructure
Colunmn-Based Storage
优势
动态增加存储列,水平切分伸缩性强
每单元(Tracker)中记录统一数据的多个版本
数据聚集性强
缺点
不适合频繁CRUD
不支持条件查询
不适合OLTP(On-Line Analytical Processing)

42. Intorduction to Distributed Infrastructure
NoSQL:Tokyo Cabinet
特性

43. Intorduction to Distributed Infrastructure
NoSQL:Tokyo Cabinet
特性
键值对存储

44. Intorduction to Distributed Infrastructure
NoSQL:Tokyo Cabinet
特性
键值对存储
insert 0.4s/1M records & search 0.33s/1M records

45. Intorduction to Distributed Infrastructure
NoSQL:Tokyo Cabinet
特性
键值对存储
insert 0.4s/1M records & search 0.33s/1M records
支持丰富的语言调用(HTTP协议)

46. Intorduction to Distributed Infrastructure
NoSQL:Tokyo Cabinet
特性
键值对存储
insert 0.4s/1M records & search 0.33s/1M records
支持丰富的语言调用(HTTP协议)
不足

47. Intorduction to Distributed Infrastructure
NoSQL:Tokyo Cabinet
特性
键值对存储
insert 0.4s/1M records & search 0.33s/1M records
支持丰富的语言调用(HTTP协议)
不足
上亿的数据存储后,性能下降很快

48. Intorduction to Distributed Infrastructure
NoSQL:Tokyo Cabinet
特性
键值对存储
insert 0.4s/1M records & search 0.33s/1M records
支持丰富的语言调用(HTTP协议)
不足
上亿的数据存储后,性能下降很快
很难扩展到多机

49. Intorduction to Distributed Infrastructure
RDB:Amoeba
Proxy Architecture(代理风格)

50. Intorduction to Distributed Infrastructure
RDB:Amoeba
Proxy Architecture(代理风格)
特性

51. Intorduction to Distributed Infrastructure
RDB:Amoeba
Proxy Architecture(代理风格)
特性
提供规则数据切分

52. Intorduction to Distributed Infrastructure
RDB:Amoeba
Proxy Architecture(代理风格)
特性
提供规则数据切分
降低db 与客户端的连接数

53. Intorduction to Distributed Infrastructure
RDB:Amoeba
Proxy Architecture(代理风格)
特性
提供规则数据切分
降低db 与客户端的连接数
读写分离

54. Intorduction to Distributed Infrastructure
RDB:Amoeba
Proxy Architecture(代理风格)
特性
提供规则数据切分
降低db 与客户端的连接数
读写分离
未来打算支持异构数据库

55. Intorduction to Distributed Infrastructure
RDB:EnterpriseDB Cluster
特性

56. Intorduction to Distributed Infrastructure
RDB:EnterpriseDB Cluster
特性
内置连接池,内存共享

57. Intorduction to Distributed Infrastructure
RDB:EnterpriseDB Cluster
特性
内置连接池,内存共享
读请求负载均衡

58. Intorduction to Distributed Infrastructure
RDB:EnterpriseDB Cluster
特性
内置连接池,内存共享
读请求负载均衡
所有节点并行查询

59. Intorduction to Distributed Infrastructure
RDB:EnterpriseDB Cluster
特性
内置连接池,内存共享
读请求负载均衡
所有节点并行查询
查询缓存

60. Intorduction to Distributed Infrastructure
Language Tower
分布式基础架构

61. Intorduction to Distributed Infrastructure
Language Tower
分布式基础架构
语言无关的通讯协议

62. Intorduction to Distributed Infrastructure
Language Tower
分布式基础架构
语言无关的通讯协议
动态脚本语言实现业务逻辑

63. Intorduction to Distributed Infrastructure
Thinking...
数据库性能监控与分析

64. Intorduction to Distributed Infrastructure
Thinking...
数据库性能监控与分析
结合主机负载与数据库指标进行监控

65. Intorduction to Distributed Infrastructure
Thinking...
数据库性能监控与分析
结合主机负载与数据库指标进行监控
数据库日志(二进制日志、慢查询)分析

66. Intorduction to Distributed Infrastructure
Thinking...
数据库性能监控与分析
结合主机负载与数据库指标进行监控
数据库日志(二进制日志、慢查询)分析
根据业务特点进行针对优化

67. Intorduction to Distributed Infrastructure
Thinking...
数据库性能监控与分析
结合主机负载与数据库指标进行监控
数据库日志(二进制日志、慢查询)分析
根据业务特点进行针对优化
DBA团队建设与职责

68. Intorduction to Distributed Infrastructure
Thinking...
数据库性能监控与分析
结合主机负载与数据库指标进行监控
数据库日志(二进制日志、慢查询)分析
根据业务特点进行针对优化
DBA团队建设与职责
日常维护数据

69. Intorduction to Distributed Infrastructure
Thinking...
数据库性能监控与分析
结合主机负载与数据库指标进行监控
数据库日志(二进制日志、慢查询)分析
根据业务特点进行针对优化
DBA团队建设与职责
日常维护数据
Top Query的分析与解决

70. Intorduction to Distributed Infrastructure
Thinking...
数据库性能监控与分析
结合主机负载与数据库指标进行监控
数据库日志(二进制日志、慢查询)分析
根据业务特点进行针对优化
DBA团队建设与职责
日常维护数据
Top Query的分析与解决
SQL由谁来写?

71. Intorduction to Distributed Infrastructure
Thank you!
Q&A