More Related Content Similar to 豆瓣技术架构的发展历程 @ QCon Beijing 2009 Similar to 豆瓣技术架构的发展历程 @ QCon Beijing 2009 (20) 3. • 2.9M 1/4
•
• 23M / 500~600/sec
• 23 PC (1U*15/2U*8)
• 12
• 38G memcached
4. • 1U (frodo)
• AMD Athlon 64 1.8GHz
• 1G 160G SATA*2
• Gentoo Linux
• MySQL 5
• Quixote (a Python web framework)
• Lighttpd + SCGI (shire)
• Memcached (!)
5. Gentoo Linux
•
• emerge mysql
• ebuild patch
•
•
• GLSA(Gentoo Linux Security Advisories)
6. MySQL
• The world’s most popular open source database
• / ==> MyISAM
• ==> InnoDB
• Replicate for backup
7. Python
•
• Battery Included
•
•
• CPUG: http://python.cn/
8. Quixote
• REST URL
• Django, TurboGears, Pylons
ZOPE
• http://www.douban.com/subject/1000001
# luz/subject/__init__.py
def _q_lookup(request, name):
subject = get_subject(name)
return lambda req: subject_ui(req, subject)
# luz/subject/subject_ui.ptl
def subject_ui [html] (request, subject):
site_header(request)
“<h1>%s</h1>” % subject.title
site_footer(request)
9. Lighttpd
•
• SCGI
• SCGI: FastCGI
Quixote
• 80 lighttpd
SCGI localhost
Quixote
10. Memcache
• MySQL
• libmemcache python Pyrex
python 3x+
def get_subject(subject_id):
subject = mc.get(‘s:’+subject_id)
if subject is None:
store.farm.execute(“select xxx, xxx from subject where id=%s”,
subject_id)
subject = Subject(*store.farm.fetchone())
mc.set(‘s:’+subject_id, subject)
return subject
11. Internet
Lighttpd
SCGI
App
FS
MySQL Memcache Static Files
13. • 1U
• pippin meriadoc ( merry)
• , 4G 250G SATA*3
•
• IP DNS
IP -_-b
• frodo
(subversion, trac, etc...)
14. Internet
Lighttpd (#$) DNS
HTTP
Proxy
SCGI
App Lighttpd (!quot;)
FS
Static Files Memcache MySQL
15. •
• innodb_buffer_pool_size
•
• IP
17. • IP(BGP)
• (arwen)
• 74G 1w SATA * 3
•
•
18. Internet
Data
Lighttpd Mining
read
SCGI write
App Replicate
MySQL MySQL
Master Slave
Memcache
Static Files
19. • 2M /
• IO
•
, etc...
•
20. • 4G 250G SATA*3
• 10000
• mod_rewrite URL
• lighttpd mod_memcache
• IO
• web
•
•
21. Internet
store.farm
App
MySQL
SCGI
Master
Lighttpd Memcache Replicate
store.farmr
HTTP Proxy
WebDAV Web Service
Replicate
Lighttpd
(w/ mod_memcache)
MySQL write
Slave
Spiders
Lighttpd
WebDAV
Memcache
Static Files
!quot;#$%
read
Data
Mining
MySQL
Slave
22. • store farmr
• replicate delay
•
•
• cache
•
•
• ...... but it works
23. replicate delay
def get_subject(sid):
sbj = mc.get(‘s:’+sid)
if sbj is None:
sbj = flush_subject(sid, store.farmr)
return sbj
def flush_subject(sid, cursor=None):
cursor = cursor or store.farm
cursor.execute(“select ... from subject”)
subject = Subject(*cursor.fetchone())
mc.set(‘s:’+sid, subject)
return subject
def update_subject(subject, props):
store.farm.execute(“update subject ...”)
store.farm.connection.commit()
flush_subject(subject.id, store.farm)
25. • Scale Up 1U
• 16G 147G SCSI *2 + 500G SATA
• SCSI RAID-0
• MySQL Slave
• memcached
• MyISAM InnoDB
•
• Sphinx
26. Internet
Sphinx
MySQL
Master
store.farm
Web Service
Replicate
Lighttpd
SCGI
App
HTTP Proxy
Memcache
Lighttpd
(w/ mod_memcache) WebDAV
store.farmr
Memcache
Lighttpd
Web Service
WebDAV
MySQL
Static Files
Slave
Spiders
Memcache
Memcache
28. • :)
•
•
•
• 3 1U 4 32G 1T SATA * 3
• otho.douban.com lotho.douban.com
• lighttpd 1.5 with aio support
• LVS
• Scale Up: 4G -> 8G
29. Internet
www.douban.com otho.douban.com
LVS LB
(Master)
Lighttpd
Lighttpd 1.5
(w/ mod_cache)
HTTP Proxy
Lighttpd Keepalived
LVS LB
(backup)
Lighttpd
Lighttpd 1.5
WebDAV
(w/ mod_cache)
Static Files
30. write
!quot;#$%
replicate
read Data
Mining
MySQL
MySQL
Slave
Master
read
Replicate
write
!quot;#$%
Data
Mining
MySQL
Slave
31. • 6.4M / (5M PV)
•
•
• CPU memcache /
32. •
• lighttpd mod_scgi round-robin
• lighttpd 1.5
• mod_proxy
• proxy.balance = fair (load based, passive
balancing)
•
• spread
33. Internet
Lighttpd
SCGI
HTTP Proxy
App Memcache
Lighttpd
HTTP Proxy
spread
HTTP Proxy
Lighttpd
Lighttpd
Log
Aggregator SCGI
spread
Lighttpd
Memcache
App
WebDAV
Static Files
34. • 11M / 3
•
• Sphinx
•
•
load
35. •
•
•
• Sphinx -> Xapian
• MogileFS
36. !quot;
Master
replicate
replicate
!quot;
%&'()
Slave
#$
Master read
replicate
Data
#$ read
Slave Mining
write
*+,-
!quot;
Master
Slave
replicate
*+,-
Slave write
replicate
Data
%&'()
Mining
#$
Slave
37. •
• store.farm[r] -> store.get_cursor(table=‘xxx’,
ro=True/False)
def flush_subject(sid, ro=False):
cursor = store.get_cursor(table=‘subject’, ro=ro)
cursor.execute(“select ... from subject”)
subject = Subject(*cursor.fetchone())
mc.set(‘s:’+sid, subject)
return subject
•
38. Internet
upload.douban.com otho.douban.com
nginx
signed POST form Uploader
(w/ proxy_store)
HTTP redirect
FileStorage
MogileFS Tracker
Gateway
App
MogileFS MogileFS
Node Node
MogileFS
Master
39. • libmemcache -> libmemcached consistent
hash memcache
• libmemcached consistent hash bug
• CPU
• libmemcached failover bug
• nginx lighttpd load balance
• spread
• nginx
40. Consistent Hash
charlee http://
tech.idv2.com/2008/07/24/
memcached-004/
41. Internet
App
SCGI
Lighttpd
HTTP Proxy
Lighttpd
HTTP Proxy
App
HTTP Proxy SCGI
Lighttpd
HTTP Proxy
Nginx
Lighttpd
HTTP Proxy
App
Lighttpd SCGI
Lighttpd
WebDAV
Static Files
42. • 13M /
• MogileFS
• Tracker DB
•
43. • 8
• 32G CPU
• (300G SCSI×2 + 1T SATA) × 3
• (1T SATA × 3) × 5
•6 2
• DoubanFS
44. DoubanFS
• hash
• hash hash
• Merkle Tree
• consistent hash
• WebDAV
• MogileFS 3 50
46. Internet
upload.douban.com otho.douban.com
nginx
signed POST form Uploader
(w/ proxy_store)
HTTP redirect
FileStorage
Gateway
App
DoubanFS DoubanFS
Node Node
47. • 16M /
•
• DoubanFS IO
•
48. • DoubanDB
•
• MySQL
• MySQL Master
• failover
• replicate delay
49. DoubanDB
• Key-Value
• Amazon Dynamo
• set(key, value), get(key), delete(key)
• memcache
• Merkle Tree
• Consistent Hash
• TokyoCabinet
• DoubanDB
• DoubanFS 2.0 DoubanDB
51. !quot;
Master1
replicate
replicate
!quot;
%&'()
Slave
#$
Master1 read
replicate
Data
#$ read
Slave Mining
write
*+,-
!quot;
Master
Master2
replicate
*+,-
Slave write
replicate
Data
%&'()
Mining
#$
Master2
52. • DoubanFS
•
•
• ngnix
• www.douban.com LVS
• RabbitMQ spread
53. •
• profile
• memcache cache
• join
•
•
•
