总结这“不务正业”的半年。 爬虫是很多搜索引擎的一部分，它的名声并不好。比起搜索引擎的分词技术、索引技术来说它很基础，似乎没有那么多花样，被认为是没啥意思的脏活累活。我在这里就分享一下爬虫这个不起眼的系统里面涉及到方方面面的技术，由于内容比较细碎，话题里面只能蜻蜓点水。 如果用几个关键词形容它：爬虫、架构、分布式系统、NOSQL、实时/离线系统对比、Google Caffeine、Percolator

1. Spider
Spider

2. “ ”
Spike

3. “ ”

4. vs.
http://www.flickr.com/photos/blueblankut/497571704/sizes/z/in/photostream/
http://www.flickr.com/photos/coreyburger/2481836757/sizes/z/in/photostream/

6. Dust

8. ip
protocol sitemap robots.txt

9. Link

10. Trie

11. Query
Cache

12. ……
http://www.flickr.com/photos/regolare/791385521/

13. Spider……
Map-
Reduce

14. Crawler Architecture
Repository
Downloader
Download Extractor
Worker Worker
save page
to repository
if 302 founded
get a link update link
http status
put downloaded
page to queue
links queue
pages queue extract links
and save
main loop will put
peek site's links to
queue
Crawler Linkbase
main loop
Site will refill itself
when it's empty
TaskLoader
Priority Heap
Scope.txt Sites
Ordered Site
and their links

16. Dust
Simhash
PageRank

17. robust html css selector lxml tidy
url
proxy UA

18. 1-NoSQL
NOSQL

19. -NoSQL
NOSQL
Heap
FIFO

20. -NoSQL
HBase
Cassandra

21. -NoSQL
Cassandra
bug
Random
patitioning
Crawler Crawler

22. -NoSQL
CAP
HBase -> CP Cassandra -> AP
Cassandra C
Crawler link

23. 2-Google
incremental processing system -
Percolator. a.k.a. Caffeine

24. 2-Google
BigTable
timestamp oracle lightweight
lock
Notification trigger
Observer Notification
Notification Percolator Worker

25. 2-Google
Map-Reduce
Locality

26. 2-Google
Trade-off
trillion:million
Map-Reduce
Page RPC MR
10 MR RPC

27. 2-Google
Percolator DBMS DBMS
scale
Percolator
Percolator shared-nothing parallel
databases

28. Thanks