Presentation held in the seminar on "NotOnlySQL databases."

1. memcached on steroids

2. “blazingly fast” – Kirk Haines, Engineyard

3. Back in the day…
Salvatore Sanfilippo

4. Back in the day…
Salvatore Sanfilippo

5. Back in the day…
LLOOGG
Google Analytics + Realtime

6. Back in the day…
LLOOGG
Google Analytics + Realtime

7. Back in the day…
LLOOGG
Google Analytics + Realtime

8. Back in the day…
“
LLOOGG is working very well…
Redis
We never experienced any stability problem
– Salvatore Sanfilippo
Google Analytics + Realtime

9. Back in the day…
community growth
@simonw
Simon Willison
@antirez just posted a redis feature request
http://groups.google.com/group/redis-db/.../cd7e7c65dba53e27
- a way of randomly fetching an item from a set without also
deleting it
20 Oct 09 via web

10. Back in the day…
community growth
@simonw
Simon Willison
@antirez just posted a redis feature request
http://groups.google.com/group/redis-db/.../cd7e7c65dba53e27
- a way of randomly fetching an item from a set without also
deleting it
20 Oct 09 via web

11. Back in the day…
VMWare acquisition

12. A little server of awesome
man, that’s totally stolen from Simon Willison
http://www.flickr.com/photos/ajc1/4663140532

13. Redis Manifesto
1. A DSL for Abstract Data Types
2. Memory storage is #1
3. Fundamental data structures for a fundamental API
4. Code is like a poem
5. We’re against complexity
6. Two levels of API
7. We optimize for joy

14. A DSL For Abstract Data Types
a key/value store?
redis> SET key hello
OK
redis> GET key
“hello”

15. A DSL For Abstract Data Types
a key/value store?
request
redis> SET key hello
OK
redis> GET key
“hello”

16. A DSL For Abstract Data Types
a key/value store?
request
redis> SET key hello
OK awesome, a REPL!
redis-cli
redis> GET key
“hello”

17. A DSL For Abstract Data Types
on par with memcache
redis> APPEND key world
(integer) 10
redis> SETNX key bye
(integer) 1
redis> GET key
“helloworld”
redis> DEL key
(integer) 1

18. A DSL For Abstract Data Types
on par with memcache: atomic counters
redis> SET participants 10
OK
redis> INCR participants
(integer) 11
“It gives Memcached a serious run for its money.”
– Mathias Meyer, Peritor GmbH

19. A DSL For Abstract Data Types
on par with memcache: cache invalidation
redis> EXPIRE key 60
OK
redis> ...

20. A DSL For Abstract Data Types
on par with memcache: cache invalidation
redis> EXPIRE key 60
OK
redis> TTL key
(integer) 53
redis>

21. A DSL For Abstract Data Types
on par with memcache: cache invalidation
redis> EXPIRE key 60
OK
redis> TTL key
(integer) 53
redis> GET key
“helloworld”
redis>

22. A DSL For Abstract Data Types
on par with memcache: cache invalidation
redis> EXPIRE key 60
OK
redis> TTL key
(integer) 53
redis> GET key
“helloworld”
redis> ...

23. A DSL For Abstract Data Types
on par with memcache: cache invalidation
redis> EXPIRE key 60
OK
redis> TTL key
(integer) 53
redis> GET key
“helloworld”
redis> GET key
(nil)

24. “
Memory is
the new disk.
–– Jim Gray, Turing Award laureate

25. Memory storage is #1
Redis key space
http://spotfireblog.tibco.com/wp-content/uploads/in-memory-analytics.jpg

26. Memory storage is #1
snapshotting mode
http://www.flickr.com/photos/bionicteaching/3212235059/

27. Memory storage is #1
snapshotting mode
every x seconds
every y operations
on SAVE / BGSAVE
http://www.flickr.com/photos/bionicteaching/3212235059/

28. Memory storage is #1
snapshotting mode
fork(); // copy on write!
dump();
move();
every x seconds
every y operations
on SAVE / BGSAVE
http://www.flickr.com/photos/bionicteaching/3212235059/

29. Memory storage is #1
Append-Only File – Write-Ahead Logging
http://www.flickr.com/photos/generationbass/4827013488/

30. Memory storage is #1
Append-Only File – Write-Ahead Logging
fsync() on every command
http://www.flickr.com/photos/generationbass/4827013488/

31. Memory storage is #1
Append-Only File – Write-Ahead Logging
fsync() on every command
fsync() every second
http://www.flickr.com/photos/generationbass/4827013488/

32. Memory storage is #1
Append-Only File – Write-Ahead Logging
fsync() on every command
fsync() every second
fsync() per OS
http://www.flickr.com/photos/generationbass/4827013488/

33. Memory storage is #1
Append-Only File – Write-Ahead Logging
fsync() on every command
fsync() every second
fsync() per OS
BGREWRITEAOF trims AOF
http://www.flickr.com/photos/generationbass/4827013488/

34. Memory storage is #1
meh, persistence
memory
overhead
http://www.flickr.com/photos/generationbass/4827013488/

35. Memory storage is #1
memory contention
?

36. Memory storage is #1
memory contention
?

37. Memory storage is #1
memory contention
Virtual Memory to the rescue
?

38. Memory storage is #1
memory contention
Virtual Memory to the rescue
?
because Salvatore is much brighter than
all of Microsoft Research

39. Memory storage is #1
memory contention
Virtual Memory to the rescue
?
because Salvatore is much brighter than
all of Microsoft Research
…and Redis types do not map to OS pages

40. Memory storage is #1
memory contentionstay in memory
all keys need to
values must be swapped in their entirety
swapped by age and size
Virtual Memory to the rescue
?
because Salvatore is much brighter than
all of Microsoft Research
…and Redis types do not map to OS pages

41. Fundamental data structures
for a fundamental API

42. Fundamental data structures
for a fundamental API
lists, we can
redis> LPUSH databases mysql
(integer) 1
redis> LPUSH databases mongodb
(integer) 2

43. Fundamental data structures
for a fundamental API
lists, we can
redis> LPUSH databases mysql
(integer) 1
redis> LPUSH databases mongodb
(integer) 2
redis> LRANGE databases 0 -1
1) “mongodb”
2) “mysql”

44. Fundamental data structures
for a fundamental API
lists, we can
redis> LPUSH databases mysql
(integer) 1
redis> LPUSH databases mongodb
(integer) 2
redis> LRANGE databases 0 -1
1) “mongodb”
Bulk reply
2) “mysql”

45. Fundamental data structures
for a fundamental API
lists, we can
redis> LPUSH databases mysql
implemented as Linked Lists
(integer) 1
redis> LPUSH databases mongodb
(integer) 2
redis> LRANGE databases 0 -1
1) “mongodb”
Bulk reply
2) “mysql”

46. Fundamental data structures
for a fundamental API
lists, we can
redis> LPUSH databases mysql
implemented as Linked Lists
(integer) 1
redis> LPUSH databases mongodb
(integer) 2
PUSH
redis>
POP
O(1)
LRANGE databases 0 -1
1) “mongodb”
Bulk reply
2) “mysql”

47. Fundamental data structures
for a fundamental API
a data structure server
lists sets
sorted hash
sets tables

48. Fundamental data structures
for a fundamental API
a data structure server
lists sets
sorted hash
sets tables

49. Fundamental data structures
for a fundamental API
a data structure server
lists sets
sorted hash
sets tables

50. APPEND key value RENAME key newkey
BLPOP key [key ...] RENAMENX key newkey
BRPOP key [key ...] timeout RPOP key
Fundamental data structures
BRPOPLPUSH source destination timeout
DECR key
DECRBY key decrement
RPOPLPUSH source destination
RPUSH key value
for a fundamental API
DEL key [key ...] RPUSHX key value
EXISTS key SADD key member
EXPIRE key seconds SCARD key
EXPIREAT key timestamp SDIFF key [key ...]
a data processing server
FLUSHALL R
FLUSHDB
GET key
SDIFFSTORE destination key [key ...]
SELECT index
SET key value
GETBIT key offset SETBIT key offset value
GETRANGE key start end SETEX key seconds value
GETSET key value SETNX key value
HDEL key field SETRANGE key offset value
HEXISTS key field SINTER key [key ...]
HGET key field SINTERSTORE destination key [key ...]
HGETALL key SISMEMBER key member
HINCRBY key field increment SMEMBERS key
HKEYS key SMOVE source destination member
HLEN key SORT key [BY pattern] [LIMIT offset count] [GET pattern [GET pattern ...]] [ASC|DESC] [ALPHA]
HMGET key field [field ...] SPOP key
HMSET key field value [field value ...] SRANDMEMBER key
HSET key field value SREM key member
HSETNX key field value STRLEN key
HVALS key SUNION key [key ...]
INCR key SUNIONSTORE destination key [key ...]
INCRBY key increment TTL key
KEYS pattern TYPE key
LINDEX key index ZADD key score member
LINSERT key BEFORE|AFTER pivot value ZCARD key
LLEN key ZCOUNT key min max
LPOP key ZINCRBY key increment member
LPUSH key value ZINTERSTORE destination numkeys key [key ...]
LPUSHX key value ZRANGE key start stop [WITHSCORES]
LRANGE key start stop ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count]
LREM key count value ZRANK key member
LSET key index value ZREM key member
LTRIM key start stop ZREMRANGEBYRANK key start stop
MGET key [key ...] ZREMRANGEBYSCORE key min max
MOVE key db ZREVRANGE key start stop [WITHSCORES]
MSET key value [key value ...] ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count]
MSETNX key value [key value ...] ZREVRANK key member
PERSIST key ZSCORE key member
RANDOMKEY ZUNIONSTORE destination numkeys key [key ...]

51. APPEND key value RENAME key newkey
BLPOP key [key ...] RENAMENX key newkey
BRPOP key [key ...] timeout RPOP key
Fundamental data structures
BRPOPLPUSH source destination timeout
DECR key
DECRBY key decrement
RPOPLPUSH source destination
RPUSH key value
for a fundamental API
DEL key [key ...] RPUSHX key value
EXISTS key SADD key member
EXPIRE key seconds SCARD key
EXPIREAT key timestamp SDIFF key [key ...]
a data processing server
FLUSHALL R
FLUSHDB
GET key
SDIFFSTORE destination key [key ...]
SELECT index
SET key value
GETBIT key offset SETBIT key offset value
GETRANGE key start end SETEX key seconds value
GETSET key value SETNX key value
HDEL key field SETRANGE key offset value
HEXISTS key field SINTER key [key ...]
HGET key field SINTERSTORE destination key [key ...]
HGETALL key SISMEMBER key member
HINCRBY key field increment SMEMBERS key
HKEYS key SMOVE source destination member
HLEN key SORT key [BY pattern] [LIMIT offset count] [GET pattern [GET pattern ...]] [ASC|DESC] [ALPHA]
HMGET key field [field ...] SPOP key
HMSET key field value [field value ...] SRANDMEMBER key
HSET key field value SREM key member
HSETNX key field value STRLEN key
HVALS key SUNION key [key ...]
INCR key SUNIONSTORE destination key [key ...]
INCRBY key increment TTL key
every operation is atomic
KEYS pattern TYPE key
LINDEX key index ZADD key score member
LINSERT key BEFORE|AFTER pivot value ZCARD key
LLEN key ZCOUNT key min max
LPOP key ZINCRBY key increment member
LPUSH key value ZINTERSTORE destination numkeys key [key ...]
LPUSHX key value ZRANGE key start stop [WITHSCORES]
LRANGE key start stop ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count]
LREM key count value ZRANK key member
LSET key index value ZREM key member
LTRIM key start stop ZREMRANGEBYRANK key start stop
MGET key [key ...] ZREMRANGEBYSCORE key min max
MOVE key db ZREVRANGE key start stop [WITHSCORES]
MSET key value [key value ...] ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count]
MSETNX key value [key value ...] ZREVRANK key member
PERSIST key ZSCORE key member
RANDOMKEY ZUNIONSTORE destination numkeys key [key ...]

52. APPEND key value RENAME key newkey
BLPOP key [key ...] RENAMENX key newkey
BRPOP key [key ...] timeout RPOP key
Fundamental data structures
BRPOPLPUSH source destination timeout
DECR key
DECRBY key decrement
RPOPLPUSH source destination
RPUSH key value
for a fundamental API
DEL key [key ...] RPUSHX key value
EXISTS key SADD key member
EXPIRE key seconds SCARD key
EXPIREAT key timestamp SDIFF key [key ...]
a data processing server
FLUSHALL R
FLUSHDB
GET key
SDIFFSTORE destination key [key ...]
SELECT index
SET key value
GETBIT key offset SETBIT key offset value
GETRANGE key start end SETEX key seconds value
GETSET key value SETNX key value
HDEL key field SETRANGE key offset value
HEXISTS key field SINTER key [key ...]
HGET key field SINTERSTORE destination key [key ...]
HGETALL key SISMEMBER key member
HINCRBY key field increment SMEMBERS key
HKEYS key SMOVE source destination member
HLEN key SORT key [BY pattern] [LIMIT offset count] [GET pattern [GET pattern ...]] [ASC|DESC] [ALPHA]
HMGET key field [field ...] SPOP key
HMSET key field value [field value ...] SRANDMEMBER key
HSET key field value SREM key member
HSETNX key field value STRLEN key
HVALS key SUNION key [key ...]
INCR key SUNIONSTORE destination key [key ...]
INCRBY key increment TTL key
every operation is atomic
KEYS pattern TYPE key
LINDEX key index ZADD key score member
LINSERT key BEFORE|AFTER pivot value ZCARD key
LLEN key ZCOUNT key min max
LPOP key ZINCRBY key increment member
LPUSH key value ZINTERSTORE destination numkeys key [key ...]
LPUSHX key value ZRANGE key start stop [WITHSCORES]
LRANGE key start stop ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count]
LREM key count value ZRANK key member
LSET key index value ZREM key member
LTRIM key start stop
MGET key [key ...]
MOVE key db
MSET key value [key value ...]
MSETNX key value [key value ...]
ZREMRANGEBYRANK key start stop
ZREMRANGEBYSCORE key min max
Mind blown.
ZREVRANGE key start stop [WITHSCORES]
ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count]
ZREVRANK key member
PERSIST key ZSCORE key member
RANDOMKEY ZUNIONSTORE destination numkeys key [key ...]

53. Fundamental data structures
for a fundamental API
CAP & ACID promises
atomicity
consistency availability durability
consistency
isolation
partition
tolerance
“kicks ACID out the door”
–Mathias Meyer, Chief Cloud Officer

54. Fundamental data structures
for a fundamental API
CAP & ACID promises
atomicity
consistency “that’s
availability like saying durability
consistency
why aren’t filesystems ACID”
isolation
partition
tolerance
“kicks ACID out the door”
–Mathias Meyer, Chief Cloud Officer

55. Code is like a poem
C
23.000 lines of code

56. We’re against complexity

57. Two levels of APIs
transactions
redis> MULTI
OK
redis> GET key
QUEUED
redis> DEL key
QUEUED
redis> EXEC
1) “helloworld”
2) (integer) 1

58. Two levels of APIs
transactions
redis> MULTI
OK
redis> GET key
QUEUED
redis> DEL key
QUEUED
redis> EXEC DISCARD
1) “helloworld”
2) (integer) 1

59. Two levels of APIs
transactions
redis> MULTI
OK
redis> GET key
QUEUED atomic
redis> DEL key
QUEUED
redis> EXEC
1) “helloworld”
2) (integer) 1

60. Two levels of there’s always a but:
APIs
server crash = partial execution
transactions
redis> MULTI
OK
redis> GET key
QUEUED atomic
redis> DEL key
QUEUED
redis> EXEC
1) “helloworld”
2) (integer) 1

61. Two levels of there’s always a but:
APIs
server crash = partial execution
transactions
redis> MULTI
OK there’s always a but but:
redis> GET key
detected on startup atomic
QUEUED => exit
redis> DEL key
QUEUED
redis> EXEC
1) “helloworld”
2) (integer) 1

62. Two levels of there’s always a but:
APIs
server crash = partial execution
transactions
redis> MULTI
OK there’s always a but but:
redis> GET key
detected on startup atomic
QUEUED => exit
redis> DEL key
QUEUED always a but but but:
there’s
redis> EXEC
redis-check-aof repairs it
1) “helloworld”
2) (integer) 1

63. Two levels of APIs
optimistic locking
redis> WATCH key
OK
redis> MULTI
OK

64. Two levels of APIs
optimistic locking
redis> WATCH key
OK
redis> MULTI
OK
SET key ciao <redis
OK

65. Two levels of APIs
optimistic locking
redis> WATCH key
OK
redis> MULTI
OK
SET key ciao <redis
OK
redis> SET key bye
QUEUED
redis> EXEC
(nil)

66. Two levels of APIs
optimistic locking
redis> WATCH key
OK
redis> MULTI
OK
SET key ciao <redis
OK
redis> SET key bye
QUEUED
redis> EXEC
(nil) GET key <redis
“ciao”

67. We optimize for joy
Pub/Sub
redis> SUBSCRIBE chat
Reading messages… (press Ctrl-C to quit)
1) “subscribe”
2) “chat”
3) (integer) 1

68. We optimize for joy
Pub/Sub
redis> SUBSCRIBE chat
Reading messages… (press Ctrl-C to quit)
1) “subscribe” type
2) “chat”
3) (integer) 1

69. We optimize for joy
Pub/Sub
redis> SUBSCRIBE chat
Reading messages… (press Ctrl-C to quit)
1) “subscribe”
2) “chat” channel
3) (integer) 1

70. We optimize for joy
Pub/Sub
redis> SUBSCRIBE chat
Reading messages… (press Ctrl-C to quit)
1) “subscribe”
2) “chat”
3) (integer) 1 message

71. We optimize for joy
Pub/Sub
redis> SUBSCRIBE chat
Reading messages… (press Ctrl-C to quit)
1) “subscribe”
2) “chat”
3) (integer) 1
PUBLISH chat “asl?” <redis
(integer) 1

72. We optimize for joy
Pub/Sub
redis> SUBSCRIBE chat
Reading messages… (press Ctrl-C to quit)
1) “subscribe”
2) “chat”
3) (integer) 1
PUBLISH chat “asl?” <redis
(integer) 1
1) “message”
2) “chat”
3) “asl?”
^C

73. We optimize for joy
Pub/Sub
redis> SUBSCRIBE chat
Reading messages… (press Ctrl-C to quit)
1) “subscribe”
2) “chat”
3) (integer) 1
PUBLISH chat “asl?” <redis
(integer) 1
1) “message”
2) “chat” in other news:
3) “asl?” PSUBSCRIBE
^C UNSUBSCRIBE

74. “
Redis is
more than just
a key-value store,
it’s a lifestyle.
–– Mathias Meyer

75. Benchmarks
redis-benchmark
$ redis-benchmark
====== SET ======
100000 requests completed in 0.88 seconds
50 parallel clients
3 bytes payload
====== GET ======
100000 requests completed in 1.23 seconds
Linux 2.6, Xeon X3320 2.5GHz, loopback

76. Benchmarks
Colin Howe
3.7x writes
speed improvement for over MySQL
2.1x reads

77. Replication
master-slave scenario
http://www.flickr.com/photos/joiearai/4393860397/

78. Replication
master-slave scenario
http://www.flickr.com/photos/joiearai/4393860397/

79. Replication
master-slave scenario
http://www.flickr.com/photos/joiearai/4393860397/

80. Replication
master-slave scenario
SLAVEOF host port
http://www.flickr.com/photos/joiearai/4393860397/

81. Replication
master-slave scenario
SLAVEOF host port
save + re-issuing
http://www.flickr.com/photos/joiearai/4393860397/

82. Who’s using it
Github
Resque
message queue service
Configuration Management
Routing Information

83. Who’s using it
Digg
Digg Streaming API
personalized news data
real time view
click counts

84. Who’s using it
wooga
Monster World
inventory system
Happy Hospital
user system

85. Who’s using it
Peter Noordhuis
# Author: Peter Noordhuis
# Description: Simple demo to showcase Redis PubSub with EventMachine
<170 lines of Ruby code>
<220 lines of HTML+JS>

86. Who’s using it
Hurl
HTTP request testing
WildlifeNearYou
API rate limiting
Crowdsourcing taxonomies
Vanity
A/B testing
The Guardian
Crowdsourced analysis of MP expenses
BNP membership list analysis

87. Who’s using it
Hurl
HTTP request testing
WildlifeNearYou
API rate limiting
Crowdsourcing taxonomies
general blogosphere hype storm
http://redis.io/topics/using-redis lists about 30 others
Vanity
A/B testing
The Guardian
Crowdsourced analysis of MP expenses
BNP membership list analysis

88. “
I think the… problem is…
believing that
there can be
“one true datastore”.
Different technologies
excel at different things.
–– Weixi Yen

89. Check out…
Simon Willison’s tutorial
http://simonwillison.net/static/2010/redis-tutorial

90. Check out…
Redis, from the Ground Up
http://blog.mjrusso.com/2010/10/17/redis-from-the-ground-up
Try Redis The Redis Cookbook
http://try.redis-db.com/ http://rediscookbook.org/
Redis: The Definitive Guide
Data modeling, caching, and messaging
coming August 2011, Salvatore & Pieter, published by O’Reilly
also check my Redis bookmarks: http://del.icio.us/lehmannro/redis

91. man, that was a hell of a ride.
Thanks.
Questions?
Now go use Redis.io!