Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2Ezs08q. Justin Ryan talks about Netflix’ scalability issues and some of the ways they addressed it. He shares successes they’ve had from unintuitively partitioning computation into multiple services to get better runtime characteristics. He introduces us to useful probabilistic data structures, innovative bi-directional data passing, open-source projects available from Netflix that make this all possible. Filmed at qconsf.com. Justin Ryan is Playback Edge Engineering at Netflix. He works on some of the most critical services at Netflix, specifically focusing on user and device authentication. Years of building developer tools has also given him a healthy set of opinions on developer productivity.

1. Justin Ryan, Senior Software Engineer @ Netflix
Twitter: quidryan, LinkedIn: www.linkedin.com/in/justin-charles-ryan
Scaling
Patterns for
Netflix’s Edge
November 2019

2. InfoQ.com: News & Community Site
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
netflix-edge-scalability-patterns/

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. Scaling Patterns for Netflix’s Edge
Justin Ryan
Laundry

5. Scaling Patterns for Netflix’s Edge
Justin Ryan
Laundry

6. Scaling Patterns for Netflix’s Edge
Justin Ryan
Laundry

7. Scaling Patterns for Netflix’s Edge
Justin Ryan
Laundry

8. Scaling Patterns for Netflix’s Edge
Justin Ryan
Space-Time Compromise

9. Scaling Patterns for Netflix’s Edge
Justin Ryan
Laundry

10. Scaling Patterns for Netflix’s Edge
Justin Ryan
Scaling
Q32011
Q32019

11. Scaling Patterns for Netflix’s Edge
Justin Ryan
Scaling
Edge

12. Scaling Patterns for Netflix’s Edge
Justin Ryan
Scaling
Problems
— Debuggability
— Infrastructure
— Managing

13. Scaling Patterns for Netflix’s Edge
Justin Ryan
Scalability

14. Scaling Patterns for Netflix’s Edge
Justin Ryan
Trade Offs

15. Scaling Patterns for Netflix’s Edge
Justin Ryan
Policies

16. Scaling Patterns for Netflix’s Edge
Justin Ryan
Accessible

17. Scaling Patterns for Netflix’s Edge
Justin Ryan
Five Use
Cases
— Melnitz
— Mantis
— Passport
— Device Types
— Sharding

18. Scaling Patterns for Netflix’s Edge
Justin Ryan
Logged In

19. Scaling Patterns for Netflix’s Edge
Justin Ryan
Cookies

20. Scaling Patterns for Netflix’s Edge
Justin Ryan
Melnitz
Trust Cookie Cookie Expiration

21. Scaling Patterns for Netflix’s Edge
Justin Ryan
Cookies
L7 Proxy
Account
Service
API
Expired
Yes
No

22. Scaling Patterns for Netflix’s Edge
Justin Ryan
Boom

23. Scaling Patterns for Netflix’s Edge
Justin Ryan
Melnitz
Copyright Columbia Pictures

24. Scaling Patterns for Netflix’s Edge
Justin Ryan
Melnitz
Set<CustomerId> Cookie Expiration

25. Scaling Patterns for Netflix’s Edge
Justin Ryan
Melnitz Bloom Filter: Compact Set which
can answer the question is an object
“Possibly in Set”?

26. Scaling Patterns for Netflix’s Edge
Justin Ryan
Melnitz

27. Scaling Patterns for Netflix’s Edge
Justin Ryan
Side Note:
Bloom Filter
— Bad Interview Question? Yes
— Used in complex databases? Yes
— Guava Library? Yes
— Usable by you in your day-to-day
jobs? Yes

28. Scaling Patterns for Netflix’s Edge
Justin Ryan
Melnitz
L7 Proxy
Account
Service
API
Bloom
Filter
Expired
Yes
No
Maybe
No

29. Scaling Patterns for Netflix’s Edge
Justin Ryan
Melnitz
Account
Service Auth
Event
Melnitz A
DB
Melnitz B
1m
L7 Proxy

30. Scaling Patterns for Netflix’s Edge
Justin Ryan
Melnitz

31. Scaling Patterns for Netflix’s Edge
Justin Ryan
Melnitz
Trade Offs
— Probabilistic Data Structure
— Off the Shelf Components

32. Scaling Patterns for Netflix’s Edge
Justin Ryan
Side Note: Bloom Filter

33. Scaling Patterns for Netflix’s Edge
Justin Ryan
Five Use
Cases
— Melnitz - Distributed Bloom Filter
— Mantis
— Passport
— Device Types
— Sharding

34. Scaling Patterns for Netflix’s Edge
Justin Ryan
Mantis

35. Scaling Patterns for Netflix’s Edge
Justin Ryan
Mantis 2 million request per second *
3 kilobytes per request *
10 micro-services =
57 GB/s =~ 4.6 PB / day

36. Scaling Patterns for Netflix’s Edge
Justin Ryan
Naive Approach
Service Elastic Search
Kibana

37. Scaling Patterns for Netflix’s Edge
Justin Ryan
Mantis

38. Scaling Patterns for Netflix’s Edge
Justin Ryan
Mantis Master
Mantis
Service
Mantis
Agent
Mantis
Source
Job
Elastic Search
Kibana
Kafka

39. Scaling Patterns for Netflix’s Edge
Justin Ryan
Mantis Master
Service
Mantis
Mantis
Source
Job
Elastic Search
Browser
Mantis
Agent
Kafka

40. Scaling Patterns for Netflix’s Edge
Justin Ryan
Mantis Master
Mantis
Service
Mantis
Agent
Mantis
Source
Job
Elastic Search
Browser
Metrics
Javascript
Transform

41. Scaling Patterns for Netflix’s Edge
Justin Ryan
Avoid Work

42. Scaling Patterns for Netflix’s Edge
Justin Ryan
Mantis
Trade Offs
— Mantis Query Language
— Scale Solution to Zero

43. Scaling Patterns for Netflix’s Edge
Justin Ryan
Five Use
Cases
— Melnitz - Distributed Bloom Filter
— Mantis - Event Querying
— Passport
— Device Types
— Sharding

44. Scaling Patterns for Netflix’s Edge
Justin Ryan
Passport

45. Scaling Patterns for Netflix’s Edge
Justin Ryan
Service
Plan
API
L7Proxy
Mid Tier A
Mid Tier B
Mid Tier C
Passport

46. Scaling Patterns for Netflix’s Edge
Justin Ryan
Passport
Service
Plan
API
L7Proxy
Mid Tier A
Mid Tier B
Mid Tier C

47. Scaling Patterns for Netflix’s Edge
Justin Ryan
Cache
Service
Plan
API
L7Proxy
Mid Tier A
Mid Tier B
Mid Tier C

48. Scaling Patterns for Netflix’s Edge
Justin Ryan
Cache
API
L7Proxy
Mid Tier A
Mid Tier B
Mid Tier C
Service
Plan

49. Scaling Patterns for Netflix’s Edge
Justin Ryan
Cache
API
L7Proxy
Mid Tier A
Mid Tier B
Mid Tier C
Service
Plan

50. Scaling Patterns for Netflix’s Edge
Justin Ryan
Passport

51. Scaling Patterns for Netflix’s Edge
Justin Ryan
Passport
Trade Offs
— Data Passing
— Heavy Data Structures

52. Scaling Patterns for Netflix’s Edge
Justin Ryan
Five Use
Cases
— Melnitz - Distributed Bloom Filter
— Mantis - Event Querying
— Passport - Heavy Data Structure
— Device Types
— Sharding

53. Scaling Patterns for Netflix’s Edge
Justin Ryan
Device Types

54. Scaling Patterns for Netflix’s Edge
Justin Ryan
Device Types
NFAPPL-01-IPAD7=5-09BEC57F64CA5F837C75F924B94D
NFAPPL-01-IPAD3=4-15B2D9B33A994CC373017A9C6586
NFAPPL-02-IOS2=4-FT7WHG8H2UR2MCJ4YKR3P0DFH6
NFANDROID1-PRV-P-GOOGLPIXEL=2-7351-0F441DF98B8D99074
NFANDROID1-PRV-4339-0E50FC7582396F5C4496ED935F2F48ED
1395
1007
964

55. Scaling Patterns for Netflix’s Edge
Justin Ryan
Device Types

56. Scaling Patterns for Netflix’s Edge
Justin Ryan
Device Types

57. Scaling Patterns for Netflix’s Edge
Justin Ryan
Device Types
Mid-Tier F
Mid-Tier E Mid-Tier D
Mid-Tier C
Mid-Tier BMid-Tier A
DTS

58. Scaling Patterns for Netflix’s Edge
Justin Ryan
Device Types
Mid-Tier CMid-Tier BMid-Tier A
DTS PubSub

59. Scaling Patterns for Netflix’s Edge
Justin Ryan
Device Types
Mid-Tier CMid-Tier BMid-Tier A
DTS PubSub
Device Types Device TypesDevice Types

60. Scaling Patterns for Netflix’s Edge
Justin Ryan
Device Types

61. Scaling Patterns for Netflix’s Edge
Justin Ryan
Device Type
Trade Offs
— Business Tuned Fallbacks
— Leverage Existing Infrastructure

62. Scaling Patterns for Netflix’s Edge
Justin Ryan
Five Use
Cases
— Melnitz - Distributed Bloom Filter
— Mantis - Event Querying
— Passport - Heavy Data Structure
— Device Types - Device Grouping
— Sharding

63. Scaling Patterns for Netflix’s Edge
Justin Ryan
Sharding

64. Scaling Patterns for Netflix’s Edge
Justin Ryan
MSL — Message Security Layer == MSL
— Secure Messaging Framework

65. Scaling Patterns for Netflix’s Edge
Justin Ryan
Sharding MSL
L7Proxy
APIAPIAPIAPIAPIAPI

66. Scaling Patterns for Netflix’s Edge
Justin Ryan
Sharding MSL
L7Proxy
MSL
APIAPIAPI

67. Scaling Patterns for Netflix’s Edge
Justin Ryan
CPU Cost Per Request
- CPU to RPS ratio
- Load Average
31%
decrease

68. Scaling Patterns for Netflix’s Edge
Justin Ryan
Latency
30%
decrease
20% p99
decrease

69. Scaling Patterns for Netflix’s Edge
Justin Ryan
Sharding MSL

70. Scaling Patterns for Netflix’s Edge
Justin Ryan
Sharding MSL
- % of Wall Clock in GC
- GC Pause Time

71. Scaling Patterns for Netflix’s Edge
Justin Ryan
Sharding MSL
- Average Latency
- 99% Latency

72. Scaling Patterns for Netflix’s Edge
Justin Ryan
Sharding
3 = 1 + 1

73. Scaling Patterns for Netflix’s Edge
Justin Ryan
Sharding
Trade Offs
— Operational Overhead

74. Scaling Patterns for Netflix’s Edge
Justin Ryan
Five Use
Cases
— Melnitz - Distributed Bloom Filter
— Mantis - Event Querying
— Passport - Heavy Data Structure
— Device Types - Device Grouping
— Sharding - Runtime Refactoring

75. Scaling Patterns for Netflix’s Edge
Justin Ryan
Trough

76. Scaling Patterns for Netflix’s Edge
Justin Ryan
Trough - Used Instances
- Reserved Instances

77. Scaling Patterns for Netflix’s Edge
Justin Ryan
Pattern #6
Trough
— Leverage Unused Resources

78. Scaling Patterns for Netflix’s Edge
Justin Ryan
Trade Odds
● Probabilistic Data Structure
● Off the Shelf Components
● Avoid Work
● Scale Solution to Zero
● Data Passing
● Heavy Data Structures
● Leverage Existing Infrastructure
● Business Tuned Fallbacks
● Runtime Refactoring
Melnitz
Melnitz
Mantis
Mantis
Passport
Passport
DTS
DTS
Sharding

79. Scaling Patterns for Netflix’s Edge
Justin Ryan
When to Use
Debuggability ✗ ✓ ✓ ✓ ✓ ✗
Infrastructure ✓ ✗ ✗ ✓ ✗ ✗
Managing ✗ ✗ ✗ ✓ ✗ ✗
Costs ✓ ✓ ✗ ✗ ✓ ✓
Melnitz
Mantis
Passport
DeviceTypesSharding
Trough

80. Scaling Patterns for Netflix’s Edge
Justin Ryan
Bonus Trade
Offs
— Do the laundry

81. Justin Ryan
jryan@netflix.com
Twitter: @quidryan
Thank
You.

82. Scaling Patterns for Netflix’s Edge
Justin Ryan
References — github.com/Netflix
◆ eureka
◆ zuul
◆ mantis
◆ msl
— Spinnaker
— Apache Pulsar
— Google Guava (Bloom Filter)

83. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
netflix-edge-scalability-patterns/