Abstract:- The Netflix data platform is constantly evolving, but at it's core, it's an all-cloud platform at a massive scale (60+ PB and over 700 billion new events per day), focused on enabling developers. In this talk, we'll dive into the current (data) technology landscape at Netflix, as well as what's in the works. We'll cover key technologies, such as Spark, Presto, Docker, and Jupyter, along with many broader data ecosystem facets (metadata, insights into jobs run, visualizing big data, etc.). Beyond just tech, we'll also dive a bit into our data platform philosophy. You'll leave with insights into how things work at Netflix, along with some ideas for re-envisioning your data platform.

1. The Netflix data platform:
Now and in the future

3. By BlueMint – https://commons.wikimedia.org/wiki/File:Netflixenvelope.jpg Under CC BY 2.5 License – https://creativecommons.org/licenses/by/2.5/

5. S3

6. Source of Truth

7. Data Processing

8. • 25% to 40% jobs
• Scala and Python
• 2.0+ J
• Tech Foundation + Community

9. • Was a great language / served us well
• (Long-term) path to zero
• Don’t create work for yourself

10. • Old friend – will go too
• No Tez – Focus on Spark
• Flip all to Spark SQL?

11. • Why not Spark SQL?
• Great for querying, but ETL???
• Impact over idealism

12. • “Niche” data processing
• Smoothing the flow
• One size does not fit all

13. Services

15. curl -X POST -H "Content-Type: application/json" -H "Authorization: Bearer
{some token}" -d '{
"id":"3d922d7a-8506-11e6-bdc4-0242ac210002",
"tags":[ "headers","PrestoJob","BigDataPortal"],
"version":"NA", "user":"someUser",
"name":"BigDataPortal.trit.PrestoJob.1474317252962",
"description":"{"username": "root", "host": "someHost", "client": "nflx-
kragle-djinn/0.3.4", "job_class": "PrestoJob"}",
"setupFile":null,"commandArgs":"--execute "select * from some table"",
"clusterCriterias":[{"tags":["type:presto","data:prod"]}],
"commandCriteria":["type:presto"],
"group":null,"disableLogArchival":true,"email":null,
"cpu":null,"memory":null,"timeout":172800,
"dependencies":[],
"applications":[]
}' "https://genie.adomain.com/api/v3/jobs"

16. Common Interface

17. Metacat

18. Metastore

19. • Common interface
• Extended Metadata
• Optimizations
• Notifications
• Canonical Data Types vs.
Metacat

20. • Lineage à Big picture clarity
• Clutter?
• Who will be affected?
• Messy (ongoing) – feedback loops needed
Charlotte

21. IFTTT for Data
Microbots

22. Cloud Resources

23. Containers

24. Higher-level interfaces

25. import kragle as kg
trans_info = kg.transport.Transporter()
.source('metacat://prodhive/default/my_table')
.target('metacat://redshift/test/demo_table')
.execute()
Kragle
(aka Big Data API)

26. Big Data Portal

30. Virtuous Cycle

38. • “Google Docs” like collaboration
• Scala Kernel for Jupyter
• Smooth dev to prod flow
Notebooks

39. Machine Learning Infrastructure

40. Points of View
& Ecosystem(s)

41. Questions?
http://jobs.netflix.com/
kurtbrown@netflix.com