SlideShare a Scribd company logo
1 of 44
1 
Personalized Page Generation for Browsing 
Recommendations 
Justin Basilico 
Page Algorithms Engineering October 10, 2014 
@JustinBasilico 
Workshop 2014
2 
?
3 
Evolution of our Recommendation Approach 
4.7 
Rating Ranking Page Generation
4 
 Interested in high-quality 
recommendations 
 Proxy question: 
 Accuracy in predicted rating 
 Measured by root mean 
squared error (RMSE) 
 Improve by 10% = $1 million! 
 Data size: 
 100M ratings (back then 
“almost massive”)
5 
Change of focus 
2006 2014
6 
Netflix Scale 
 > 50M members 
 > 40 countries 
 > 1000 device types 
 Hours: > 2B/month 
 Plays: > 30M/day 
 Log 100B events/day 
 34.2% of peak US 
downstream traffic
7 
“Emmy Winning” 
Approach to Recommendation
8 
Goal 
Help members find content to watch and enjoy 
to maximize member satisfaction and retention
9 
Everything is a Recommendation 
Rows 
Ranking 
Over 75% of what 
people watch 
comes from our 
recommendations 
Recommendations 
are driven by 
Machine Learning
10 
Top Picks 
Personalization awareness 
Diversity
11 
Personalized genres 
 Genres focused on user interest 
 Derived from tag combinations 
 Provide context and evidence 
 How are they generated? 
 Implicit: Based on recent plays, 
ratings & other interactions 
 Explicit: Taste preferences
12 
Similarity 
 Recommend videos similar 
to one you’ve liked 
 “Because you watched” 
rows 
 Pivots 
 Video information page 
 In response to user actions 
(search, list add, …)
13 
Support for Recommendations 
Behavioral Support Social Support
14 
Learning to Recommend
15 
Machine Learning Approach 
Problem 
Data 
Metrics 
Model Algorithm
16 
Data 
 Plays 
 Duration, bookmark, time, 
device, … 
 Ratings 
 Metadata 
 Tags, synopsis, cast, … 
 Impressions 
 Interactions 
 Search, list add, scroll, … 
 Social
17 
Models & Algorithms 
 Regression (Linear, logistic, elastic net) 
 SVD and other Matrix Factorizations 
 Factorization Machines 
 Restricted Boltzmann Machines 
 Deep Neural Networks 
 Markov Models and Graph Algorithms 
 Clustering 
 Latent Dirichlet Allocation 
 Gradient Boosted Decision 
Trees/Random Forests 
 Gaussian Processes 
 …
18 
Personalized Rating Prediction
19 
Rating Prediction 
 Based on first year progress prize 
 Top 2 algorithms 
 Matrix Factorization (SVD++) 
 Restricted Boltzmann Machines 
(RBM) 
 Ensemble: Linear blend 
Videos 
R 
≈ 
Users 
U 
V 
(99% Sparse) d 
Videos 
Users 
× d
20 
Ranking by ratings 
4.7 4.6 4.5 4.5 4.5 4.5 4.5 4.5 4.5 4.5 
Niche titles 
High average ratings… by those who would watch it
21 
RMSE
22 
Personalized Ranking
23 
Learning to Rank 
 Approaches: 
 Point-wise: Loss over items 
(Classification, ordinal regression, MF, …) 
 Pair-wise: Loss over preferences 
(RankSVM, RankNet, BPR, …) 
 List-wise: (Smoothed) loss over ranking 
(LambdaMART, DirectRank, GAPfm, …) 
 Ranking quality measures: 
 Recall@N, Precision@N, NDCG, MRR, 
ERR, MAP, FCP, … 
1 
0.9 
0.8 
0.7 
0.6 
0.5 
0.4 
0.3 
0.2 
0.1 
0 
0 20 40 60 80 100 
Importance 
Rank 
NDCG MRR FCP
24 
Example: Two features, linear model 
Popularity 
Predicted Rating 
1 
2 
3 
4 
5 
Linear Model: 
Final Ranking 
frank(u,v) = w1 p(v) + w2 r(u,v)
25 
Personalized Ranking
26 
“Learning to Row” 
Personalized Page Generation
27 
Page-level algorithmic challenge 
10,000s of 
possible 
rows … 
10-40 
rows 
Variable number of 
possible videos per 
row (up to thousands) 
1 personalized page 
per device
28 
Balancing a Personalized Page 
Accurate vs. Diverse 
Discovery vs. Continuation 
Depth vs. Coverage 
Freshness vs. Stability 
Recommendations vs. Tasks
29 
Browsing Modeling 
More likely 
to see 
Less likely
30 
Row Lifecycle 
Select 
Candidates 
Filter 
Rank 
Deduplicate 
Choose 
Format
31 
Building a page algorithmically 
Approaches 
1. Rule-based 
2. Row-independent 
3. Stage-wise 
4. Page-wise
32 
Rule-based Approach 
 Create rules for location of rows 
 Non-personalized layout 
 Optimize by AB testing 
 Example: 
 First: Continue Watching (if any) 
 Next: Top picks 
 Next: Popular on Netflix 
 Next 3: Because you watched (if 
any) 
 Next 3: Genres 
 Next: Recently added 
 … 
 Pros: 
 Backed up by AB testing 
 Easy to compute 
 Very familiar page layout 
 Cons: 
 One size fits nobody 
 Local optima 
 Hard to add new kinds of rows 
 Rules get complicated 
 Still need to pick best rows within 
groupings
33 
Row-independent Approach 
 Treat row selection as a standard 
ranking problem 
 Greedily rank rows by f(r | u, d) 
 Sort and take top n 
 Approaches: 
 Top-n recommendation 
algorithms 
 Pros: 
 Available algorithms 
 Fast 
 High accuracy 
 Cons: 
 No consideration of diversity 
 People watch videos, row 
interaction is indirect
34 
Stage-wise Approach 
 Use information about already 
picked rows 
 Pick next rows by f(r | u, d, p1:n) 
 Approaches: 
 Train scoring model 
 Greedy 
 K-step look-ahead 
 Pros: 
 Can balance accuracy and 
diversity 
 Follows navigation pattern 
 Cons: 
 May need to evaluate each row 
multiple times from deduping 
 Still greedy 
 Hard to get training data
35 
Page-wise approach 
 Optimize the whole page 
 Total page fitness f(p | u, d) 
 Approaches: 
 Constrained optimization 
 Approximate integer 
programming 
 Pros: 
 Includes information about entire 
page 
 Can include diversity and 
deduping into objective 
 Able to synthesize rows 
 Cons: 
 Computationally expensive 
 Huge space of possible pages
36 
Obey constraints 
 Device 
 Screen size 
 Processing limitations 
 Certain rows may be required 
 Continue Watching 
 My List 
 …
37 
Row Features 
 Quality of items 
 Features of items 
 Quality of evidence 
 User-row interactions 
 Item/row metadata 
 Recency 
 Item-row affinity 
 Row length 
 Position on page 
 Title 
 Diversity 
 Similarity 
 Freshness 
 …
38 
Page-Level Metrics 
 How do you measure the quality of the homepage? 
 Ease of discovery 
 Diversity 
 Novelty 
 … 
 Challenges: 
 Position effects 
 Row-video generalization 
 Approach: 2D versions of ranking quality metrics
39 
Simple example: Recall @ row-by-column 
Page Variant 1 
Drama 
Action 
Comedy 
Page Variant 2 
Sitcoms 
Drama 
Top Picks 
Sci-Fi 
Comedy 
Page Variant 3 
✔ 
✔ 
✔ ✔ 
✔ 
Sitcoms 
User Watched 
1/3 
1/3 
1/3 
0/3 
1/3 
2/3 
2/3 
3/3 
✔ 3/3 ✔
40 
Offline Results: Recall @ row-by-column 
Existing 
Template 
0 5 10 15 20 25 30 35 
Recall 
Row 
Best 
personalized 
layout
41 
Conclusions
42 
Evolution of Recommendation Approach 
4.7 
Rating Ranking Page Generation
43 
Research Directions 
Presentation 
effects 
Full-page 
optimization 
Context 
Awareness 
Scaling 
Personalized 
learning to 
rank 
Cold start
Thank You Justin Basilico 
jbasilico@netflix.com 
44 @JustinBasilico 
We’re hiring

More Related Content

What's hot

Lessons Learned from Building Machine Learning Software at Netflix
Lessons Learned from Building Machine Learning Software at NetflixLessons Learned from Building Machine Learning Software at Netflix
Lessons Learned from Building Machine Learning Software at NetflixJustin Basilico
 
Deep Learning for Recommender Systems
Deep Learning for Recommender SystemsDeep Learning for Recommender Systems
Deep Learning for Recommender SystemsJustin Basilico
 
Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
 Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se... Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...Sudeep Das, Ph.D.
 
Context Aware Recommendations at Netflix
Context Aware Recommendations at NetflixContext Aware Recommendations at Netflix
Context Aware Recommendations at NetflixLinas Baltrunas
 
Shallow and Deep Latent Models for Recommender System
Shallow and Deep Latent Models for Recommender SystemShallow and Deep Latent Models for Recommender System
Shallow and Deep Latent Models for Recommender SystemAnoop Deoras
 
A Multi-Armed Bandit Framework For Recommendations at Netflix
A Multi-Armed Bandit Framework For Recommendations at NetflixA Multi-Armed Bandit Framework For Recommendations at Netflix
A Multi-Armed Bandit Framework For Recommendations at NetflixJaya Kawale
 
Artwork Personalization at Netflix
Artwork Personalization at NetflixArtwork Personalization at Netflix
Artwork Personalization at NetflixJustin Basilico
 
Missing values in recommender models
Missing values in recommender modelsMissing values in recommender models
Missing values in recommender modelsParmeshwar Khurd
 
Netflix talk at ML Platform meetup Sep 2019
Netflix talk at ML Platform meetup Sep 2019Netflix talk at ML Platform meetup Sep 2019
Netflix talk at ML Platform meetup Sep 2019Faisal Siddiqi
 
Personalizing "The Netflix Experience" with Deep Learning
Personalizing "The Netflix Experience" with Deep LearningPersonalizing "The Netflix Experience" with Deep Learning
Personalizing "The Netflix Experience" with Deep LearningAnoop Deoras
 
Recommendations for Building Machine Learning Software
Recommendations for Building Machine Learning SoftwareRecommendations for Building Machine Learning Software
Recommendations for Building Machine Learning SoftwareJustin Basilico
 
Past, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspectivePast, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspectiveXavier Amatriain
 
Calibrated Recommendations
Calibrated RecommendationsCalibrated Recommendations
Calibrated RecommendationsHarald Steck
 
Netflix Recommendations Feature Engineering with Time Travel
Netflix Recommendations Feature Engineering with Time TravelNetflix Recommendations Feature Engineering with Time Travel
Netflix Recommendations Feature Engineering with Time TravelFaisal Siddiqi
 
Recent Trends in Personalization at Netflix
Recent Trends in Personalization at NetflixRecent Trends in Personalization at Netflix
Recent Trends in Personalization at NetflixJustin Basilico
 
Interactive Recommender Systems with Netflix and Spotify
Interactive Recommender Systems with Netflix and SpotifyInteractive Recommender Systems with Netflix and Spotify
Interactive Recommender Systems with Netflix and SpotifyChris Johnson
 
Data council SF 2020 Building a Personalized Messaging System at Netflix
Data council SF 2020 Building a Personalized Messaging System at NetflixData council SF 2020 Building a Personalized Messaging System at Netflix
Data council SF 2020 Building a Personalized Messaging System at NetflixGrace T. Huang
 
Crafting Recommenders: the Shallow and the Deep of it!
Crafting Recommenders: the Shallow and the Deep of it! Crafting Recommenders: the Shallow and the Deep of it!
Crafting Recommenders: the Shallow and the Deep of it! Sudeep Das, Ph.D.
 
Recommending for the World
Recommending for the WorldRecommending for the World
Recommending for the WorldYves Raimond
 

What's hot (20)

Lessons Learned from Building Machine Learning Software at Netflix
Lessons Learned from Building Machine Learning Software at NetflixLessons Learned from Building Machine Learning Software at Netflix
Lessons Learned from Building Machine Learning Software at Netflix
 
Deep Learning for Recommender Systems
Deep Learning for Recommender SystemsDeep Learning for Recommender Systems
Deep Learning for Recommender Systems
 
Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
 Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se... Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
 
Context Aware Recommendations at Netflix
Context Aware Recommendations at NetflixContext Aware Recommendations at Netflix
Context Aware Recommendations at Netflix
 
Shallow and Deep Latent Models for Recommender System
Shallow and Deep Latent Models for Recommender SystemShallow and Deep Latent Models for Recommender System
Shallow and Deep Latent Models for Recommender System
 
A Multi-Armed Bandit Framework For Recommendations at Netflix
A Multi-Armed Bandit Framework For Recommendations at NetflixA Multi-Armed Bandit Framework For Recommendations at Netflix
A Multi-Armed Bandit Framework For Recommendations at Netflix
 
Artwork Personalization at Netflix
Artwork Personalization at NetflixArtwork Personalization at Netflix
Artwork Personalization at Netflix
 
Recent Trends in Personalization at Netflix
Recent Trends in Personalization at NetflixRecent Trends in Personalization at Netflix
Recent Trends in Personalization at Netflix
 
Missing values in recommender models
Missing values in recommender modelsMissing values in recommender models
Missing values in recommender models
 
Netflix talk at ML Platform meetup Sep 2019
Netflix talk at ML Platform meetup Sep 2019Netflix talk at ML Platform meetup Sep 2019
Netflix talk at ML Platform meetup Sep 2019
 
Personalizing "The Netflix Experience" with Deep Learning
Personalizing "The Netflix Experience" with Deep LearningPersonalizing "The Netflix Experience" with Deep Learning
Personalizing "The Netflix Experience" with Deep Learning
 
Recommendations for Building Machine Learning Software
Recommendations for Building Machine Learning SoftwareRecommendations for Building Machine Learning Software
Recommendations for Building Machine Learning Software
 
Past, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspectivePast, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspective
 
Calibrated Recommendations
Calibrated RecommendationsCalibrated Recommendations
Calibrated Recommendations
 
Netflix Recommendations Feature Engineering with Time Travel
Netflix Recommendations Feature Engineering with Time TravelNetflix Recommendations Feature Engineering with Time Travel
Netflix Recommendations Feature Engineering with Time Travel
 
Recent Trends in Personalization at Netflix
Recent Trends in Personalization at NetflixRecent Trends in Personalization at Netflix
Recent Trends in Personalization at Netflix
 
Interactive Recommender Systems with Netflix and Spotify
Interactive Recommender Systems with Netflix and SpotifyInteractive Recommender Systems with Netflix and Spotify
Interactive Recommender Systems with Netflix and Spotify
 
Data council SF 2020 Building a Personalized Messaging System at Netflix
Data council SF 2020 Building a Personalized Messaging System at NetflixData council SF 2020 Building a Personalized Messaging System at Netflix
Data council SF 2020 Building a Personalized Messaging System at Netflix
 
Crafting Recommenders: the Shallow and the Deep of it!
Crafting Recommenders: the Shallow and the Deep of it! Crafting Recommenders: the Shallow and the Deep of it!
Crafting Recommenders: the Shallow and the Deep of it!
 
Recommending for the World
Recommending for the WorldRecommending for the World
Recommending for the World
 

Similar to Personalized Page Generation for Browsing Recommendations

MLconf NYC Justin Basilico
MLconf NYC Justin BasilicoMLconf NYC Justin Basilico
MLconf NYC Justin BasilicoMLconf
 
acmsigtalkshare-121023190142-phpapp01.pptx
acmsigtalkshare-121023190142-phpapp01.pptxacmsigtalkshare-121023190142-phpapp01.pptx
acmsigtalkshare-121023190142-phpapp01.pptxdongchangim30
 
Recommendation at Netflix Scale
Recommendation at Netflix ScaleRecommendation at Netflix Scale
Recommendation at Netflix ScaleJustin Basilico
 
Rokach-GomaxSlides (1).pptx
Rokach-GomaxSlides (1).pptxRokach-GomaxSlides (1).pptx
Rokach-GomaxSlides (1).pptxJadna Almeida
 
Rokach-GomaxSlides.pptx
Rokach-GomaxSlides.pptxRokach-GomaxSlides.pptx
Rokach-GomaxSlides.pptxJadna Almeida
 
Big & Personal: the data and the models behind Netflix recommendations by Xa...
 Big & Personal: the data and the models behind Netflix recommendations by Xa... Big & Personal: the data and the models behind Netflix recommendations by Xa...
Big & Personal: the data and the models behind Netflix recommendations by Xa...BigMine
 
Qcon SF 2013 - Machine Learning & Recommender Systems @ Netflix Scale
Qcon SF 2013 - Machine Learning & Recommender Systems @ Netflix ScaleQcon SF 2013 - Machine Learning & Recommender Systems @ Netflix Scale
Qcon SF 2013 - Machine Learning & Recommender Systems @ Netflix ScaleXavier Amatriain
 
Cikm 2013 - Beyond Data From User Information to Business Value
Cikm 2013 - Beyond Data From User Information to Business ValueCikm 2013 - Beyond Data From User Information to Business Value
Cikm 2013 - Beyond Data From User Information to Business ValueXavier Amatriain
 
MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix
MLConf - Emmys, Oscars & Machine Learning Algorithms at NetflixMLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix
MLConf - Emmys, Oscars & Machine Learning Algorithms at NetflixXavier Amatriain
 
Xavier amatriain, dir algorithms netflix m lconf 2013
Xavier amatriain, dir algorithms netflix m lconf 2013Xavier amatriain, dir algorithms netflix m lconf 2013
Xavier amatriain, dir algorithms netflix m lconf 2013MLconf
 
Telecom datascience master_public
Telecom datascience master_publicTelecom datascience master_public
Telecom datascience master_publicVincent Michel
 
Justin Basilico, Research/ Engineering Manager at Netflix at MLconf SF - 11/1...
Justin Basilico, Research/ Engineering Manager at Netflix at MLconf SF - 11/1...Justin Basilico, Research/ Engineering Manager at Netflix at MLconf SF - 11/1...
Justin Basilico, Research/ Engineering Manager at Netflix at MLconf SF - 11/1...MLconf
 
Recommendations for Building Machine Learning Software
Recommendations for Building Machine Learning SoftwareRecommendations for Building Machine Learning Software
Recommendations for Building Machine Learning SoftwareJustin Basilico
 
Marketing Analytics with R Lifting Campaign Success Rates
Marketing Analytics with R Lifting Campaign Success RatesMarketing Analytics with R Lifting Campaign Success Rates
Marketing Analytics with R Lifting Campaign Success RatesRevolution Analytics
 
An introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolboxAn introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolboxElasticsearch
 
Recsys2016 Tutorial by Xavier and Deepak
Recsys2016 Tutorial by Xavier and DeepakRecsys2016 Tutorial by Xavier and Deepak
Recsys2016 Tutorial by Xavier and DeepakDeepak Agarwal
 
Deep Learning for Recommender Systems @ TDC SP 2019
Deep Learning for Recommender Systems @ TDC SP 2019Deep Learning for Recommender Systems @ TDC SP 2019
Deep Learning for Recommender Systems @ TDC SP 2019Gabriel Moreira
 
UX Days 2019 by Flupa - Conférence : Nicolas Duval et Dawid Woldu
UX Days 2019 by Flupa - Conférence : Nicolas Duval et Dawid WolduUX Days 2019 by Flupa - Conférence : Nicolas Duval et Dawid Woldu
UX Days 2019 by Flupa - Conférence : Nicolas Duval et Dawid WolduFlupa
 
R Tool for Visual Studio และการทำงานร่วมกันเป็นทีม โดย เฉลิมวงศ์ วิจิตรปิยะกุ...
R Tool for Visual Studio และการทำงานร่วมกันเป็นทีม โดย เฉลิมวงศ์ วิจิตรปิยะกุ...R Tool for Visual Studio และการทำงานร่วมกันเป็นทีม โดย เฉลิมวงศ์ วิจิตรปิยะกุ...
R Tool for Visual Studio และการทำงานร่วมกันเป็นทีม โดย เฉลิมวงศ์ วิจิตรปิยะกุ...BAINIDA
 

Similar to Personalized Page Generation for Browsing Recommendations (20)

Learning to Personalize
Learning to PersonalizeLearning to Personalize
Learning to Personalize
 
MLconf NYC Justin Basilico
MLconf NYC Justin BasilicoMLconf NYC Justin Basilico
MLconf NYC Justin Basilico
 
acmsigtalkshare-121023190142-phpapp01.pptx
acmsigtalkshare-121023190142-phpapp01.pptxacmsigtalkshare-121023190142-phpapp01.pptx
acmsigtalkshare-121023190142-phpapp01.pptx
 
Recommendation at Netflix Scale
Recommendation at Netflix ScaleRecommendation at Netflix Scale
Recommendation at Netflix Scale
 
Rokach-GomaxSlides (1).pptx
Rokach-GomaxSlides (1).pptxRokach-GomaxSlides (1).pptx
Rokach-GomaxSlides (1).pptx
 
Rokach-GomaxSlides.pptx
Rokach-GomaxSlides.pptxRokach-GomaxSlides.pptx
Rokach-GomaxSlides.pptx
 
Big & Personal: the data and the models behind Netflix recommendations by Xa...
 Big & Personal: the data and the models behind Netflix recommendations by Xa... Big & Personal: the data and the models behind Netflix recommendations by Xa...
Big & Personal: the data and the models behind Netflix recommendations by Xa...
 
Qcon SF 2013 - Machine Learning & Recommender Systems @ Netflix Scale
Qcon SF 2013 - Machine Learning & Recommender Systems @ Netflix ScaleQcon SF 2013 - Machine Learning & Recommender Systems @ Netflix Scale
Qcon SF 2013 - Machine Learning & Recommender Systems @ Netflix Scale
 
Cikm 2013 - Beyond Data From User Information to Business Value
Cikm 2013 - Beyond Data From User Information to Business ValueCikm 2013 - Beyond Data From User Information to Business Value
Cikm 2013 - Beyond Data From User Information to Business Value
 
MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix
MLConf - Emmys, Oscars & Machine Learning Algorithms at NetflixMLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix
MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix
 
Xavier amatriain, dir algorithms netflix m lconf 2013
Xavier amatriain, dir algorithms netflix m lconf 2013Xavier amatriain, dir algorithms netflix m lconf 2013
Xavier amatriain, dir algorithms netflix m lconf 2013
 
Telecom datascience master_public
Telecom datascience master_publicTelecom datascience master_public
Telecom datascience master_public
 
Justin Basilico, Research/ Engineering Manager at Netflix at MLconf SF - 11/1...
Justin Basilico, Research/ Engineering Manager at Netflix at MLconf SF - 11/1...Justin Basilico, Research/ Engineering Manager at Netflix at MLconf SF - 11/1...
Justin Basilico, Research/ Engineering Manager at Netflix at MLconf SF - 11/1...
 
Recommendations for Building Machine Learning Software
Recommendations for Building Machine Learning SoftwareRecommendations for Building Machine Learning Software
Recommendations for Building Machine Learning Software
 
Marketing Analytics with R Lifting Campaign Success Rates
Marketing Analytics with R Lifting Campaign Success RatesMarketing Analytics with R Lifting Campaign Success Rates
Marketing Analytics with R Lifting Campaign Success Rates
 
An introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolboxAn introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolbox
 
Recsys2016 Tutorial by Xavier and Deepak
Recsys2016 Tutorial by Xavier and DeepakRecsys2016 Tutorial by Xavier and Deepak
Recsys2016 Tutorial by Xavier and Deepak
 
Deep Learning for Recommender Systems @ TDC SP 2019
Deep Learning for Recommender Systems @ TDC SP 2019Deep Learning for Recommender Systems @ TDC SP 2019
Deep Learning for Recommender Systems @ TDC SP 2019
 
UX Days 2019 by Flupa - Conférence : Nicolas Duval et Dawid Woldu
UX Days 2019 by Flupa - Conférence : Nicolas Duval et Dawid WolduUX Days 2019 by Flupa - Conférence : Nicolas Duval et Dawid Woldu
UX Days 2019 by Flupa - Conférence : Nicolas Duval et Dawid Woldu
 
R Tool for Visual Studio และการทำงานร่วมกันเป็นทีม โดย เฉลิมวงศ์ วิจิตรปิยะกุ...
R Tool for Visual Studio และการทำงานร่วมกันเป็นทีม โดย เฉลิมวงศ์ วิจิตรปิยะกุ...R Tool for Visual Studio และการทำงานร่วมกันเป็นทีม โดย เฉลิมวงศ์ วิจิตรปิยะกุ...
R Tool for Visual Studio และการทำงานร่วมกันเป็นทีม โดย เฉลิมวงศ์ วิจิตรปิยะกุ...
 

Recently uploaded

Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?Juan Carlos Gonzalez
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
IEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
IEEE Computer Society’s Strategic Activities and Products including SWEBOK GuideIEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
IEEE Computer Society’s Strategic Activities and Products including SWEBOK GuideHironori Washizaki
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...Daniel Zivkovic
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 

Recently uploaded (20)

Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
IEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
IEEE Computer Society’s Strategic Activities and Products including SWEBOK GuideIEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
IEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 

Personalized Page Generation for Browsing Recommendations

  • 1. 1 Personalized Page Generation for Browsing Recommendations Justin Basilico Page Algorithms Engineering October 10, 2014 @JustinBasilico Workshop 2014
  • 2. 2 ?
  • 3. 3 Evolution of our Recommendation Approach 4.7 Rating Ranking Page Generation
  • 4. 4  Interested in high-quality recommendations  Proxy question:  Accuracy in predicted rating  Measured by root mean squared error (RMSE)  Improve by 10% = $1 million!  Data size:  100M ratings (back then “almost massive”)
  • 5. 5 Change of focus 2006 2014
  • 6. 6 Netflix Scale  > 50M members  > 40 countries  > 1000 device types  Hours: > 2B/month  Plays: > 30M/day  Log 100B events/day  34.2% of peak US downstream traffic
  • 7. 7 “Emmy Winning” Approach to Recommendation
  • 8. 8 Goal Help members find content to watch and enjoy to maximize member satisfaction and retention
  • 9. 9 Everything is a Recommendation Rows Ranking Over 75% of what people watch comes from our recommendations Recommendations are driven by Machine Learning
  • 10. 10 Top Picks Personalization awareness Diversity
  • 11. 11 Personalized genres  Genres focused on user interest  Derived from tag combinations  Provide context and evidence  How are they generated?  Implicit: Based on recent plays, ratings & other interactions  Explicit: Taste preferences
  • 12. 12 Similarity  Recommend videos similar to one you’ve liked  “Because you watched” rows  Pivots  Video information page  In response to user actions (search, list add, …)
  • 13. 13 Support for Recommendations Behavioral Support Social Support
  • 14. 14 Learning to Recommend
  • 15. 15 Machine Learning Approach Problem Data Metrics Model Algorithm
  • 16. 16 Data  Plays  Duration, bookmark, time, device, …  Ratings  Metadata  Tags, synopsis, cast, …  Impressions  Interactions  Search, list add, scroll, …  Social
  • 17. 17 Models & Algorithms  Regression (Linear, logistic, elastic net)  SVD and other Matrix Factorizations  Factorization Machines  Restricted Boltzmann Machines  Deep Neural Networks  Markov Models and Graph Algorithms  Clustering  Latent Dirichlet Allocation  Gradient Boosted Decision Trees/Random Forests  Gaussian Processes  …
  • 19. 19 Rating Prediction  Based on first year progress prize  Top 2 algorithms  Matrix Factorization (SVD++)  Restricted Boltzmann Machines (RBM)  Ensemble: Linear blend Videos R ≈ Users U V (99% Sparse) d Videos Users × d
  • 20. 20 Ranking by ratings 4.7 4.6 4.5 4.5 4.5 4.5 4.5 4.5 4.5 4.5 Niche titles High average ratings… by those who would watch it
  • 23. 23 Learning to Rank  Approaches:  Point-wise: Loss over items (Classification, ordinal regression, MF, …)  Pair-wise: Loss over preferences (RankSVM, RankNet, BPR, …)  List-wise: (Smoothed) loss over ranking (LambdaMART, DirectRank, GAPfm, …)  Ranking quality measures:  Recall@N, Precision@N, NDCG, MRR, ERR, MAP, FCP, … 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 20 40 60 80 100 Importance Rank NDCG MRR FCP
  • 24. 24 Example: Two features, linear model Popularity Predicted Rating 1 2 3 4 5 Linear Model: Final Ranking frank(u,v) = w1 p(v) + w2 r(u,v)
  • 26. 26 “Learning to Row” Personalized Page Generation
  • 27. 27 Page-level algorithmic challenge 10,000s of possible rows … 10-40 rows Variable number of possible videos per row (up to thousands) 1 personalized page per device
  • 28. 28 Balancing a Personalized Page Accurate vs. Diverse Discovery vs. Continuation Depth vs. Coverage Freshness vs. Stability Recommendations vs. Tasks
  • 29. 29 Browsing Modeling More likely to see Less likely
  • 30. 30 Row Lifecycle Select Candidates Filter Rank Deduplicate Choose Format
  • 31. 31 Building a page algorithmically Approaches 1. Rule-based 2. Row-independent 3. Stage-wise 4. Page-wise
  • 32. 32 Rule-based Approach  Create rules for location of rows  Non-personalized layout  Optimize by AB testing  Example:  First: Continue Watching (if any)  Next: Top picks  Next: Popular on Netflix  Next 3: Because you watched (if any)  Next 3: Genres  Next: Recently added  …  Pros:  Backed up by AB testing  Easy to compute  Very familiar page layout  Cons:  One size fits nobody  Local optima  Hard to add new kinds of rows  Rules get complicated  Still need to pick best rows within groupings
  • 33. 33 Row-independent Approach  Treat row selection as a standard ranking problem  Greedily rank rows by f(r | u, d)  Sort and take top n  Approaches:  Top-n recommendation algorithms  Pros:  Available algorithms  Fast  High accuracy  Cons:  No consideration of diversity  People watch videos, row interaction is indirect
  • 34. 34 Stage-wise Approach  Use information about already picked rows  Pick next rows by f(r | u, d, p1:n)  Approaches:  Train scoring model  Greedy  K-step look-ahead  Pros:  Can balance accuracy and diversity  Follows navigation pattern  Cons:  May need to evaluate each row multiple times from deduping  Still greedy  Hard to get training data
  • 35. 35 Page-wise approach  Optimize the whole page  Total page fitness f(p | u, d)  Approaches:  Constrained optimization  Approximate integer programming  Pros:  Includes information about entire page  Can include diversity and deduping into objective  Able to synthesize rows  Cons:  Computationally expensive  Huge space of possible pages
  • 36. 36 Obey constraints  Device  Screen size  Processing limitations  Certain rows may be required  Continue Watching  My List  …
  • 37. 37 Row Features  Quality of items  Features of items  Quality of evidence  User-row interactions  Item/row metadata  Recency  Item-row affinity  Row length  Position on page  Title  Diversity  Similarity  Freshness  …
  • 38. 38 Page-Level Metrics  How do you measure the quality of the homepage?  Ease of discovery  Diversity  Novelty  …  Challenges:  Position effects  Row-video generalization  Approach: 2D versions of ranking quality metrics
  • 39. 39 Simple example: Recall @ row-by-column Page Variant 1 Drama Action Comedy Page Variant 2 Sitcoms Drama Top Picks Sci-Fi Comedy Page Variant 3 ✔ ✔ ✔ ✔ ✔ Sitcoms User Watched 1/3 1/3 1/3 0/3 1/3 2/3 2/3 3/3 ✔ 3/3 ✔
  • 40. 40 Offline Results: Recall @ row-by-column Existing Template 0 5 10 15 20 25 30 35 Recall Row Best personalized layout
  • 42. 42 Evolution of Recommendation Approach 4.7 Rating Ranking Page Generation
  • 43. 43 Research Directions Presentation effects Full-page optimization Context Awareness Scaling Personalized learning to rank Cold start
  • 44. Thank You Justin Basilico jbasilico@netflix.com 44 @JustinBasilico We’re hiring