Shared at San Francisco Big Analytics Meetup (2021-07-13)

1. System Design for
Recommendations & Search
Eugene Yan, Applied Scientist @ Amazon
San Francisco Big Analytics Meetup (2021-07-13)

2. About me
● Applied Scientist @ Amazon
○ Machine Learning for Recommendations & Search
● Previously: VP, Data Science @ Lazada
○ Led the team on e-commerce ML systems
○ Facilitated integration with Alibaba (acquired 2016)
● Previously: Data Science Lead @ Healthtech startup
○ Early detection of preventable diseases
○ Healthcare resource allocation
● More at eugeneyan.com

3. Topics
● Context: Batch vs. Real-time (i.e., on-demand)
● System design
○ Offline vs. Online environments
○ Candidate retrieval vs. Ranking
● Industry Examples
○ Alibaba, FaceBook, DoorDash
● How to build an MVP
○ If there’s time...

4. Batch vs. Real-time

5. Source: https:/
/softwareengineeringdaily.com/2018/11/09/converged-data-platform-unifying-streaming-data-using-mapr/

6. Recommender
(REST/gRPC)
Recommender
(key-value DB)
Recommendations
refreshed
periodically
Recommendations
generated
on-demand

7. Recommender
(REST/gRPC)
Recommender
(key-value DB)
Recommendations
refreshed
periodically
Recommendations
generated in
real-time

8. Recommender
(REST/gRPC)
Recommender
(key-value DB)
Recommendations
refreshed
periodically
Recommendations
generated in
real-time
● Pre-computed
● Decouple compute from serving
● Lower operational load

9. Recommender
(REST/gRPC)
Recommender
(key-value DB)
Recommendations
refreshed
periodically
Recommendations
generated in
real-time
● Pre-computed
● Decouple compute from serving
● Lower operational load
● Responsive to
time-sensitive context
● Reduced cost on
non-visiting users

10. Recommender
(REST/gRPC)
Recommender
(key-value DB)
Recommendations
refreshed
periodically
Recommendations
generated in
real-time
● Pre-computed
● Decouple compute from serving
● Lower operational load
We will focus on
real-time aka
on-demand
● Responsive to
time-sensitive context
● Reduced cost on
non-visiting users

11. System Design

12. Online
Offline

13. Online
Offline
● Host batch processes such as
training, index/graph building
● Load data into feature stores

14. Online
Offline
● Host batch processes such as
training, index/graph building
● Load data into feature stores
● Uses artifacts from offline
environment to serve requests
● Candidate retrieval and ranking

15. Retrieval Ranking

16. Retrieval Ranking
● Fast but coarse
● Searches millions of items to
get hundreds of candidates
● Approx NN, Graphs, etc.

17. Retrieval Ranking
● Fast but coarse
● Searches millions of items to
get hundreds of candidates
● Approx NN, Graphs, etc.
● Slower but more precise
● Ranks hundreds of candidates
● Adds more features
● Classification or learning-to-rank

18. Online
Offline
Retrieval Ranking

19. Online
Offline
Retrieval Ranking
Train embedding
model
Train ranking
model
Embed items
from catalog
Build Approx.
NN index
Embed input
item or query
Retrieve top k
candidates
Build feature store
(e.g., item, user)
Add features to
candidates
Rank top k
candidates

20. Online
Offline
Retrieval Ranking
Train embedding
model
Training data
Embed items
from catalog
Build Approx.
NN index
Item data

21. Online
Offline
Retrieval Ranking
Train embedding
model
Training data Training data
Train ranking
model
Embed items
from catalog
Build Approx.
NN index
Item data
Build feature store
(e.g., item, user)
Item & user data

22. Online
Offline
Retrieval Ranking
Train embedding
model
Training data Training data
Train ranking
model
Embed items
from catalog
Build Approx.
NN index
Item data
Embedding
model
ANN
Index
Build feature store
(e.g., item, user)
Item & user data
Feature
Store
Ranking
model

23. Online
Offline
Retrieval Ranking
Embed input
item or query
Retrieve top k
candidates
Request
Embedding
model
ANN
Index

24. Online
Offline
Retrieval Ranking
Embed input
item or query
Retrieve top k
candidates
Request
Embedding
model
ANN
Index
Add features to
candidates
Rank top k
candidates
Feature
Store
Ranking
model
Results

25. Online
Offline
Retrieval Ranking
Train embedding
model
Training data Training data
Train ranking
model
Embed items
from catalog
Build Approx.
NN index
Item data
Embed input
item or query
Retrieve top k
candidates
Request
Embedding
model
ANN
Index
Build feature store
(e.g., item, user)
Item & user data
Add features to
candidates
Rank top k
candidates
Feature
Store
Ranking
model
Results

26. Industry Examples

27. Building item embeddings for candidate retrieval (Alibaba)
Source: https:/
/arxiv.org/abs/1803.02349

28. Source: https:/
/arxiv.org/abs/1803.02349
Building item embeddings for candidate retrieval (Alibaba)

29. Source: https:/
/arxiv.org/abs/1803.02349
Ranker
Train
embedding
Candidate
Retrieval
Building item embeddings for candidate retrieval (Alibaba)

30. Building a graph network for ranking (Alibaba)
Source: https:/
/arxiv.org/abs/2005.12002

31. Building a graph network for ranking (Alibaba)
Source: https:/
/arxiv.org/abs/2005.12002

32. Building a graph network for ranking (Alibaba)
Source: https:/
/arxiv.org/abs/2005.12002

33. Building a graph network for ranking (Alibaba)
Source: https:/
/arxiv.org/abs/2005.12002
Ranker
Candidate
Retrieval
Train
Ranker
Feature
Stores
Graph

34. Building embeddings for retrieval in search (Facebook)
Source: https:/
/arxiv.org/abs/2006.11632

35. Building embeddings for retrieval in search (Facebook)
Source: https:/
/arxiv.org/abs/2006.11632
Train
Embedding
model

36. Building embeddings for retrieval in search (Facebook)
Source: https:/
/arxiv.org/abs/2006.11632
Train
Embedding
model

37. Building embeddings for retrieval in search (Facebook)
Source: https:/
/arxiv.org/abs/2006.11632
Train
Embedding
model

38. Building embeddings for retrieval in search (Facebook)
Source: https:/
/arxiv.org/abs/2006.11632
Train
Embedding
model

39. Building embeddings for retrieval in search (Facebook)
Source: https:/
/arxiv.org/abs/2006.11632
Train
Embedding
model
Offline
Online

40. Building embeddings for retrieval in search (Facebook)
Source: https:/
/arxiv.org/abs/2006.11632
Train
Embedding
model
Retrieval
Ranker
Feature
Store
Offline
Online

41. Building graphs for query expansion & retrieval (DoorDash)
Source: https:/
/doordash.engineering/2020/12/15/understanding-search-intent-with-better-recall/

42. Building graphs for query expansion & retrieval (DoorDash)
Source: https:/
/doordash.engineering/2020/12/15/understanding-search-intent-with-better-recall/

43. Building graphs for query expansion & retrieval (DoorDash)
Source: https:/
/doordash.engineering/2020/12/15/understanding-search-intent-with-better-recall/

44. Building graphs for query expansion & retrieval (DoorDash)
Source: https:/
/doordash.engineering/2020/12/15/understanding-search-intent-with-better-recall/

45. Building graphs for query expansion & retrieval (DoorDash)
Source: https:/
/doordash.engineering/2020/12/15/understanding-search-intent-with-better-recall/

46. Building graphs for query expansion & retrieval (DoorDash)
Source: https:/
/doordash.engineering/2020/12/15/understanding-search-intent-with-better-recall/
Knowledge
Graph
Retrieval
Ranker

47. How to build an MVP

48. Source: https:/
/ai.facebook.com/blog/powered-by-ai-instagrams-explore-recommender-system/
Instagram’s ig2vec
Training: Self-supervised Representation Learning

49. Source: https:/
/eng.uber.com/uber-eats-query-understanding/
Uber’s Query2Vec
Training: Self-supervised Representation Learning

50. Training: Self-supervised Representation Learning
Source: https:/
/arxiv.org/abs/1803.02349
Alibaba’s item2vec

51. Retrieval: Approximate nearest neighbors
Source: https:/
/eugeneyan.com/writing/real-time-recommendations/

52. Ranking: Logistic regression
Source: https:/
/eugeneyan.com/writing/real-time-recommendations/
• • •
User
features
Item
features
Probability of
interaction

53. Serving: Multiple instances + Load Balancer (or SageMaker)
Source: https:/
/eugeneyan.com/writing/real-time-recommendations/

54. Fin
eugeneyan • eugeneyan.com