More Related Content
Similar to Modeling Impression discounting in large-scale recommender systems
Modeling Impression discounting in large-scale recommender systems
- 1. Modeling Impression DiscountingIn Large-scale Recommender Systems KDD 2014, presented by Mitul Tiwari 1 Pei Lee, Laks V.S. Lakshmanan University of British Columbia Vancouver, BC, Canada Mitul Tiwari, Sam Shah LinkedIn Mountain View, CA, USA 2015/3/12
- 2. 2 LinkedIn By TheNumbers 300M members 2 new members/sec
- 3. Large-scale Recommender Systems atLinkedIn3 People You May Know Skills Endorsements
- 4. Recommender Systems atLinkedIn 4
- 5. What is ImpressionDiscounting? 5 Examples People You May Know Skills Endorsement Problem Definition
- 6. 5 days ago: Recommended,but not invited Today: If we recommend again, would you invite or not? If you are not likely to invite, we better discount this recommendation. 1 day ago: Recommended again, but not invited Example: People You May Know
- 7. Example: Skills Endorsement If you are likely to endorse, we will show a skills suggestion again; Otherwise, we should leave this space for other skills suggestions Recommended Recommended Again
- 8. Problem Definition Impressions:recommendations shown to user Conversion: positive action - invite, endorse, etc. In natural language: We already impressed an item several times with no conversion. How much should we discount that item? More formally: For an user u and item i, given an impression history (T1, T2 , …, Tn) between u and i, can we predict the conversion rate for u on i?
- 9.
- 10. Outline 10 Define animpression User, Item, Conversion Rate Features How many times the user saw this item? When is the last time the user saw this item? … Data analysis: impression features and conversion relationship Impression discounting model fitting Experimental evaluation
- 11.
- 12. Features on ImpressedItems 12 ImpCount (frequency): the number of historical impressions before the current impression, associated with the same (user, item) LastSeen (recency): the day difference between the last impression and the current impression, associated with the same (user, item) Position: the offset of item in the recommendation list of user UserFreq: the interaction frequency of user in a recommender system
- 13. Data Analysis: PYMKDataset Data: 1.09 B impression tuples Training dataset: 80% of data 0.55 billion unique impressions 0.87 billion total impressions 20 millions invitations Testing dataset: 20% of data 0.14 billion unique impressions (3.7%) 0.22 billion total impressions (2.4%) 5.2 millions invitations
- 14. Skills Endorsement Dataset 14 Total dataset size: 190 million impression tuples Training dataset: 80% of data Testing dataset: 20% of data
- 15. Tencent SearchAds Dataset 15 Publicly available for KDD Cup 2012 by the Tencent search engine Total Size: 150 million impression sequences CTR of the Ad at the 1st, 2nd and 3rd position: 4.8%, 2.7%, and 1.4%
- 16. Data Analysis: ConversionRate Changes with Impression Count 16 If we show an item to a user repeatedly, the conversion rate decreases
- 17. 17 The conversionrate decreases with both ImpCount and LastSeen PYMK Invitation Rate Changes with Impression Count and Last Seen
- 18. 18 Similar observation: Theconversion rate also decreases with both ImpCount and LastSeen Endorsement Rate Changes with Impression Count and Last Seen
- 19. Impression Discounting ModelFitting 20 Workflow: Model fitting process: Fundamental discounting functions Aggregation Model Offline and online
- 20.
- 21.
- 22. Conversion Rate withLastSeen 23
- 23. Aggregation Models 24 LinearAggregation Multiplicative Aggregation f(Xi) is one of discounting function given earlier
- 24. Anti-Noise Regression Model 26 The sparsity of observations increases variance in the conversion rate. Typically happens when the number of observations are relatively low.
- 25. Density Weighted Regression 27 Add a weight with square error Original RMSE: Density weighted RMSE: vi is assessed by the number of observations in a small neighborhood of (Xi, yi)
- 26.
- 27. Offline Evaluation: Precision@k 29 Precision at top k Precision improvement for different behavior sets: More features yield higher precision
- 28. Online Evaluation: BucketTest on PYMK 30 Use behavior set (LastSeen, ImpCount) On LinkedIn PYMK online system
- 29. Conclusion 31 Impression Discounting Model Learnt a discounting function to capture ignored impressions Linear or multiplicative aggregation model Anti-noise regression model Offline and Online evaluation (Bucket testing)
- 30. Questions? 32 Related workin the paper Want to know more contact: mtiwari@linkedin.com
Editor's Notes
- #2 Good afternoon! I am Mitul Tiwari. Today I am going to talk about Modeling Impression Discounting in Large-scale Recommender Systems. This is joint work with Pei Lee, Laks, and Sam Shah. Most of the work was done when Pei Lee was interning at LinkedIn last summer.
- #3 Let me start with giving some context. LinkedIn is the largest professional network with more than 300M members, and its growing very fast with more than 2 members joining LinkedIn per second.
- #4 Here are a couple of examples of large scale recommender systems at LinkedIn. People You May Know: recommending other people to connect with. Daily we process 100s of tera-bytes of data and 100s of billions of edges to recommend connections for each of 300M members. Suggested skills endorsements: is another example where we recommend a member and skill combination for endorsements.
- #5 There are many such large scale recommender systems at LinkedIn. For example, Jobs recommendations, news articles suggestions, companies to follow, groups to join, similar profile, related search query etc.
- #6 Let me describe what we mean by impression discounting using a couple of examples and formulate the problem.
- #7 Let me start with the case of People You May Know or PYMK. Let’s say 5 days ago a Deepak was recommended to me to invite and connect on LinkedIn, and I ignored that recommendation. And Deepak is recommended to me again in PYMK 1 day ago. Now today PYMK systems needs to figure out whether to recommend Deepak again to me or not. Basically, if I am not likely to invite Deepak to connect then better discount this recommendation from my PYMK recommendations. Note that this is just an example since I am already connected to Deepak and I work very closely with him
- #8 Here is another example of skills endorsement suggestions. We have very small real-estate on the website to display endorsement suggestions. If we recommended certain skills in the past for some of your connections and you did not take any action on those suggestion. If you are not likely to endorse with those skills then it’s better if we suggest some other skills to endorse.
- #9 Let me define certain terms. Impressions are recommendation that are seen by users. Conversion is a positive action taken on a recommendation. For example, in case of PYMK, invitation to connect. Or in case of Skills Endorsements, endorsing a connection for a skill. In Impression discounting problem, we have given a history of impressions or recommendations seen by users, and we would like to discount or remove the items from the recommendation set if it’s not likely to lead to conversion.
- #10 Our goal is to build an impression discounting framework that can be used as a feature in the recommendation engine model or a plugin that can be applied on top of the existing recommendation engine. Our proposed impression discounting model only relies on the historical impression records, and can be treated as a plugin for existing recommendation engines. We use the dotted rectangle to circle the newly built recommender system with impression discounting, which produces discounted impressions with a higher overall conversion rate.
- #11 Here is the outline of the rest of my talk. Let me define impression more in detail in terms of user, item, conversion, and behavior features. Then I will describe the exploratory data analysis work to find out the correlation between impression features and conversion rate Then will describe the impression discounting model fitting And finally will conclude with offline and online experimental evaluations
- #12 An impression can be seen as a tuple with multiple fields such as User; (2) Item; (3) Conversion; (4) Features such as how many times this item was shown to the user; when was the last time this item was shown to the user, etc (5) Timestamp; (6) Recommendation score of the item to the use
- #13 We can derive various features from impressions such as: (1) Impression count: the number of times the item was seen by the user; (2) Last seen: when was the last time the item was seen; (3) at what position the item was seen; (4) how frequently user
- #17 Here are some data analysis results that shows that conversation rate drops as the number of the same item shown to the user increase The decrease in conversion rate is much more steep in case of endorsements Also note that in case of PYMK the decrease is more gradual. That mean, a recommendation in PYMK still has chance of acceptance even after multiple impressions
- #21 Given this data and after doing correlation studies between various behavior features and conversion rate We would like to learn discounting functions that fits this data
- #22 In the plot we show various discounting functions that we use to fit the relationship be tween conversion rate and the number of times an item was shown to a user.
- #23 These are various discounting functions that we use to fit the data
- #25 After fitting the best discounting function for each of the feature, we combined these functions using linear or multiplicative aggregation
- #27 Note that in certain cases we didn’t have a lot of data points. For example, a very few items were shown to users more than 40 times and there is a lot of variance because of small number of data points. We developed an anti-noise regression technique to give lower weightage to such data points in the cost functions
- #28 We adopted this technique from density based clustering to give lower weight to certain observations in the cost function Throw away information that you don’t need – analogy – denoising auto encoders in deep learning
- #29 Our proposed impression discounting model only relies on the historical impression records, and can be treated as a plugin for existing recommendation engines. We use the dotted rectangle to circle the newly built recommender system with impression discounting, which produces discounted impressions with a higher overall conversion rate.
- #32 Talked about modeling impression discounting in large scale recommender systems at LinkedIn such as People You May Know and Skills Endorsement Suggestions.