In this presentation, Snap Interactive goes through how they use MongoDB to scale Facebook's Real-time endpoint. In building AreYouInterested.com, SNAP Interactive wanted to instantly update people's profiles based on users' individual "likes". In this talk, Mike and Justin from SNAP Interactive will go through how they configured MongoDB to handle thousands of user profile updates.
MongoDB .local Toronto 2019: Keep your Business Safe and Scaling Holistically...
Scaling Facebook's Realtime Endpoint with MongoDB, Snap Interactive
1. Scaling The Facebook Realtime
Endpoint Using MongoDB
PRESENTED BY:
Justin Medoy and Mike Sherov
SNAP Interactive
jmedoy@snap-interactive.com
mikesherov@snap-interactive
3. What are Facebook Realtime Updates?
Facebook says: "Real-time updates enable your
application to subscribe to changes in data in
Facebook."
What it means: "You provide a URL,Facebook pings it
when users do stuff."
4. Pings from Facebook
● Every minute we get around 20 pings from
facebook that contain data for around 11,000 users
{
"object": "user",
"entry":
[
{
"uid": 1335845740,
"changed_fields":
[
"name",
"picture"
],
"time": 232323
},....
]
}
5. WHAT?!? Where's the data?
● Facebook tells you that something about the field
changed, but not what the current data is.
6. Retrieving User Data from the
Graph
● Solution: go back to Facebook and grab
the user's data
https://graph.facebook.com?
ids=<USERID>&fields=music,movies,likes
*This will only get data that the user has made publicly available
● To avoid timeouts each call to Facebook only asks
for the data for 25 users
*Our CURL timeouts for Facebook have been lowered from the
default 60 seconds to 25 seconds
7. Update the user's profile
● Facebook won't tell you exactly what's
changed but we can figure it out from our
own data
All Data - Stored Data = Changed Data
● The next step is to update the user's
profile with this changed data
8. Mongo Architecture
● Mongo 2.0.2
● Mongo PHP driver 1.2.10
● Two separate replica sets
○ User data
○ Interest data
● Why separate replica sets?
○ Keep as much of the index as possible in
memory
○ Disk reads are expensive
9. User Data Replica Set
Design Challenge
● Random access pattern over 106 million
documents
10. User Data Replica Set
● Large $in queries
● High page faults in
MMS
● We upgraded from
32G to 128G on
each node
11. Indexes
● We added duplicates of some of our
indexes with reversed fields
● Updating all of these extra indexes was a
huge bottleneck
13. Indexes
● There were certain minutes when Facebook would
tell us that the data had changed for more than
40,000 users
○ limit the amount of data Facebook can send in one minute
● High number of writes and a large number of
indexes prevented the secondaries from reading
the oplog because of the global write lock
○ Increase the size of the oplog
○ This is fixed in 2.2.1
14. Indexes and the realtime endpoint
profile.sync_1_installed_1_platforms.facebook_1
● Filtered 11,000 users a minute down to a few hundred
○ moved filtering logic out of PHP into the index
● Added efficiency from covered index
○ All we need is platforms.facebook, which is part of the
index
15. Interest Replica Set
Different set of challenges than User repl set
● Needs to power typeahead
● 64 million interests
● Access pattern based on interest popularity
○ Lady Gaga is going to get accessed more than Ladybug, Javascript
17. The Typeahead
● Add an array with the first few characters of
interest
● Add an index on that field
● This allows us to have 10 entries in 1 index
instead of 10 separate indexes
http://docs.mongodb.org/manual/core/indexes/#index-type-multikey
19. Lessons Learned
● Don't over index
● Covered indexes when possible
● indexes to reduce size of returned data
● Keep everything in memory
● Multikey index for typeaheads
● Utilize -1 in index for natural sorting
20. SNAP Interactive, Inc.
Contact Information
● SNAP Interactive, Inc.
SNAP-Interactive.com
● Justin Medoy
Team Lead / Software Engineer
JMedoy@snap-interactive.com
● Mike Sherov
Lead Developer
mike@snap-interactive.com
@mikesherov
● For more information on our open positions, email
jobs@snap-interactive.com or check our website at meet people like you
www.snap-interactive.com/jobs/job-openings