How Beeswax scaled Python to tens of thousands of QPS.

1. Python at Scale
Concurrency at Beeswax
Ron Rothman

3. Advertiser*
Ad Exchange
Advertiser*
Beeswax
Advertiser*
Advertiser*
ad
❕ 200 ms
❕ 2 million per second
❕ non-infinite $ budget
❕ 99.99% uptime
❕ ~optimal bids
ad request bid requests
bids

4. Event Collection
Ad
Exchange Bidder
Event
Collector
Event Log
Stream
BeeswaxInternet

5. Event Collection
Ad
Exchange Bidder
Event
Collector
Event Log
Stream
BeeswaxInternet

6. Event Collection
Ad
Exchange Bidder
Event
Collector
Event Log
Stream
Load O(10K/sec)
Response time < 250 ms p99
Availability ♾️ Nines
1. Durably record the event
2. Update counters (database)
1
2

7. def handle_request(req):
'''Process an incoming event.'''
unmarshall(req) # processing
validate(req) # processing
# some business logic # processing
record_event(req) # network i/o
update_db_counters(req) # network i/o
return_response()
work
work
work
wait
wait

8. 10 requests/sec
(100 ms per request)

10. def handle_request(req):
'''Process an incoming event.'''
unmarshall(req) # processing
validate(req) # processing
# some business logic # processing
record_event(req) # network i/o
update_db_counters(req) # network i/o
return_response()
work
work
work
wait
wait

11. Layers of Concurrency
Many machines EC2; Autoscale groups
Many processes Preforking web servers
Many threads Greenlets; asyncio
Containers ECS tasks
Serverless Lambdas

12. Elastic Load Balancer
EC2 EC2 EC2 EC2 EC2
autoscale
group
event
notifications
(HTTP requests)

13. EC2 instance
event
notifications
(from LB)
web server
processes

14. web server
process
greenlets/threads
requests

15. Threads or Greenlets?

16. Threads Greenlets
Preemptive Cooperative
Requires extensive locking Requires no locking
Lightweight Very lightweight
Leverage multiple cores* Single core
*Sadly, untrue for CPython

17. Gevent
● Create & manipulate greenlets
● Allows you to do non-blocking i/o

18. def handle_request(req):
'''Process an incoming event.'''
unmarshall(req) # processing
validate(req) # processing
# some business logic # processing
record_event(req) # network i/o
update_db_counters(req) # network i/o
return_response()
work
work
work
wait
wait

19. Gevent
● Create & manipulate greenlets
● Allows you to do non-blocking i/o
● Makes (i/o) libraries that you use non-blocking!
○ "monkey patching"

20. def handle_request(req):
'''Process an incoming event.'''
unmarshall(req) # processing
validate(req) # processing
# some business logic # processing
record_event(req) # network i/o
update_db_counters(req) # network i/o
return_response()
work
work
work
yields
yields

21. Happily Ever After?

22. Reality def handle_request(req):
'''Process an incoming event.'''
unmarshall(req)
decrypt(req)
validate(req)
check_whether_duplicate(req) # BLOCKING i/o
# some business logic
# more business logic
# even more business logic
record_event(req) # nonblocking i/o
update_db_counters(req) # BLOCKING i/o
update_other_db_counters(req) # BLOCKING i/o
update_other_other_db_counters(req) # BLOCKING i/o
return_response()

23. Reality def handle_request(req):
'''Process an incoming event.'''
unmarshall(req)
decrypt(req)
validate(req)
check_whether_duplicate(req) # BLOCKING i/o
# some business logic
# more business logic
# even more business logic
record_event(req) # nonblocking i/o
update_db_counters(req) # BLOCKING i/o
update_other_db_counters(req) # BLOCKING i/o
update_other_other_db_counters(req) # BLOCKING i/o
return_response()

24. Reality def handle_request(req):
'''Process an incoming event.'''
unmarshall(req)
decrypt(req)
validate(req)
check_whether_duplicate(req) # BLOCKING i/o
# some business logic
# more business logic
# even more business logic
record_event(req) # nonblocking i/o
update_db_counters(req) # BLOCKING i/o
update_other_db_counters(req) # BLOCKING i/o
update_other_other_db_counters(req) # BLOCKING i/o
return_response()

25. Reality def handle_request(req):
'''Process an incoming event.'''
unmarshall(req)
decrypt(req)
validate(req)
check_whether_duplicate(req) # BLOCKING i/o
# some business logic
# more business logic
# even more business logic
record_event(req) # nonblocking i/o
update_db_counters(req) # BLOCKING i/o
update_other_db_counters(req) # BLOCKING i/o
update_other_other_db_counters(req) # BLOCKING i/o
return_response()

26. Redis client ✅ pure python
DynamoDB client ✅ pure python
Aerospike client ⛔ wrapped C code
Which DB Libraries Can Be Monkey Patched?

27. Too Much Processing? Yield Often.
def handle_request(req):
'''Process an incoming event.'''
unmarshall(req)
decrypt(req)
greenlet_yield()
validate(req)
check_whether_duplicate(req) # BLOCKING i/o
# some business logic
# more business logic
greenlet_yield()
# even more business logic
record_event(req) # nonblocking i/o
...

28. Blocking C Code? Batch & Timeouts.
def handle_request(req):
'''Process an incoming event.'''
unmarshall(req)
decrypt(req)
validate(req)
# BLOCKING i/o
check_whether_duplicate(req, timeout_ms=5, max_tries=3)
# business logic
record_event(req) # nonblocking i/o
update_counters_in_memory(req) # occasional i/o
return_response()

29. � Blocking I/O calls waste CPU
� You're not as I/O bound as you think hope
� C extensions play by different rules

30. ● Blocking I/O calls waste CPU
■ Gevent + monkey patch
● You're not as I/O bound as you think hope
■ Buffer & batch
● C extensions play by different rules
■ Short timeouts w/retries

31. Thank You 🙏�
ron {at} beeswax.com

32. References
● Gevent
● Falcon
● Bottle
● The Sharp Corners of Gevent
● Beeswax