Presented by: Moshe Zadka, SurveyMonkey Presented at All Things Open 2020 Abstract: Modern micro-services architecture means that a typical front-end service call can fan out to many calls on the backend, easily twenty or more. Being able to write API client code that is fast is important. The talk will cover how to use async networking techniques to write faster API clients.

1. Screaming Fast API Clients
Moshe Zadka – https://cobordism.com
2020

2. Acknowledgement of Country
San Francisco Bay Area Peninsula
Ancestral homeland of the Ramaytush Ohlone

3. Latency is the Site Killer
Every 100ms of latency in your site lose more customers

4. (Micro)service Architecture
Layers

5. (Micro)service Architecture
Fan-out

6. Lognormal Black Swans
Lognormal: 1/x (kinda)
Normal: e−x2

7. Averages Lie
Only good for normal distributions

8. Your Backend is Slow
Lognormal, not normal

9. Multiplicity Magnifies Outliers
With 5 queries:
P90 becomes P50
P99 becomes P90

10. Measure
Histograms, not averages

11. Measure
All layers

12. Let’s Write Some Code
@app . route ( ’/ ’)
def h e l l o w o r l d ( ) :
a l l v a l u e s = sum(
CLIENT . get (URL ) . json ( ) [ ” value ”]
f o r x in range (FANOUT)
)
r e t u r n json . dumps ( d i c t ( t o t a l=a l l v a l u e s ))

13. Let’s Write Some Code
@app . route ( ’/ ’)
async def h e l l o w o r l d ( request ) :
a l l v a l u e s = await d e f e r . g a t h e r R e s u l t s ( [
CLIENT . get (URL ) . addCallback ( treq . j s o n c o n t e n
f o r x in range (FANOUT)
] )
t o t a l = sum( r e s [” value ”] f o r r e s in a l l v a l u e s )
r e t u r n f ’ Total { t o t a l } ’

14. Let’s Simulate
With fanout of 10:
P50: each: 0.04 seq: 0.82 par 0.3
P90: each: 0.23 seq: 1.8 par 0.98
P99: each: 1.04 seq: 4.33 par 3.05

15. Timing Out and Retry
Temporary slow-downs

16. Let’s Write Some Code
def g et w ith ti me ou t ( u r l ) :
def t r y ( i g n=None ) :
r e t u r n CLIENT . get (URL ) . addCallback ( treq . j s o n
d = t r y ()
d . addTimeout ( 0 . 1 )
d . addErrback ( t r y )
r e t u r n d

17. Let’s Simulate
P50: 0.18
P90: 0.51
P99: 1.66

18. Let’s Simulate
Retried requests: 25

19. Let’s Write Some Code
def g et w ith ti me ou t ( u r l ) :
def t r y ( i g n=None ) :
r e t u r n CLIENT . get (URL ) . addCallback ( treq . j s o n
d = t r y ()
d . addTimeout ( 0 . 1 )
d . addErrback ( t r y )
d . addTimeout ( 0 . 4 )
r e t u r n d

20. Let’s Simulate
P50: 0.19
P90: 0.53
P99: 0.6

21. Summary
Latency
Backend latency
SLA
Measurement
Simulation