1) The document presents benchmark results that compare the performance of different cloud providers and instance types across metrics like CPU, concurrency, memory, and disk performance. 2) There is significant variation in performance both between different cloud providers and instance types, and also within the same instance type over time. 3) While the cloud provides benefits like easy resizing and deployment, raw performance is generally better with dedicated hosting and benchmarks show the cloud is not always lower cost. Careful performance testing is needed to select the best instance type for each application.

1. The Secret Guide to
Cloud Performance
Jan 30 2014 @ DevCon TLV
Gideon Greenspan
Co-Founder, Cloudlook

2. Site/DB
Site/DB
Site/DB
OS etc. Site/DB
OS + web
server + DBMS
OS etc. Site/DB
Website +
Database
OS etc. Site/DB
Types of hosting
OS + web
server + DBMS
Hardware
Hypervisor
Host OS
Hardware
Hardware
Dedicated
Cloud / VPS
Shared
Secret Guide to Cloud Performance
Slide 2
Cloudlook

3. “The Cloud” vs Old-School VPS
• Deploy/decommission in minutes
• Scale up/down in minutes
• Management APIs
• Automatic scaling
• Billing by the hour
• Migration on hardware failure
Secret Guide to Cloud Performance
Slide 3
Cloudlook

4. Some public cloud providers
Secret Guide to Cloud Performance
Slide 4
Cloudlook

5. Cloud specifications — AWS EC2
Secret Guide to Cloud Performance
Slide 5
Cloudlook

6. Cloud specs — Rackspace
Secret Guide to Cloud Performance
Slide 6
Cloudlook

7. The trouble with the cloud
• Vague system specifications
• Variation between instances
– Noisy neighbors?
– Diverse underlying hardware
• Variation over time within an instance
• One-time benchmarks insufficient
Secret Guide to Cloud Performance
Slide 7
Cloudlook

8. Cloudlook: continuous benchmarking
• 6 cloud providers, 35 instance types
• 15,000 instances per month
• 50 samples per instance over 1 hour
• All 64-bit CentOS (even on Azure!)
• $2,000 per month
• Available free for your server too!
Secret Guide to Cloud Performance
Slide 8
Cloudlook

9. What does Cloudlook measure?
• Integer CPU performance
• Concurrency
• Memory bandwidth
• Sequential disk access
All done with…
• Random disk access
• Host disk cache effect
Secret Guide to Cloud Performance
Slide 9
Cloudlook

10. The Secret Guide
• Introduction
• Benchmarks
• Conclusions
Secret Guide to Cloud Performance
Slide 10
Cloudlook

11. Measuring integer CPU
• PHP scripts run as interpreted bytecode
• Built-in PHP functions are compiled C
• We benchmarked 14 heavy functions
across 66 versions of PHP ≥ 5.0.4
• Two performed consistently (±0.2%)
– sha1() — hash function
– levenshtein() — deep string comparison
Secret Guide to Cloud Performance
Slide 11
Cloudlook

12. CPU results
Core i7
equivalent
Between
instances
Within
instances
1
GoGrid (all)
2.05 GHz
± 6%
± 4%
2
HP Cloud (all)
2.01 GHz
± 12%
± 10%
3
Rackspace Performance 1 (all)
2.01 GHz
± 4%
± 4%
4
Rackspace Standard (all)
1.93 GHz
± 6%
± 5%
5
Windows Azure (all)
1.74 GHz
± 3%
± 11%
6
Digital Ocean ≥1GB
1.62 GHz
± 19%
± 12%
7
Amazon m1.medium
1.48 GHz
± 6%
± 5%
Secret Guide to Cloud Performance
Slide 12
Cloudlook

13. CPU variation per instance
Rackspace Cloud Performance 1 2GB
Digital Ocean 2GB (sometimes)
Secret Guide to Cloud Performance
Slide 13
Cloudlook

14. Measuring concurrency
• Parallel CPU-heavy HTTP requests
• Precise client-side synchronization
– microtime() + usleep()
• Avoid CPU temperature bias
Parallel
HTTP
requests
1
2
3
4
Secret Guide to Cloud Performance
Slide 14
Cloudlook

15. Concurrency results
Measured concurrency
5
4
Windows Azure
Digital Ocean + HP Cloud
Rackspace Performance 1
GoGrid
Amazon EC2
Rackspace Standard
3
2
1
1
2
3
4
5
6
7
8
Specified vCPUs / cores
Secret Guide to Cloud Performance
Slide 15
Cloudlook

16. Measuring memory bandwidth
• Create a 64 MB string in memory
– Largest L1+L2+L3 cache currently ~41 GB
• substr() to read successive windows
– memcpy() in C does the actual work
• Window length = 128 kb
– Too short ⇒ time in bytecode interpreter
– Too long ⇒ copy too big for L1+L2 cache
Secret Guide to Cloud Performance
Slide 16
Cloudlook

17. Memory results
Memory
bandwidth
Between
instances
Within
instances
1
GoGrid
6.7 GB/s
± 10%
± 7%
2
Rackspace Performance 1
6.7 GB/s
± 7%
± 4%
3
Digital Ocean ≥1GB
6.3 GB/s
± 10%
± 8%
4
HP Cloud
5.9 GB/s
± 9%
± 12%
5
Amazon m1.medium
5.2 GB/s
± 25%
± 4%
6
Rackspace Standard
4.1 GB/s
± 16%
± 7%
7
Windows Azure
3.6 GB/s
± 5%
± 6%
Secret Guide to Cloud Performance
Slide 17
Cloudlook

18. Memory performance variation
HP Cloud Standard Medium
Rackspace Cloud Standard 4GB
Digital Ocean 4GB
Amazon m1.large
Secret Guide to Cloud Performance
Slide 18
Cloudlook

19. Measuring disk performance
• Write benchmark files in /tmp
• Avoid operating system cache
– Total size > instance memory
– Explicitly clear cache (our instances)
– Read each block once (our instances)
• “Random” is pseudo-random
– Fibonacci hashing scheme
Secret Guide to Cloud Performance
Slide 19
Cloudlook

20. OS etc. Site/DB
OS etc. Site/DB
OS etc. Site/DB
Host cache interference
Hypervisor
Host OS
Hardware
Secret Guide to Cloud Performance
• Results can be skewed
• Big instances: write lots of
other data inbetween
• Small instances: wait a
week and pray…
• But still worth measuring
because it’s a good thing!
Slide 20
Cloudlook

21. Disk results (no host cache)
10,000
EC2 m1.medium Primary
Disk Seeks — ops/s
EC2 m1.medium Secondary
Rackspace Standard >= 2GB
Rackspace Performance 1 >= 2GB Both
1,000
Azure >= 1.5 GB Primary
Azure >= 1.5 GB Secondary
HP Cloud >= 2GB Both
Digital Ocean >= 1GB
100
GoGrid >= 2 GB
10
10
100
1000
Disk Throughput — MB/s
Secret Guide to Cloud Performance
Slide 21
Cloudlook

22. Who has a host cache?
Primary disk rate
(no host cache)
Primary disk rate
(host cache)
Host cache
speedup
HP Cloud Small
109 MB/s
528 MB/s
4.8 ⤬
HP Cloud Medium
124 MB/s
320 MB/s
2.6 ⤬
HP Cloud Large
110 MB/s
117 MB/s
1.1 ⤬
Digital Ocean 1 GB
244 MB/s
709 MB/s
2.9 ⤬
Digital Ocean 2 GB
267 MB/s
703 MB/s
2.6 ⤬
Digital Ocean 4 GB
260 MB/s
709 MB/s
2.7 ⤬
Digital Ocean 8 GB
318 MB/s
474 MB/s
1.5 ⤬
Secret Guide to Cloud Performance
Slide 22
Cloudlook

23. The Secret Guide
• Introduction
• Benchmarks
• Conclusions
Secret Guide to Cloud Performance
Slide 23
Cloudlook

24. Same instance, different stats
EC2 m1.medium
GoGrid 4 GB
Rackspace
Performance 1
4 GB
Integer CPU
1.4 : 1
1.2 : 1
1.5 : 1
Concurrency
1.1 : 1
1.5 : 1
1.5 : 1
Memory
3.8 : 1
1.6 : 1
1.7 : 1
Disk throughput
3.0 : 1
10.0 : 1
3.0 : 1
Disk seeks
15.8 : 1
4.2 : 1
3.3 : 1
Secret Guide to Cloud Performance
Slide 24
Cloudlook

25. Dedicated vs Cloud
Hetzner EX-40
GoGrid 2 GB
GoGrid 8 GB
Integer CPU
3.49 GHz
2.04 GHz
2.09 GHz
Concurrency
6.82 ⤬
1.83 ⤬
4.61 ⤬
Memory
12.9 GB/s
6.6 GB/s
6.9 GB/s
Disk throughput
155 MB/s
387 MB/s
381 MB/s
Disk seeks
134 ops/s
127 ops/s
112 ops/s
32 GB
2 GB
8 GB
Disk size
1000 GB
100 GB
400 GB
Price
$67/mo
$66/mo
$263/mo
Memory size
Secret Guide to Cloud Performance
Slide 25
Cloudlook

26. Bad reasons to host in the cloud
• Hardware abstraction (peace of mind)
– Last week Rackspace Cloud lost my data
• Easy resizing
– Dedicated is at top of scale anyway
– Can’t resize Rackspace Performance servers
• Lower costs
– No. That’s why hosting companies like it…
Secret Guide to Cloud Performance
Slide 26
Cloudlook

27. Some good reasons
• Small instances
• Fast deploy/decommission
• Tools and APIs
• Other cloud services
– Storage and CDN
– Databases and messaging
Secret Guide to Cloud Performance
Slide 27
Cloudlook

28. Image from http://www.kapowwowobjects.com
So: Beware of fashions…
Secret Guide to Cloud Performance
Slide 28
Cloudlook

29. … and measure your own
Secret Guide to Cloud Performance
Slide 29
Cloudlook

30. Thank you!
www.cloudlook.com
Secret Guide to Cloud Performance
Slide 30
Cloudlook