Compute Cloud Performance Showdown: 18 Months Later (OCI, AWS, IBM Cloud, GCP, Azure)

Chicago Oracle Users Group
April 15, 2021
7:00pm EST
Compute Cloud Performance Showdown: 18 Months Later
Oracle Cloud, Amazon Web Services, IBM Cloud, Google Cloud, Microsoft Azure

© Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 2 of 58
@Revelation_Tech
INTRODUCTION

@Revelation_Tech
About Me
• Master’s degree in Computer Science from George Mason University
• Recent emphasis on cloud, DevOps, middleware, and security in current projects
• Oracle ACE, OCE, OCA
• Author, Blogger, Presenter
• @Ahmed_Aboulnaga
Sponsor

@Revelation_Tech
CONCLUSIONS

@Revelation_Tech
Disclaimer
• The results of the test results documented in this presentation should not be considered
definitive
• A lot of details surrounding setup, configuration, and assumptions regarding the test cases
are not documented in this presentation
• Results vary with repeated testing:
‒ Ongoing/unknown backend and hardware changes at each provider
‒ Varying load on backend hardware due to multitenancy
‒ Lack of backend access makes interpreting results difficult at times
• Testing limitations exist (see next slides)

@Revelation_Tech
Objective
• Conduct 2nd round of tests to compare compute cloud (i.e., IaaS) performance across 5
cloud providers against comparable medium-sized instance types:
‒ Amazon Web Services (AWS)
‒ Oracle Cloud Infrastructure (OCI)
‒ IBM Cloud
‒ Google Cloud Platform (GCP)
‒ Microsoft Azure
• Compare performance of the following:
‒ Linux Host
‒ Oracle WebLogic Server 14c
‒ Oracle Database 19c
• Compare results to 2019 tests

@Revelation_Tech
Testing Limitations
• Tested on single virtual machine (i.e., did not recreate instances and retest)
• Did not test different instance types
• Did not test across different data centers
• All virtual machines are multitenant-based (i.e., no dedicated hardware)
• All instances and software configured exactly identically*, and no tuning performed

@Revelation_Tech
Summary of Results | January 2019
• Similar tests were conducted in January 2019
‒ At the time, difficult to perform apples-to-apples comparison of cloud providers
‒ Nothing alarming in the performance results; more powerful CPUs yielded better performance
• Linux Host performance:
‒ Amazon Web Services had the slight processing edge due to newer and higher-end CPU model
• Oracle WebLogic Server performance:
‒ Microsoft Azure slightly underperforms compared to the other providers
• Oracle Database performance:
‒ Microsoft Azure consistently had the poorest throughput and performance

@Revelation_Tech
Findings from Performance Tests | October 2020
▪ IBM and Azure have slight edge on CPU
▪ OCI completely dominates on I/O (again)
▪ Unacceptable I/O from AWS, IBM, and GCP
▪ OCI has the best performance-to-cost value
▪ Azure slightly outperformed the others
▪ OCI slightly underperformed relative to the others
▪ GCP has the slight edge here
▪ AWS dips compared to 2019 and freezes towards end of test
▪ Azure demonstrates notable improvements, but still lags
behind the others
Interpret results with caution!

@Revelation_Tech
Concluding Our Performance Tests | What We Learned
 Difficult to conclusively determine a leader
 Specifications (expectations) ≠ Performance (reality)
 Inconsistency in performance is likely attributed to multitenancy
 IBM Cloud has improved in terms of stability and service offerings
 Google Cloud Platform, Microsoft Azure, and IBM Cloud have improved in
performance
 No competition to Oracle Cloud Infrastructure in terms of cost and overall value
 Amazon Web Services remains the gold standard that all other CSPs are trying to catch up
to, but there are compelling reasons to consider other providers (specifically Oracle Cloud
Infrastructure and Google Cloud Platform).

@Revelation_Tech
COMPARING THE 5 CLOUD PROVIDERS

@Revelation_Tech

@Revelation_Tech
AWS Experiences
• EC2 instance startup/shutdown second slowest to OCI
• Remains the gold standard that all other CSPs are trying to catch up to

@Revelation_Tech
OCI Experiences
• Took much longer to startup/shutdown than other providers
– Never experienced this with other tenancies
• Remains the lowest cost of all providers and has the best overall value

@Revelation_Tech
IBM Experiences
Improvements
• No longer forces you to use pay for a hardware firewall using PayPal
• Now allows full firewall management via Security Groups
Odd
• Consistently takes 12 seconds for Linux password prompt to appear
• VMs default to America/Chicago CDT timezone despite data centers in
Northern Virginia (all others default to UTC or GMT)
• Large number of interfaces on instance
Cons
• Consistently overcharges in monthly invoices; was informed this is due to
merging of billing systems (but they never undercharge)
• Restart required when attaching a new Security Group to a VM*
• Only provider that does not disable ‘root’ login

@Revelation_Tech
GCP Experiences
• Only provider to always warns you about data loss before shutting down (ask yourself why)
• Firewall rule names can only have hyphens and lowercase letters
• Only provider that recommended increasing hardware after a series of stress tests

@Revelation_Tech
Azure Experiences
• Offers Oracle Linux images
• Frequently renamed disks (alternated /dev/sdb with /dev/sdc between reboots)

@Revelation_Tech
Other Observations – Shutdown Times
• AWS and OCI take much longer to shutdown the OS and poweroff the VMs than the others
• Results have been relatively reproducible
• Have not experienced these long shutdown times in other accounts or in 2019
OS Shutdown
Duration
Full VM Poweroff /
Deallocation
AWS 4:05 minutes 4:31 minutes
OCI 15:40 minutes 15:40 minutes
IBM 0:09 minutes 8:08 minutes
GCP 0:05 minutes 0:38 minutes
Azure 0:05 minutes 4:22 minutes

@Revelation_Tech
CPU/Memory Cost Comparison
Instance Type vCPUs RAM
Cost
(per hour)
m5.4xlarge 16 64 GB
$0.8980
$0.7680
VM.Standard2.8
VM.StandardE2.8
VM.Standard2.8
16
16
16
120 GB
64 GB
120 GB
$0.5104
$0.2400
$0.5104
B1.16x64 16 64 GB
$0.7722
$0.7680
(Custom) 16 64 GB
$0.9183
$0.6037
D16s_v3
D16as_v4
16
16
64 GB
64 GB
$0.8980
$0.7680
* CPU/memory cost only (excludes storage, firewall, static IPs, load balancers, etc.)
$647
$367
$556
$661 $647
$553
$173
$553
$435
$553
AWS OCI IBM GCP Azure
Cost Per Month
CPU/memory only, based on official pricing sheets, not actuals
2019 2020
* Values with strikethrough were previously used in 2019 tests

@Revelation_Tech
Virtual Machines Specifications
• Some variation in instances configuration compared to 2019 tests
Region N. Virginia US-ASHBURN-AD-1
NA East
(WDC01)
us-east4
(Northern Virginia)
East US
Profile / Type / Shape m5.4xlarge
VM.Standard2.8
VM.Standard.E2.8
VM.Standard2.8
B1.16x64 (custom)
D16s_v3
D16as_v4
vCPU 16 16 16 16 16
Memory 64 GB
120 GB
64 GB
120 GB
64 GB 64 GB 64 GB
Kernel
3.10.0-957.el7.x86_64
3.10.0-1127.10.1.el7.x86_64
4.14.35-1818.5.4.el7uek.x86_64
4.14.35-1902.301.1.el7uek.x86_64
3.10.0-957.1.3.el7.x86_64
3.10.0-1127.8.2.el7.x86_64
3.10.0-957.1.3.el7.x86_64
3.10.0-1127.10.1.el7.x86_64
3.10.0-957.1.3.el7.x86_64
4.14.35-
1902.302.2.el7uek.x86_64
Operating System
RHEL 7.6
RHEL 7.7
OL 7.6
OL 7.8
RHEL 7.6
RHEL 7.7
RHEL 7.6
RHEL 7.8
RHEL 7.6
OL 7.7
Model Intel Xeon Platinum 8175M CPU
Intel Xeon Platinum 8167M CPU
AMD EPYC 7551 32-Core
Intel Xeon Platinum 8167M CPU
Intel Xeon CPU E5-2683 v3
Intel Xeon Gold 6130
Intel Xeon CPU
Intel Xeon CPU E5-2673 v3
AMD EPYC 7452 32-Core
MHz 2.50 GHz
2.00 GHz
2.00 GHz
2.00 GHz
2.10 GHz
2.20 GHz
2.40 GHz
2.35 GHz

@Revelation_Tech
Test Cases
• In 2019, tests were taken on separate days:
– Test 1 on weekday (Jan 2019)
– Test 2 on weekend (Jan 2019)
• In 2020, tests were also taken on separate days:
– Test 3 on weekday (Sep 2020)
– Test 4 on weekend (Sep 2020)
– Test 5 on weekday (Oct 2020)
– Test 6 on weekend (Oct 2020)
Legend

@Revelation_Tech
Evaluating the Chosen OCI CPU Models
• OCI tests were repeated on both VM.Standard.E2.8 (AMD) and VM.Standard2.8 (Intel)
instances.
• This is due to the skewed results compared to 2019 when switching the CPU model.
0
500
1,000
1,500
2,000
2,500
3,000
VM.Standard.E2.8
AMD EPYC 7551
VM.StandardE2.8
Intel Xeon
Platinum 8167M
CPU Stress Test (2020)
(higher is better)
AMD EPYC 7551
Intel Xeon
Platinum 8167M
Release Date Q2 2017 Q3 2017
Frequency 2.0 GHz 2.0 GHz
Turbo Frequency 3.0 GHz 2.4 GHz
L1 Cache 2048 KB 832 KB
Cores 32 26
Threads 64 52
Multiprocessing 2 8
https://www.cpu-world.com/Compare/359/AMD_EPYC_7551_vs_Intel_Xeon_8167M.html
▪ Impossible to identify
which of the selected
AMD or Intel CPUs
on OCI are more
powerful based on
specifications alone.
▪ We began OCI tests
with the AMD EPYC
7551 CPU model first.
▪ The OCI AMD model
is 53% lower in cost
and 20% lower in
CPU performance
than the Intel model.
OCI CPU Challenge
OCI CPU Findings
Processor Specifications Actual Performance Test Results

@Revelation_Tech
Comparing CPU Marks
• Should research CPU specifications in conjunction with instance shapes.
• The “CPU Mark” is a measure of the CPU’s performance (higher is better).
• Table below is based off of comparisons from www.CPUBenchmark.net.
• Google Cloud processor specifications remain unknown.
https://www.cpubenchmark.net/compare/Intel-Xeon-Platinum-8175M-vs-AMD-EPYC-7551-vs-Intel-Xeon-Gold-6130-vs-AMD-EPYC-7452/3311vs3089vs3126vs3600
Intel Xeon
Platinum
8167M @
2.00 GHz
▪ Azure should
outperform other
providers based on the
CPU Mark ratings.
▪ IBM Cloud should
underperform relative
to the other providers
based on CPU Mark.
▪ OCI AMD model is 53%
lower in cost than its
Intel model, so likely will
perform worse.
▪ Keep in mind that AWS,
IBM, and Azure are
priced identically.
▪ Other factors have
impact on overall
performance, but no
details are provided;
controller,
motherboard, etc.
Our Expectations
?
?
?

@Revelation_Tech
LINUX STRESS TESTS

@Revelation_Tech
Testing Tool
• stress-ng
– yum install stress-ng
• Simple workload generator that will stress test a server for the following features:
– CPU compute
– Cache thrashing
– Drive stress
– I/O syncs
– VM stress
– Socket stressing
– Context Switching
– Process creation/termination
– Much more
stress-ng: info: [12157] successful run completed in 322.04s (5 mins, 22.04 secs)
stress-ng: info: [12157] stressor bogo ops real time usr time sys time bogo ops/s bogo ops/s
stress-ng: info: [12157] (secs) (secs) (secs) (real time) (usr+sys time)
stress-ng: info: [12157] cpu 637597 309.01 5134.20 0.00 2063.32 124.19
bogo
Bogus operations per second, are not comparable between different stressors.
bogo ops/s (real time)
Total bogo operations per second based on wall clock run time. The wall clock time reflects the apparent run
time. The more processors one has on a system the more the work load can be distributed onto these and
hence the wall clock time will reduce and the bogo ops rate will increase. This is essentially the “apparent”
bogo ops rate of the system.

@Revelation_Tech
Stress Tests
• Stress Test:
– Number of Tests: 4
– Types: CPU, Memory, I/O, Large File Copy
– Duration: 15 minutes

@Revelation_Tech
CPU Stress Test
• Findings:
– AWS and GCP tested with same specification as 2019, but…
– IBM and Azure benefitted notably from a change in CPU model
– OCI consistent with 2019 tests (compare Tests #1 and #2 with #5 and #6)
• Command:
stress-ng --cpu 2000 --timeout 15m --verbose --metrics-brief
0
1,000
2,000
3,000
4,000
CPU Stress Test
(higher is better)
Switch to
AMD
Upgrade
to AMD
Different
CPU model
For all host, application, and
database performance testing:
▪ AWS tests in 2019 and 2020 are
using exactly identical shapes.
▪ OCI tests in 2020 were
performed against AMD (Tests 3
& 4) and Intel (Tests 5 & 6)
CPUs.
▪ OCI tests conducted in 2019
(Tests 1 & 2) use the identical
CPU model as 2020 (Tests 5 &
6).
▪ IBM Cloud tests conducted in
2019 (Tests 1 & 2) had an older
Intel CPU model compared to
2020 (Tests 3, 4, 5, & 6).
▪ GCP tests conducted in 2019
and 2020 are on identically
sized VMs, but it is unknown if
the underlying CPU has
changed.
▪ Azure tests conducted in 2019
were on Intel-based CPUs (Tests
1 & 2) vs stronger AMD CPUs in
2020 (Tests 3, 4, 5, & 6).
Important Note About All Tests

@Revelation_Tech
Memory Stress Test
• Findings:
– Nothing alarming; comparable results
• Command (other variations tested too):
stress-ng --vm 8 --vm-bytes 6G --timeout 15m --metrics-brief
0
50,000
100,000
150,000
200,000
250,000
Memory Stress Test
(higher is better)

@Revelation_Tech
I/O Stress Test
• 3 tests conducted in 2019 due to wildly inconsistent results among many providers
• Findings:
– Inconsistent results, but OCI generally on top
– AWS and IBM consistently worse than the rest
stress-ng --io 16 --timeout 15m --verbose --metrics-brief
0
10,000
20,000
30,000
I/O Stress Test
(higher is better)

@Revelation_Tech
Large File Copy Stress Test
• Findings:
– OCI generally on top and outperforms all others by huge margin
stress-ng --hdd 8 --hdd-bytes 2G --timeout 15m --metrics-brief
0
50,000
100,000
150,000
200,000
250,000
Large File Copy Stress Test
(higher is better) Default Root
Volume Size
AWS 10 GB
OCI 40 GB
IBM 25 GB
GCP 20 GB
Azure 30 GB
▪ Involves lots of writes to the
root volume.
▪ All providers, with the
exception of OCI, gave low
disk space warnings.
Large File Copy Test

@Revelation_Tech
Interpreting the Results
▪ Performance varied up to
22% on the same VM.
▪ Multitenancy may have
unforeseen and unexpected
impact on performance.
▪ Best performance-to-cost
value.
▪ The VM.Standard AMD
model is 20% poorer in
performance than its Intel
counterpart, but 53% lower
in cost.
▪ Did not change the cost of
their instance shape
(B1.16x64), but upgraded
their backend CPU models.
▪ Switched out their backend
CPU model (a 24% boost in
the CPU Mark) which
delivered a 22% real-world
performance boost.
▪ Performance varied up to
22% on the same VM.
▪ Multitenancy may have
unforeseen and unexpected
impact on performance.
▪ No information on CPU
model.
▪ Benefited from a change in
shape from the Intel to the
AMD model, delivering a
17% boost in performance
at a 15% reduction in cost.
▪ Despite the performance
improvement, results are
underwhelming considering
the AMD model has a 67%
higher CPU mark than the
Intel.
▪ No comment. ▪ No comment. ▪ No comment. ▪ No comment. ▪ No comment.
▪ Simply unacceptable.
▪ Real-world tests may be
needed for further
validation.
▪ Consistently outperforms all
other providers.
needed for further
validation.
▪ Inconsistent performance.
▪ Impact may be due to
multitenancy.
▪ Inconsistent performance.
▪ Impact may be due to
multitenancy.
needed for further
validation.
▪ Consistently outperforms all
other providers.
needed for further
validation.
needed for further
validation.
needed for further
validation.
CPU
Stress
Test
Memory
Stress
Test
I/O
Stress
Test
Large File
Copy
Stress Test
? ✓ ✓ ? ✓
✓
   
   
✓
✓ ✓ ✓ ✓ ✓

@Revelation_Tech
Final Thoughts
What impressed us…
▪ OCI’s extremely competitive pricing provides the
best performance-to-cost value.
▪ IBM is the only provider to upgrade its backend
CPU model, while offering the instance type at the
same cost.
▪ OCI destroys all other providers in I/O related
performance tests.
▪ Azure’s AMD instance type is 17% more powerful
and 15% lower in cost than its Intel equivalent.
What disappointed us…
▪ AWS and IBM I/O performance is unacceptable.
▪ Incredible delay in login time (and WebLogic
startup) in the IBM VMs is concerning.
▪ IBM hourly cost is identical to AWS and Azure yet
offers an older generation CPU model, but
surprisingly performs similarly.
▪ Based on CPU benchmarks, Azure underwhelms in
CPU tests, but at least delivers nearly identical
performance to AWS and IBM at the same price.
✓ 

@Revelation_Tech
APPLICATION SERVER LOAD TESTS

@Revelation_Tech
Testing Scenario
• Oracle WebLogic Server 14.1.1.0.0
• Single node (no cluster, no load balancer)
• 8 GB managed server
• Application specifics:
– Minimalistic ADF application with 3 use cases
(2 web pages, RESTful services, computation intensive operations)
– Uses standard SH (Sales History) schema in the Oracle Database 19c
• Tests conducted not identical to 2019 tests, so no direct comparison offered

@Revelation_Tech
Test
• Apache JMeter 5.0
– https://jmeter.apache.org
• “The Apache JMeter application is open source
software, a 100% pure Java application
designed to load test functional behavior and
measure performance.”
• Load Test:
– Parameters: ./poc-run-all.sh -Jusers=30 -Jloops=10 -Jrampup=120 -Jlthreads=64

@Revelation_Tech
Results – Average Response Times
• No comparison to 2019 tests
0
20
40
60
80
100
120
Average Response Time (sec)
(lower is better)
AMD
▪ Azure has a slight edge.
▪ Unclear why on OCI the inferior
AMD performed identically to Intel,
possibly indicating non-CPU and
non-I/O related.
▪ Overall all providers performed
close to each other, with the
exception of 2 of the 3 tests on
OCI.
Application Performance Thoughts

@Revelation_Tech
Results – Throughput (Transactions Per Second)
• Nothing notable about throughput
• OCI generally had slightly lower TPS than the others
• OCI took longer to complete than the rest (~8 minutes)
Encrypted Sales Data (Test #4)
Product Sales Report (Test #4)
Heavy Load Service (Test #4)
Product Sales Report (Test #4)
Azure vs OCI

@Revelation_Tech
Results – Number of Errors
• All errors were:
Non HTTP response code: javax.net.ssl.SSLHandshakeException/Non
HTTP response message: Remote host terminated the handshake
Non HTTP response code: org.apache.http.NoHttpResponseException/Non
HTTP response message: pochostX.revtech.poc:7002 failed to respond
0
20
40
60
Errors
(lower is better)
Number of Transactions
(total)
Error Rate
AWS 12,000 0.25%
OCI 12,000 0.03%
IBM 12,000 1.18%
GCP 12,000 0.33%
Azure 12,000 0.62%
▪ OCI experienced considerably
less errors, indicating its ability
to handle application load
better (but spread out
longer?).
▪ IBM ranks worst in terms of its
ability to handle load relative
to the other providers.
Thoughts on Error Rates

@Revelation_Tech
▪ Generally comparable
response times between
AWS, IBM, and GCP.
▪ Cannot explain the identical
Average Response Times
between the AMD and Intel
tests.
▪ Unclear why higher Average
Response Times in general
in 2 of the 3 tests.
AWS, IBM, and GCP.
AWS, IBM, and GCP.
▪ Slight lead over all other
providers.
throughput between AWS,
IBM, GCP, and Azure.
▪ Generally lower TPS than
the others on average.
▪ Longer times to completion
than the others (but also
least errors).
▪ Not as bad as IBM, not as
good as OCI.
▪ Most tests ran error-free.
▪ Had the lowest error rate of
all providers (0.03%).
▪ Possibly tied to throughput.
▪ Highest error rate of all
providers (1.18%).
good as OCI.
good as OCI.
Average
Response
Times
Throughput
(TPS)
Number
of Errors

✓ ✓ ✓ ✓
✓ ✓ ✓ ✓
✓ 
? ? ?


@Revelation_Tech
DATABASE STRESS TESTS

@Revelation_Tech
Testing Tool
• SwingBench 2.6
– http://www.dominicgiles.com/swingbench.html
• “Swingbench is a free load generator (and benchmarks) designed to stress test an Oracle
database (11g,12c)”

@Revelation_Tech
Test
• Oracle Database 19.0.0.0
• Stress Test:
– Number of Tests: 4
– Users: 100
– Duration: 48 minutes
– Load Ratio: Select (40%)
Insert (15%)
Update (30%)
Delete (10%)
– Database Setup: Single node (no RAC)
File system datafiles (no ASM)
All testing against CDBROOT
Default DBCA configuration

@Revelation_Tech
Results – Throughput
• AWS dips a bit compared to 2019 tests
• OCI results are difficult to interpret as it pertains to the different CPU models
• IBM consistent, despite improved CPU model
• GCP shows minor gains compared to 2019 tests
• Azure demonstrates notable improvements, but still lags behind others
0
20,000,000
40,000,000
60,000,000
80,000,000
Total Completed Transactions
(higher is better)
AMD
▪ Azure benefited from change
in instance type, but still lags
behind the others.
▪ Last test (Test #6) on OCI is
lower than expected, indicating
potential backend
infrastructure loads.
Interpreting Throughout Results

@Revelation_Tech
Results – Throughput Breakdown
0
10,000,000
20,000,000
30,000,000
AWS Oracle IBM Google Azure
Total SELECT Transactions
(higher is better)
0
5,000,000
10,000,000
15,000,000
Total INSERT Transactions
(higher is better)
0
10,000,000
20,000,000
30,000,000
Total UPDATE Transactions
(higher is better)
0
2,000,000
4,000,000
6,000,000
8,000,000
Total DELETE Transactions
(higher is better)
▪ AWS, OCI, and GCP are
generally comparable.
▪ IBM slightly lower.
▪ Azure, despite improving,
still underwhelms.

@Revelation_Tech
Results – Average Response Times
0.0
1.0
Average SELECT Response Times in
Seconds
(lower is better)
0
10
20
30
40
50
Average INSERT Response Times in
Seconds
(lower is better)
0
10
20
30
Average UPDATE Response Times in
Seconds
(lower is better)
0
10
20
30
Average DELETE Response Times in
Seconds
(lower is better)

@Revelation_Tech
Results – Number of Errors
• Cause of errors are unknown
• Errors are spread across the DML operations
• No conclusions ascertained from findings
Total Number of
Transactions
Total Number of
Errors
AWS 183,743,512 100
OCI 182,586,407 98
IBM 154,872,575 0
GCP 217,700,922 0
Azure 129,590,715 96

@Revelation_Tech
▪ 183 million transactions.
throughput to OCI.
▪ Less throughput compared
to 2019 tests, despite
identical specifications.
▪ Freezes towards end of test
(see next slide).
throughput to AWS.
▪ Unclear why Intel
underperformed compared
to AMD.
▪ Ranked 4th place.
▪ Results identical to 2019,
despite improved CPU
model.
▪ Highest number of
completed transactions.
OCI, and GCP.
▪ More throughput compared
to 2019 tests.
▪ Lowest number of
completed transactions.
average response times
between AWS, OCI, IBM,
and GCP.
▪ Best SELECT response times.
and GCP.
▪ Worst SELECT response
times.
and GCP.
and GCP.
▪ Much better than 2019
results.
▪ Still lags between all others.
▪ Small, but negligible.
▪ Spread across DML
operations.
▪ Small, but negligible.
operations.
▪ No errors. ▪ No errors. ▪ Small, but negligible.
operations.
Throughput
Average
Response
Times
Number
of Errors
?
✓ ✓ ✓ ✓ ✓
✓ ✓
 
✓ ? ✓ ✓ 

@Revelation_Tech
Point-In-Time Performance Captures
Transactions Per Minute
100 Users Logged On
25 Minute Mark
Transactions Per Minute
100 Users Logged On
45 Minute Mark
▪ Screenshots captured “best”
values within that time
period.
▪ AWS begins tests incredibly
fast, but shockingly and
consistently comes to a
standstill the last ¼ of the
test.
▪ AWS TPS may be tied to
limits on service thresholds.

@Revelation_Tech
Final Thoughts
What impressed us…
▪ OCI’s AMD model is 53% lower in cost, yet
performed near the top.
▪ GCP demonstrated overall best performance with
the highest throughput, comparable response
times, and zero errors.
What disappointed us…
▪ AWS comes to a complete halt in the last quarter of
the test.
▪ Despite notable improvements compared to 2019,
Azure continues to underperform in database
related operations relative to the other providers.
▪ AWS experienced a small dip compared to 2019.
✓ 

@Revelation_Tech
THE VERDICT

@Revelation_Tech
Cloud Service Provider Comparison
• Chart ranked on each metric, and provides weighted and unweighted
totals
• Support, service offerings, and other factors not included in these tests
are not considered
• Highest weights given to: cost, CPU performance, application response,
database throughput, and database errors
Host Application Database Rank
Cost
CPU
Mark
CPU Memory I/O
App
Response
App Errors
DB
Response
DB
Throughput
DB Errors Unweighted Weighted
AWS 3 2 4 5 1 4 4 5 4 4 36 25
OCI 5 4 2 5 5 2 5 4 4 4 40 24
IBM 4 5 5 5 1 4 1 4 3 5 36 24
GCP 3 2 3 5 1 3 3 5 5 5 36 26
Azure 3 1 4 5 1 5 2 3 2 4 30 22
Weight 1.00 0.05 1.00 0.05 0.90 1.00 0.50 0.80 1.00 1.00
▪ AWS dips compared to
2019 despite identical
hardware specifications.
▪ Azure ranked top in
application response times
but lagged in database
response times.
▪ Azure has highest CPU
mark, but is not reflected in
the performance tests.
▪ OCI reported 20% higher
performance on CPU tests
on Intel vs. AMD, but
performed similarly on
application and database
tests.
Inconsistent Results

@Revelation_Tech
• Only 7 providers covered, those that
qualify as “hyperscale cloud providers”
• Cloud lock-in is real
• There is a strong case for stating that
multi-cloud is the wrong answer
Gartner Magic Quadrant for Cloud Infrastructure & Platform Services
https://www.gartner.com/en/documents/3989743/magic-quadrant-for-cloud-infrastructure-and-platform-ser
https://www.theregister.com/2020/09/09/gartner_on_cloud_aws_fails/

@Revelation_Tech
Gartner Magic Quadrant for Cloud Infrastructure & Platform Services
◦ “Very strong performer in most Gartner use cases”
◦ “Customers continue to believe incorrectly that AWS reduces prices broadly; however, the decreases are often not universally
applied across all services”
◦ Storage “has not experienced a price reduction since 2014” despite greatly reduced prices for storage components
◦ AWS will continue to fail to “reduce prices broadly”
◦ “Thoughtfully architected, hyperscale cloud architecture”
◦ Ideal for Oracle’s E-Business Suite (EBS)
◦ Microsoft Partnership “can overcome shortfalls in Oracle’s cloud products via Azure services”
◦ Best suited for lift-and-shift and hybrid scenarios
◦ Benefits from Red Hat Technology
◦ “10 years late to market”
◦ “Negligible worldwide market share in the application PaaS category”
◦ “Diminishing mind share among developers who perceive IBM as a provider of legacy technologies”
◦ Strong for “every use case”
◦ “Closed a number of critical capability gaps between GCP and Azure”
◦ “Number of GCP outages during the last year, with devastating impact on customers”
◦ “Strong in all use cases”
◦ Particularly good for “extended cloud and edge” and for “Microsoft-centric organizations”
◦ “The lowest ratio of availability zones to regions of any vendor in this Magic Quadrant”
◦ “Does not provide any form of guaranteed capacity to customers”
https://www.gartner.com/en/documents/3989743/magic-quadrant-for-cloud-infrastructure-and-platform-ser
https://www.theregister.com/2020/09/09/gartner_on_cloud_aws_fails/

@Revelation_Tech

Compute Cloud Performance Showdown: 18 Months Later (OCI, AWS, IBM Cloud, GCP, Azure)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Compute Cloud Performance Showdown: 18 Months Later (OCI, AWS, IBM Cloud, GCP, Azure)

Similar to Compute Cloud Performance Showdown: 18 Months Later (OCI, AWS, IBM Cloud, GCP, Azure) (20)

More from Revelation Technologies

More from Revelation Technologies (20)

Recently uploaded

Recently uploaded (20)

Compute Cloud Performance Showdown: 18 Months Later (OCI, AWS, IBM Cloud, GCP, Azure)