The goal of this test plan is to test SPECTRE and MELTDOWN performance impact on Intel CPU. We will run CPU intensive workloads in Virtual Machine(s) running on non-patched and patched ESXi host and observe performance impact.
We will test impact on network, storage and memory performance because these I/O intensive workloads requires CPU caching which is impacted by vulnerabilities remediation.
Qualification of performance is very specific and hard subject. The performance impact varies across different hardware and software configurations. However, performed tests are very well described in this document so the reader can understand all conditions of the test and observed results. The reader can also perform tests on his specific hardware and software configurations.
Scaling API-first – The story of a global engineering organization
Spectre meltdown performance_tests - v0.3
1. David Pasek – dpasek@vmware.com
VMWARE - TAM Program
SPECTRE AND MELTDOWN
PERFORMANCE IMPACT TESTS
March 14, 2018, Document version: 0.3
2. Purpose of this test plan
The goal of this test plan is to test SPECTRE and MELTDOWN performance impact on Intel
CPU. We will run CPU intensive workloads in Virtual Machine(s) running on non-patched
and patched ESXi host and observe performance impact.
We will test impact on network, storage and memory performance because these I/O intensive
workloads requires CPU caching which is impacted by vulnerabilities remediation.
Qualification of performance is very specific and hard subject. The performance impact varies
across different hardware and software configurations. However, performed tests are very
well described in this document so the reader can understand all conditions of the test and
observed results. The reader can also perform tests on his specific hardware and software
configurations.
3. Specifications
Hypervisors (ESXi) Hardware and Software Specifications
ESX01
ESX01 - without any Spectre Patches
• Intel NUC D54250WYKH
• 1 x CPU i5-4250U @ 1.30 GHz
• 1 x 2 Cores / 4 logical CPU with Hyper-threading
• ESXi 6.5 Update 1 (Build 5969303) - 2017-07-27
ESX02
ESX02 - with Spectre Patches for Hypervisor-Specific Remediation
• Intel NUC D54250WYKH
• 1 x CPU i5-4250U @ 1.30 GHz
• 1 x 2 Cores / 4 logical CPU with Hyper-threading
• ESXi 6.5 Patch 02 (Build 7388607) - 2017-12-19
ESX03
ESX03 - with Spectre Patches for Hypervisor-Specific and Hypervisor-Assisted
Guest Remediation
• Intel NUC D54250WYKH
• 1 x CPU i5-4250U @ 1.30 GHz
• 1 x 2 Cores / 4 logical CPU with Hyper-threading
• ESXi 6.5 Patch 02 (Build 7526125) - 2018-01-09
4. VM Hardware and Software Specifications
MS-Windows
MS-VM01
• 4 vCPU
• 4 GB RAM
• VM Hardware 11
• NIC (VMXNET3) MTU 1500
• 1x SCSI Controller – LSI Logic SAS
▪ 40 GB Disk (OS) – Thick, eager-zeroed
• 1x SCSI Controller – VMware Paravirtual
▪ 5 GB Disk (DATA) – Thick, eager-zeroed
• OS – MS Windows 2012 R2 – without Spectre/Meltdown updates
• IP address: 192.168.5.36
• Software
▪ CPU-Z
▪ IOmeter
▪ nuttcp
▪ Redis 3.0.503
MS-VM02
• 4 vCPU
• 4 GB RAM
• VM Hardware 11
• NIC (VMXNET3) MTU 1500
• 1x SCSI Controller – LSI Logic SAS
▪ 40 GB Disk (OS) – Thick, eager-zeroed
• 1x SCSI Controller – VMware Paravirtual
▪ 5 GB Disk (DATA) – Thick, eager-zeroed
• OS – MS Windows 2012 R2 – without Spectre/Meltdown updates
• IP address: 192.168.5.37
• Software
▪ CPU-Z
▪ IOmeter
▪ nuttcp
▪ Redis 3.0.503
MS-VM11
• 4 vCPU
• 4 GB RAM
• VM Hardware 11
• NIC (VMXNET3) MTU 1500
• 1x SCSI Controller – LSI Logic SAS
▪ 40 GB Disk (OS) – Thick, eager-zeroed
• 1x SCSI Controller – VMware Paravirtual
▪ 5 GB Disk (DATA) – Thick, eager-zeroed
5. • OS – MS Windows 2012 R2 – with Spectre/Meltdown updates (MS KB
4056898)
• IP address: 192.168.5.46
• Software
▪ CPU-Z
▪ IOmeter
▪ nuttcp
▪ Redis 3.0.503
MS-VM12
• 4 vCPU
• 4 GB RAM
• VM Hardware 11
• NIC (VMXNET3) MTU 1500
• 1x SCSI Controller – LSI Logic SAS
▪ 40 GB Disk (OS) – Thick, eager-zeroed
• 1x SCSI Controller – VMware Paravirtual
▪ 5 GB Disk (DATA) – Thick, eager-zeroed
• OS – MS Windows 2012 R2 – with Spectre/Meltdown updates (MS KB
4056898)
• IP address: 192.168.5.47
• Software
▪ CPU-Z
▪ IOmeter
▪ nuttcp
▪ Redis 3.0.503
Linux
LIN-VM01
• 4 vCPU
• 4 GB RAM
• VM Hardware 11
• NIC (VMXNET3) MTU 1500
• 1x SCSI Controller – LSI Logic SAS
▪ 40 GB Disk (OS) – Thick, eager-zeroed
• OS – Centos 7 – without Spectre/Meltdown updates
▪ Linux 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux
• IP address: 192.168.5.31
• Software
▪ Redis
▪ Nuttcp
▪ Iftop
▪ Bc
6. LIN-VM02
• 4 vCPU
• 4 GB RAM
• VM Hardware 11
• NIC (VMXNET3) MTU 1500
• 1x SCSI Controller – LSI Logic SAS
▪ 40 GB Disk (OS) – Thick, eager-zeroed
• 1x SCSI Controller – VMware Paravirtual
▪ 5 GB Disk (DATA) – Thick, eager-zeroed
• OS – Centos 7 – without Spectre/Meltdown updates
▪ Linux 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux
• IP address: 192.168.5.32
• Software
▪ Redis
▪ Nuttcp
▪ Iftop
▪ Bc
LIN-VM11
• 4 vCPU
• 4 GB RAM
• VM Hardware 11
• NIC (VMXNET3) MTU 1500
• 1x SCSI Controller – LSI Logic SAS
▪ 40 GB Disk (OS) – Thick, eager-zeroed
• 1x SCSI Controller – VMware Paravirtual
▪ 5 GB Disk (DATA) – Thick, eager-zeroed
• OS – Centos 7 – with Spectre/Meltdown updates
▪ Linux 3.10.0-693.17.1.el7.x86_64 #1 SMP Thu Jan 25 20:13:58 UTC
2018 x86_64 x86_64 x86_64 GNU/Linux
• IP address: 192.168.5.41
• Software
▪ Redis
▪ Nuttcp
▪ Iftop
▪ Bc
LIN-VM12
• 4 vCPU
• 4 GB RAM
• VM Hardware 11
• NIC (VMXNET3) MTU 1500
• 1x SCSI Controller – LSI Logic SAS
▪ 40 GB Disk (OS) – Thick, eager-zeroed
• 1x SCSI Controller – VMware Paravirtual
▪ 5 GB Disk (DATA) – Thick, eager-zeroed
7. • OS – Centos 7 – with Spectre/Meltdown updates
▪ Linux 3.10.0-693.17.1.el7.x86_64 #1 SMP Thu Jan 25 20:13:58 UTC
2018 x86_64 x86_64 x86_64 GNU/Linux
• IP address: 192.168.5.42
• Software
▪ Redis
▪ Nuttcp
▪ Iftop
▪ Bc
8. Performance Testing tools
CPU-Z - https://www.cpuid.com/softwares/cpu-z.html
Download: https://www.cpuid.com/downloads/cpu-z/cpu-z_1.83-en.exe
CPU-Z is a freeware that gathers information on some of the main devices of your system
IOMETER - http://www.iometer.org/
Download: http://www.iometer.org/doc/downloads.html
IOmeter is an I/O subsystem measurement and characterization tool for single and clustered
systems.
NUTTCP
Install:
RedHat 7: yum install --enablerepo=Unsupported_EPEL nuttcp
CentOS 7: yum install epel-release nuttcp
MS-Windows: http://nuttcp.net/nuttcp/latest/binaries/nuttcp-8.1.4.win64.zip
TTCP (Test TCP) as client/server network performance measurement tool.
Usage …
Server part is started by following command
nuttcp -S -N 100
Client part is started by following command
cat /dev/zero | nuttcp -t -s -N 100 czchoapint092
Other nuttcp examples:
Server and Client
nuttcp -r -S -P 5000 -N 20
cat /dev/zero | nuttcp -t -s -N 20 -P 5000 czchoapint094
Larger buffers
nuttcp -r -l 8972 -S -P 5000 -N 20
cat /dev/zero | nuttcp -t -l 8972 -s -N 20 -P 5000 czchoapint094
UDP traffic
nuttcp -r -u -l 8972 -w4m -S -P 5000 -N 20
cat /dev/zero | nuttcp -t -u -l 8972 -w4m -s -N 20 -P 5000 czchoapint094
REDIS - https://redis.io/
Install:
CentOS 7: yum install redis
MS-Windows: https://dingyuliang.me/redis-3-2-install-redis-windows/
Download: https://github.com/MicrosoftArchive/redis/releases/download/win-
3.2.100/Redis-x64-3.2.100.zip
Redis is an open source (BSD licensed), in-memory data structure store, used as a database,
cache and message broker. It supports data structures such as strings, hashes, lists, sets,
sorted sets with range queries, bitmaps, hyperloglogs and geospatial indexes with radius
queries. Redis has built-in replication, Lua scripting, LRU eviction, transactions and different
9. levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic
partitioning with Redis Cluster.
10. Spectre/Meldown OS remediations
ESXi
Use VMware Update Manager and patches based on VMSA-2018-02 and VMSA-2018-04.
MS-Windows
To protect MS-Windows apply updates available here
http://www.catalog.update.microsoft.com/Search.aspx?q=KB4056898
To enable the fix change Registry Settings
reg add "HKEY_LOCAL_MACHINESYSTEMCurrentControlSetControlSession
ManagerMemory Management" /v FeatureSettingsOverride /t REG_DWORD /d 0 /f
reg add "HKEY_LOCAL_MACHINESYSTEMCurrentControlSetControlSession
ManagerMemory Management" /v FeatureSettingsOverrideMask /t REG_DWORD /d 3 /f
reg add "HKLMSOFTWAREMicrosoftWindows NTCurrentVersionVirtualization" /v
MinVmVersionForCpuBasedMitigations /t REG_SZ /d "1.0" /f
Restart the server for changes to take effect.
Linux / Centos
Use “yum update” and apply the latest OS updates.
Spectre/Meldown remediation checkers
ESXi
ESXi command to get information if microcode is updated …
if [ `vsish -e get /hardware/msr/pcpu/0/addr/0x00000048 2&>1 > /dev/null ;echo $?` -eq 0 ];
then echo -e "nIntel Security Microcode Updatedn";else echo -e "nIntel Security Microcode
NOT Updatedn";fi
MS-Windows
MS-Windows test tool for SPECTRE/MELTDOWN remediation
Installation
• Article: https://support.microsoft.com/en-us/help/4073119/protect-against-
speculative-execution-side-channel-vulnerabilities-in
• PowerShell 5.0 is required
11. Install-Module SpeculationControl
Vulnarability Check (PowerShell command)
Set-ExecutionPolicy RemoteSigned -Scope Currentuser
Get-SpeculationControlSettings
Linux / Centos
Linux test tool for SPECTRE/MELTDOWN remediation
Installation
• Blog: https://www.cyberciti.biz/faq/check-linux-server-for-spectre-meltdown-
vulnerability/
• Tool:
cd /root
wget –O spectre-meltdown-checker.sh
https://raw.githubusercontent.com/speed47/spectre-meltdown-checker/master/spectre-
meltdown-checker.sh
chmod 755 ./spectre-meltdown-checker.sh
Vulnarability Check (Shell command)
/root/spectre-meltdown-checker.sh
12. Spectre/Meldown remediation status of VMs on ESXi hosts
MS-Windows
MS-VM01 on ESX01
VM Guest OS – MS Windows 2012 R2 – without Spectre/Meltdown updates
ESXi 6.5 Update 1 (Build 5969303) - 2017-07-27
13. MS-VM01 on ESX02
VM Guest OS – MS Windows 2012 R2 – without Spectre/Meltdown updates
ESXi 6.5 Patch 02 (Build 7388607) - 2017-12-19
14. MS-VM01 on ESX03
VM Guest OS – MS Windows 2012 R2 – without Spectre/Meltdown updates
ESXi 6.5 Patch 02 (Build 7526125) - 2018-01-09
15. MS-VM02 on ESX01
VM Guest OS – MS Windows 2012 R2 – without Spectre/Meltdown updates
ESXi 6.5 Update 1 (Build 5969303) - 2017-07-27
16. MS-VM02 on ESX02
VM Guest OS – MS Windows 2012 R2 – without Spectre/Meltdown updates
ESXi 6.5 Patch 02 (Build 7388607) - 2017-12-19
17. MS-VM02 on ESX03
VM Guest OS – MS Windows 2012 R2 – without Spectre/Meltdown updates
ESXi 6.5 Patch 02 (Build 7526125) - 2018-01-09
18. MS-VM11 on ESX01
VM Guest OS – MS Windows 2012 R2 – with Spectre/Meltdown updates (MS KB 4056898)
ESXi 6.5 Update 1 (Build 5969303) - 2017-07-27
19. MS-VM11 on ESX02
VM Guest OS – MS Windows 2012 R2 – with Spectre/Meltdown updates (MS KB 4056898)
ESXi 6.5 Patch 02 (Build 7388607) - 2017-12-19
20. MS-VM11 on ESX03
VM Guest OS – MS Windows 2012 R2 – with Spectre/Meltdown updates (MS KB 4056898)
ESXi 6.5 Patch 02 (Build 7526125) - 2018-01-09
21. MS-VM12 on ESX01
VM Guest OS – MS Windows 2012 R2 – with Spectre/Meltdown updates (MS KB 4056898)
ESXi 6.5 Update 1 (Build 5969303) - 2017-07-27
22. MS-VM12 on ESX02
VM Guest OS – MS Windows 2012 R2 – with Spectre/Meltdown updates (MS KB 4056898)
ESXi 6.5 Patch 02 (Build 7388607) - 2017-12-19
23. MS-VM12 on ESX03
VM Guest OS – MS Windows 2012 R2 – with Spectre/Meltdown updates (MS KB 4056898)
ESXi 6.5 Patch 02 (Build 7526125) - 2018-01-09
24. Linux / Centos
LIN-VM01 on ESX01
VM Guest OS – Centos 7 – without Spectre/Meltdown updates (Linux 3.10.0-
514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64
GNU/Linux)
ESXi 6.5 Update 1 (Build 5969303) - 2017-07-27
25. LIN-VM01 on ESX02
VM Guest OS – Centos 7 – without Spectre/Meltdown updates (Linux 3.10.0-
514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64
GNU/Linux)
ESXi 6.5 Patch 02 (Build 7388607) - 2017-12-19
26. LIN-VM01 on ESX03
VM Guest OS – Centos 7 – without Spectre/Meltdown updates (Linux 3.10.0-
514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64
GNU/Linux)
ESXi 6.5 Patch 02 (Build 7526125) - 2018-01-09
27. LIN-VM02 on ESX01
VM Guest OS – Centos 7 – without Spectre/Meltdown updates (Linux 3.10.0-
514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64
GNU/Linux)
ESXi 6.5 Update 1 (Build 5969303) - 2017-07-27
28. LIN-VM02 on ESX02
VM Guest OS – Centos 7 – without Spectre/Meltdown updates (Linux 3.10.0-
514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64
GNU/Linux)
ESXi 6.5 Patch 02 (Build 7388607) - 2017-12-19
29. LIN-VM02 on ESX03
VM Guest OS – Centos 7 – without Spectre/Meltdown updates (Linux 3.10.0-
514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64
GNU/Linux)
ESXi 6.5 Patch 02 (Build 7526125) - 2018-01-09
30. LIN-VM11 on ESX01
VM Guest OS – Centos 7 – with Spectre/Meltdown updates (Linux 3.10.0-
693.17.1.el7.x86_64 #1 SMP Thu Jan 25 20:13:58 UTC 2018 x86_64 x86_64 x86_64
GNU/Linux)
ESXi 6.5 Update 1 (Build 5969303) - 2017-07-27
31. LIN-VM11 on ESX02
VM Guest OS – Centos 7 – with Spectre/Meltdown updates (Linux 3.10.0-
693.17.1.el7.x86_64 #1 SMP Thu Jan 25 20:13:58 UTC 2018 x86_64 x86_64 x86_64
GNU/Linux)
ESXi 6.5 Patch 02 (Build 7388607) - 2017-12-19
32. LIN-VM11 on ESX03
VM Guest OS – Centos 7 – with Spectre/Meltdown updates (Linux 3.10.0-
693.17.1.el7.x86_64 #1 SMP Thu Jan 25 20:13:58 UTC 2018 x86_64 x86_64 x86_64
GNU/Linux)
ESXi 6.5 Patch 02 (Build 7526125) - 2018-01-09
33. LIN-VM12 on ESX01
VM Guest OS – Centos 7 – with Spectre/Meltdown updates (Linux 3.10.0-
693.17.1.el7.x86_64 #1 SMP Thu Jan 25 20:13:58 UTC 2018 x86_64 x86_64 x86_64
GNU/Linux)
ESXi 6.5 Update 1 (Build 5969303) - 2017-07-27
34. LIN-VM12 on ESX02
VM Guest OS – Centos 7 – with Spectre/Meltdown updates (Linux 3.10.0-
693.17.1.el7.x86_64 #1 SMP Thu Jan 25 20:13:58 UTC 2018 x86_64 x86_64 x86_64
GNU/Linux)
ESXi 6.5 Patch 02 (Build 7388607) - 2017-12-19
35. LIN-VM12 on ESX03
VM Guest OS – Centos 7 – with Spectre/Meltdown updates (Linux 3.10.0-
693.17.1.el7.x86_64 #1 SMP Thu Jan 25 20:13:58 UTC 2018 x86_64 x86_64 x86_64
GNU/Linux)
ESXi 6.5 Patch 02 (Build 7526125) - 2018-01-09
36. Performance tests
MS Windows OS
CPU performance (Win/CPU-Z) - single VM on top of ESXi host
Verification type Design
Test type Performance
Tested area CPU
Test name CPU performance (Win/CPU-Z) of VM on top of ESXi host
Test description Verification of Spectre/Meltdown security patches impact on CPU
performance
Tasks
Step 1/ Generate CPU workload leveraging CPU-Z benchmaring tool.
Run CPU-Z on MS-VM
Step 2/ Note CPU performance (CPU-Z benchmark single thread and
multi thread)
Test combinations
• ESXi host without security patches – OS without security
patches (MS-VM01 on ESX01)
• ESXi host without security patches – OS with security patches
(MS-VM11 on ESX01)
• ESXi host with Hypervisor-Specific Remediation security
patches – OS without security patches (MS-VM01 on ESX02)
• ESXi host with Hypervisor-Specific Remediation security
patches – OS with security patches (MS-VM11 on ESX02)
• ESXi host with Hypervisor-Specific and Hypervisor-Assisted
Guest Remediation security patches – OS with security patches
(MS-VM11 on ESX03.)
Compare results and quantify impact.
Expected results Lower CPU performance on systems with security patches.
Test tools: CPU-Z
Test result: passed
Test notes:
37. Test results
ESXi host without
security patches
ESX01
ESXi host with
Hypervisor-Specific
Remediation
security patches
ESX02
ESXi host with
Hypervisor-Specific
and Hypervisor-
Assisted
Guest Remediation
security patches
ESX03
OS without
security patches
MS-VM01
Single Thread:
233.9
235.4
237.1
236.7
236.7
AVG = 236.3
Multi Thread:
624.3
624.5
624.2
621.8
624.9
AVG = 624.3
Single Thread:
236.0
233.8
233.3
233.4
233.9
AVG = 233.7
Multi Thread:
616.6
623.4
622.4
624.6
624.1
AVG = 623.3
Single Thread:
233.8
253.6
233.0
234.6
235.1
AVG = 234.5
Multi Thread:
623.7
623.3
624.5
625.0
623.8
AVG = 624
OS with security
patches
MS-VM11
Single Thread:
234.5
235.3
232.0
225.8
236.5
AVG = 231.9
Multi Thread:
622.2
619.1
621.0
612.8
621.2
AVG = 620.4
Single Thread:
232.1
234.3
233.4
234.7
234.1
AVG = 233.9
Multi Thread:
622.2
623.6
621.0
620.9
622.3
AVG = 621.8
Single Thread:
208.1
207.4
202.9
206.1
209.7
AVG = 207.2
Multi Thread:
604.7
597.5
602.3
609.5
610.6
AVG = 605.5
38. Storage performance (Win/IOmeter) – Single VM storage performance to local disk
Verification type Design
Test type Performance
Tested area Storage
Test name Storage performance (Win/IOmeter) – single VM storage
performance to local disk
Test description Verification of CPU performance impact to storage performance
Tasks
Step 1/ Run IOmeter GUI on MS-VM01.
Step 2/ Run disk IO testing tools (VM01 with IOmeter GUI and
dynamo) and generate load to disk on shared storage.
I/O workload patterns for tests
• 512B, 100% Random, 50% Write
• 64kB, 100% Random, 50% Write
Multi threading configurations
• 4 Workers / 1 Outstanding IO
Disk Size 10GB (20000000 sectors)
Step 3/ Note storage performance (I/O per second = IOPS), data
throughput(MB/s), response time (ms) and CPU load
Test combinations
• ESXi host without security patches – OS without security
patches (MS-VM01 and MS-VM02 on ESX01)
• ESXi host without security patches – OS with security patches
(MS-VM11 and MS-VM12 on ESX01)
• ESXi host with Hypervisor-Specific Remediation security
patches – OS without security patches (MS-VM01 and MS-
VM02 on ESX02)
• ESXi host with Hypervisor-Specific Remediation security
patches – OS with security patches (MS-VM11 and MS-VM12
on ESX02)
• ESXi host with Hypervisor-Specific and Hypervisor-Assisted
Guest Remediation security patches – OS with security patches
(MS-VM11 and MS-VM12 on ESX03.)
Compare results and quantify impact.
Expected results Lower storage performance on systems with security patches.
Test tools: IOmeter
Test result: passed
Test notes:
42. Network performance (Win/IOmeter) between two VMs within the same ESXi host
Verification type Design
Test type Performance
Tested area Network
Test name Network performance (Win/IOmeter) between two VMs within the
same ESXi host
Test description Verification of CPU performance impact to network performance
Tasks
Step 1/ Run IOmeter GUI on MS-VM01.
Step 2/ Remove all storage workers
Step 3/ Run IOmeter dynamo on MS-VM02 connected to IOmeter host
<hostname of VM01> … dynamo.exe –i MS-VM01 –m MS-VM02
Step 4/ Create 8 network workers. Assign specification I/O Size 512B,
100% Read to all network workers. Set test duration 30 seconds.
Step 5/ Generate network workload between two MS-VM’s on the
same ESXi host
Step 6/ Note network performance (packets per second), throughput
(MB/s), Response Time (ms) and CPU load (%)
Test combinations
• ESXi host without security patches – OS without security
patches (MS-VM01 and MS-VM02 on ESX01)
• ESXi host without security patches – OS with security patches
(MS-VM11 and MS-VM12 on ESX01)
• ESXi host with Hypervisor-Specific Remediation security
patches – OS without security patches (MS-VM01 and MS-
VM02 on ESX02)
• ESXi host with Hypervisor-Specific Remediation security
patches – OS with security patches (MS-VM11 and MS-VM12
on ESX02)
• ESXi host with Hypervisor-Specific and Hypervisor-Assisted
Guest Remediation security patches – OS with security patches
(MS-VM11 and MS-VM12 on ESX03.)
Compare results and quantify impact.
Expected results Lower network performance on systems with security patches.
Test tools: IOmeter
Test result: passed
Test notes:
45. In-Memory database performance (Win/Redis) - single VM on top of ESXi host
Verification type Design
Test type Performance
Tested area Database
Test name Database performance from VM to In-Memory DB (Redis)
Test description Verification of CPU performance impact to in-memory database
performance
Tasks
Step 1/ Install and Run Redis DB on WIN-VM01
Step 2/ Run redis-benchmark
redis-benchmark -t get,set –n 1000000 -c 8
Step 3/ Note DB performance (transactions per second) and CPU load
(%)
Test combinations
• ESXi host without security patches – OS without security
patches (VM01 on ESX01)
• ESXi host without security patches – OS with security patches
(VM11 on ESX01)
• ESXi host with Hypervisor-Specific Remediation security
patches – OS without security patches (VM01 on ESX02)
• ESXi host with Hypervisor-Specific Remediation security
patches – OS with security patches (VM11 on ESX02)
• ESXi host with Hypervisor-Specific and Hypervisor-Assisted
Guest Remediation security patches – OS with security patches
(VM11 on ESX03.)
Compare results and quantify impact.
Expected results Lower memory performance on systems with security patches.
Test tools: RedisDB
Test result: passed
Test notes:
47. Linux OS
Network performance (Linux/NUTTCP) between two VMs within the same ESXi host
Verification type Design
Test type Performance
Tested area Network
Test name Network performance (Linux/NUTTCP) between two VMs within
the same ESXi host
Test description Verification of CPU performance impact to network performance
Tasks
Step 1/ Run
nuttcp -r -S -P 5501
nuttcp -r -S -P 5502
nuttcp -r -S -P 5503
nuttcp -r -S -P 5504
nuttcp -r -S -P 5505
nuttcp -r -S -P 5506
nuttcp -r -S -P 5507
nuttcp -r -S -P 5508
on LIN-VM01.
Step 2/ Run
iftop -F 192.168.4.32/32
on LIN-VM01 to monitor traffic
Step 2/ Change IP address bellow to VM01 and run script (/tmp/run.sh)
#!/bin/bash
PORT_START=5501
LOGDIR="/tmp"
IP="192.168.4.31"
for i in `seq 1 8`;
do
echo "Process $i"
port=$(expr $PORT_START + $i - 1)
echo " port $port"
logfile="$LOGDIR/job$i.log"
echo " logfile $logfile"
echo " target IP address $IP"
( /usr/bin/nuttcp -t -b -P $port -T 30 $IP > $logfile ) &
sleep 0.1
done
on LIN-VM02 to generate workload.
Step 4/ Note network throughput (Mbps) of each process and calculate
sum.
SHOW RESULTS: cat /tmp/job*
SUM: cat /tmp/job* | cut -c29-38 | paste -s -d+ | bc
48. Test combinations
• ESXi host without security patches – OS without security
patches (LIN-VM01 and LIN-VM02 on ESX01)
• ESXi host without security patches – OS with security patches
(LIN-VM11 and LIN-VM12 on ESX01)
• ESXi host with Hypervisor-Specific Remediation security
patches – OS without security patches (LIN-VM01 and LIN-
VM02 on ESX02)
• ESXi host with Hypervisor-Specific Remediation security
patches – OS with security patches (LIN-VM11 and LIN-VM12
on ESX02)
• ESXi host with Hypervisor-Specific and Hypervisor-Assisted
Guest Remediation security patches – OS with security patches
(LIN-VM11 and LIN-VM12 on ESX03.)
Compare results and quantify impact.
Expected results Lower network performance on systems with security patches.
Test tools: IOmeter
Test result: passed
Test notes:
49. Test results
ESXi host without
security patches
ESX01
ESXi host with
Hypervisor-Specific
Remediation
security patches
ESX02
ESXi host with
Hypervisor-Specific
and Hypervisor-
Assisted
Guest Remediation
security patches
ESX03
OS without
security patches
LIN-VM01
LIN-VM02
Mbps:
10497.4184
10625.7293
10290.1794
10048.5660
9479.2741
AVG: 10278.7213
Mbps:
10421.9657
10157.4198
10673.1855
10052.0098
10610.0493
AVG: 10396.4783
Mbps:
9338.4298
9395.8587
10205.4507
9680.9801
8942.6938
AVG: 9471.7562
OS with security
patches
LIN-VM11
LIN-VM12
Mbps:
9592.9527
9761.0624
10655.8847
10283.9328
9630.8711
AVG: 9891.9554
Mbps:
10626.2253
10390.4692
9941.6684
10011.0204
10373.6655
AVG: 10258.2286
Mbps:
7794.9779
8703.3505
8383.7530
7298.3641
7165.8537
AVG: 7825.6983
50. In-Memory database performance (Linux/Redis) - single VM on top of ESXi host
Verification type Design
Test type Performance
Tested area Database
Test name Database performance from VM to In-Memory DB (Redis)
Test description Verification of CPU performance impact to in-memory database
performance
Tasks
Step 1/ Install and Run Redis DB on LIN-VM01 (Linux or FreeBSD
OS are required)
Step 2/ Run redis-benchmark
redis-benchmark -t get,set –n 1000000 -c 8
Step 3/ Note DB performance (transactions per second) and CPU load
(%)
Test combinations
• ESXi host without security patches – OS without security
patches (VM01 on ESX01)
• ESXi host without security patches – OS with security patches
(VM11 on ESX01)
• ESXi host with Hypervisor-Specific Remediation security
patches – OS without security patches (VM01 on ESX02)
• ESXi host with Hypervisor-Specific Remediation security
patches – OS with security patches (VM11 on ESX02)
• ESXi host with Hypervisor-Specific and Hypervisor-Assisted
Guest Remediation security patches – OS with security patches
(VM11 on ESX03.)
Compare results and quantify impact.
Expected results Lower memory performance on systems with security patches.
Test tools: RedisDB
Test result: passed
Test notes:
52. Findings
CPU Performance on MS Windows
ESXi host without
security patches
ESXi host with
Hypervisor-Specific
Remediation
security patches
ESXi host with
Hypervisor-Specific
and Hypervisor-
Assisted
Guest Remediation
security patches
MS Windows 2012
R2 without security
patches
Single Thread:
236.3
Multi Thread:
624.3
Single Thread:
233.7
Multi Thread:
623.3
Single Thread:
234.5
Multi Thread:
624
MS Windows 2012
R2 with security
patches
Single Thread:
231.9
Multi Thread:
620.4
Single Thread:
233.9
Multi Thread:
621.8
Single Thread:
207.2
Multi Thread:
605.5
Secured system performance impact
CPU Single Thread ~ -12%
<< this is probably because ESXi hardware has just a 2 CPU cores (Intel NUC) and
ESXi VMkernel is probably using more CPU resources on CPU core 0. Such
performance impact was not observed on enterprise server hardware where the
impact in single CPU thread was negligible.
CPU Multi Thread ~ -3%
Storage performance (Win/IOmeter) – I/O size 512B
ESXi host without
security patches
ESXi host with
Hypervisor-Specific
Remediation
security patches
ESXi host with
Hypervisor-Specific
and Hypervisor-
Assisted
Guest Remediation
security patches
MS Windows 2012
R2 without security
patches
20.25 MB/s
39550 IOPS
19.91 MB/s
38887 IOPS
20.13 MB/s
39316 IOPS
MS Windows 2012
R2 with security
patches
19.47 MB/s
38027 IOPS
17.77 MB/s
34707 IOPS
15.63 MB/s
30527 IOPS
Secured system storage performance impact is ~ -23%
53. Storage performance (Win/IOmeter) – I/O size 64kB
ESXi host without
security patches
ESXi host with
Hypervisor-Specific
Remediation
security patches
ESXi host with
Hypervisor-Specific
and Hypervisor-
Assisted
Guest Remediation
security patches
MS Windows 2012
R2 without security
patches
499.36 MB/s
7619 IOPS
498.8 MB/s
7611 IOPS
501.76 MB/s
7656 IOPS
MS Windows 2012
R2 with security
patches
497.82 MB/s
7596 IOPS
499.15 MB/s
7616 IOPS
495.68 MB/s
7563 IOPS
Secured system performance impact is ~ 1% which is negligible. In other words, for
larger I/O size negative performance impact has not been observed.
Network performance (Win/IOmeter) – I/O size 512B
ESXi host without
security patches
ESXi host with
Hypervisor-Specific
Remediation
security patches
ESXi host with
Hypervisor-Specific
and Hypervisor-
Assisted
Guest Remediation
security patches
MS Windows 2012
R2 without security
patches
463.59 MB/s 445.37 MB/s 425.22 MB/s
MS Windows 2012
R2 with security
patches
450.01 MB/s 381.92 MB/s 212.83 MB/s
Secured system network performance impact is ~ -54%
<< Even bigger impact (~ 60%) was observed on enterprise server hardware
54. In-Memory database performance (Win/Redis)
ESXi host without
security patches
ESXi host with
Hypervisor-Specific
Remediation
security patches
ESXi host with
Hypervisor-Specific
and Hypervisor-
Assisted
Guest Remediation
security patches
MS Windows 2012
R2 without security
patches
TPS set,get:
140067, 149696
TPS set,get:
141180, 148153
TPS set,get:
143787, 147371
MS Windows 2012
R2 with security
patches
TPS set,get:
142081, 145618
TPS set,get:
139777, 145926
TPS set,get:
82531, 84119
Secured system memory performance impact is ~ -42%
<< Similar impact (~ 40%) for set transaction but even bigger impact (~ 50%) was
observed for get transaction on enterprise server hardware
Network performance (Linux/NUTTCP)
ESXi host without
security patches
ESXi host with
Hypervisor-Specific
Remediation
security patches
ESXi host with
Hypervisor-Specific
and Hypervisor-
Assisted
Guest Remediation
security patches
CentOS 7 without
security patches
10278.72 Mbps 10396.48 Mbps 9471.76 Mbps
CentOS 7 with
security patches
9891.96 Mbps 10258.23 Mbps 7825.7 Mbps
Secured system network performance impact is ~ -24%
<< Less impact (~ 9%) was observed on enterprise server hardware
In-Memory database performance (Linux/Redis)
ESXi host without
security patches
ESXi host with
Hypervisor-Specific
Remediation
security patches
ESXi host with
Hypervisor-Specific
and Hypervisor-
Assisted
Guest Remediation
security patches
CentOS 7 without
security patches
TPS set,get:
156633, 150487
TPS set,get:
153840, 149583
TPS set,get:
153623, 154284
CentOS 7 with
security patches
TPS set,get:
106133, 104803
TPS set,get:
105516, 106492
TPS set,get:
43956, 43843
Secured system memory performance impact is ~ -70%
<< Similar impact was observed on enterprise server hardware
55. Conclusion
Qualification of performance is very specific and hard subject. The performance impact varies
across different hardware and software configurations. However, performed tests are very
well described in this document so the reader can understand all conditions of the test and
observed results. The reader can also perform tests on his specific hardware and software
configurations.
Tests in this document are focused on CPU, Memory, Storage and Network. It is worth to
mention that these tests are synthetic created to test the impact on specific infrastructure
component. Real workloads are usually mix of CPU, Memory, Storage and Network,
therefore the impact is the combination of extreme impacts of these synthetic tests.
The performance impact of VMware ESXi patches
We did not observe performance penalty after application of ESXi patches (Hypervisor-
Specific and Hypervisor-Assisted Guest Remediation security patches). The performance
penalty on CPU, Memory and Storage was observed after application of security patches in
to Guest Operating Systems and CPU Microcode. The only exception are Network
performance tests where we have observed up to 8% performance penalty after application of
ESXi patches even the Guest OS was still unpatched.
The performance impact of GuestOS and CPU Microcode patches
After application of all security remediation for Windows 2012 R2 and ESXi 6.5 we have
observed following performance impacts
• CPU
o ~ 12% negative performance impact on single thread CPU performance
o ~ 3% (negligible) negative performance impact on multi thread CPU
performance
• Memory
o ~ 42% negative performance impact on memory performance
• Storage
o ~ 23% negative performance impact on storage performance with small I/O
size (512B)
o No performance impact on storage performance with 64kB I/O size
• Network
o ~ 54% negative performance impact on network performance impact with
small I/O size (512B)
After application of all security remediation for CentOS 7 and ESXi 6.5 we have observed
following performance impacts
• Memory
o ~ 70% negative performance impact on memory performance
• Network
o ~ 24% negative performance impact on network performance