I am running esxi 6.5 with the latest patches and vmware tools 10.1.5
I am having very inconsistent performance issues with both of my hosts. Basically the windows 2016/windows 10 guests are sluggish at times. nothing will load and the os is basically unresponsive when interacting with the gui. The issue seems to be stemming from disk performance but I am not 100% certain that this is the cause, it may be a side affect.
What I have noticed is that some vms show a average response time for the disk of about 2000ms. Yet if i check the performance monitor at a host level the disk and datastores are all showing sub 1ms response time. I am not able to explain the inconsistencies there.
I have a local ssd datastore on each host as well as a rather fast nvme iscsi san that is connected via 100gb mellanox connectx4 cards. I see the issue with both hosts and both datastores. The issue seems to be worse now with the most recent patches and vmware tools drivers. I am using vmxnet3 network cards and paravirtual scsi controllers on all vms.
I have run disk benchmarks on the vms and the resutls vary. I have already seen it where i run a disk benchmark on a guest, get horrible results, vmotion it to the other host, and benchmarks to the san are fine, and then i vmotion the guest back to the original host and the results are fine the second time I run it.
here is an example of a bad test, the reads are terrible:
-----------------------------------------------------------------------
CrystalDiskMark 5.2.0 x64 (C) 2007-2016 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes
Sequential Read (Q= 32,T= 2) : 0.655 MB/s
Sequential Write (Q= 32,T= 2) : 5384.173 MB/s
Random Read 4KiB (Q= 32,T= 2) : 0.026 MB/s [ 6.3 IOPS]
Random Write 4KiB (Q= 32,T= 2) : 617.822 MB/s [150835.4 IOPS]
Sequential Read (T= 1) : 2.306 MB/s
Sequential Write (T= 1) : 1907.004 MB/s
Random Read 4KiB (Q= 1,T= 1) : 53.942 MB/s [ 13169.4 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 52.104 MB/s [ 12720.7 IOPS]
Test : 50 MiB [C: 5.2% (15.6/299.5 GiB)] (x1) [Interval=5 sec]
Date : 2017/03/25 20:29:18
OS : Windows 10 Enterprise [10.0 Build 14393] (x64)
a few seconds later on the same setup i get perfectly fine results:
-----------------------------------------------------------------------
CrystalDiskMark 5.2.0 x64 (C) 2007-2016 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes
Sequential Read (Q= 32,T= 2) : 6655.386 MB/s
Sequential Write (Q= 32,T= 2) : 5654.851 MB/s
Random Read 4KiB (Q= 32,T= 2) : 695.193 MB/s [169724.9 IOPS]
Random Write 4KiB (Q= 32,T= 2) : 609.216 MB/s [148734.4 IOPS]
Sequential Read (T= 1) : 1810.393 MB/s
Sequential Write (T= 1) : 1626.112 MB/s
Random Read 4KiB (Q= 1,T= 1) : 53.266 MB/s [ 13004.4 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 54.289 MB/s [ 13254.2 IOPS]
Test : 50 MiB [C: 5.2% (15.7/299.5 GiB)] (x1) [Interval=5 sec]
Date : 2017/03/25 20:32:21
OS : Windows 10 Enterprise [10.0 Build 14393] (x64)