I finally found some time to run a series of iPerf network performance tests between Windows server 2008 R2, Windows 2012 and a Linux Debian virtual machines. The tests compare bandwidth throughput between vmxnet3, e1000 going through a 1Gbit and a 10Gbit physical network card. I also tested jumbo frames (MTU 9000) vs (MTU 1500) I wanted to see the speed difference between the two.
I instilled iPerf version 2.0.5-3 on both virtual machines. I chose to test TCP window size of 1024k, smaller window size may give you different results. I set up one machine to listen for traffic while another machine is sending traffic trying to saturate the network through 4 parallel connections.
Client (sending traffic): iperf -c 192.168.11.151 -w 1024k -P 4
Server (receiving traffic): iperf -s -w 1024k
Download iPerf (version 2.0.5-3 was used for this test)
https://iperf.fr/
Physical Hardware
———————–
Two Identical Physical Servers:
Dell PowerEdge R720xd
ESXi 6.0 build 2615704
Intel Xeon CPU E5-2620
12CPUs @ 2.099GHz / 98GB RAM
1Gbit NIC – Broadcom BCM5720 Gigabit Ethernet
10Gbit NIC – Intel 10 Gigabit X540-AT2
Virtual Machines:
———————-
Windows 2008 R2 SP1 64-bit (2vCPU, 4GB RAM, HW11)
Windows 2012 R2 64-bit (2vCPU, 4GB RAM, HW11)
Debian 7.2 64-Bit (2vCPU, 4GB RAM, HW11)
What affects the results:
—————————–
-CPU inside guest pinned to 100%, iPerf tests are CPU intensive
-High %RDY in esxtop, turn off any power management on host
-Multiple vCPU VM’s will have different results as Windows uses RSS
-CPU type, Intel vs AMD will have different results
-CPU clock speed/CPU Architecture
-Different window size
-Memory bus speed
-Firewall, Anti-virus, etc…
Each test below is between two virtual machines, each running on a separate physical host. For example, Test#1 below is between two VM’s. VM1 is on Host1 and VM2 is on Host2, both VM’s are configured with E1000 virtual network adapter and the two ESXi hosts are connected together through a 1Gbit physical NIC. Test#2 is the same but the two VM’s are configured with a VMXNET3 virtual network adapter.
Each test below is between two virtual machines, both on the same host. For Test#13, two VM’s running on the same ESXi host, both VM’s are configured with an E1000 and are connected to a portgroup on the same vSwitch without any physical NIC’s attached, this causes the traffic to be internal to to the vSwitch. Test#14 is the same but both VM’s are using a VMXNET3 instead of E1000.
Linux TCP Tuning to allow 1024k window size
=====================================
The default window size for Debian Linux is 256Kbyte, I had to adjust advanced parameters for tcp to allow a window size of 1024k. The below parameters had to be adjusted to tune tcp and allow a larger window size. Another note is that window size doubles for Linux, to allow it to listen for 1024k i had to run “iperf -s -w 512k” and that configured 1024k window.
You can see below that a 1024k window size was requested but it defaults to 256 Kbyte
root@iperf-client:~# iperf -s -w 1024k
————————————————————
Server listening on TCP port 5001
TCP window size: 256 KByte (WARNING: requested 1.00 MByte)
To change the default 256KByte window size in Linux you will need to add some parameters to change the tcp behavior. Add the following parameters to the bottom of this file: “/etc/sysctl.conf”
net.core.wmem_max=12582912
net.core.rmem_max=12582912
net.ipv4.tcp_rmem= 10240 87380 12582912
net.ipv4.tcp_wmem= 10240 87380 12582912
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_no_metrics_save = 1
net.core.netdev_max_backlog = 5000
Run “sysctl -p” to reload the changes
For some reason Linux doubles the window size as seen below
root@iperf-client:~# iperf -s -w 1024k
————————————————————
Server listening on TCP port 5001
TCP window size: 2.00 MByte (WARNING: requested 1.00 MByte)
————————————————————
The fix is to request a 512k window to get a 1.00 MByte window size
root@iperf-client:~# iperf -s -w 512k
————————————————————
Server listening on TCP port 5001
TCP window size: 1.00 MByte (WARNING: requested 512 KByte)
————————————————————
192.168.1.1