More and more companies seem to use IP-hash as there load balancing policy. The main argument seems to be increased bandwidth and better redundancy. Even when the distributed vSwitch is used, most organizations still choose IP-hash over the new load balancing policy “Route based on physical NIC load”. This article compares both load-balancing policies and lists the characteristics, requirements and constraints of both load-balancing policies.

IP-Hash
The main reason for selecting IP-Hash seems to be increased bandwidth as you aggregate multiple uplinks, unfortunately adding more uplinks does not proportionally increase the available bandwidth for the virtual machines.

How IP-Hash works
Based on the source and destination IP address together the VMkernel distributes the load across the available NICs in the vSwitch. The calculation of outbound NIC selection is described in KB article 1007371. To calculate the IP-hash yourself convert both the source and destination IP-addresses to a Hex value and compute the modulo over the number of available uplinks in the team. For example

Virtual Machine 1 opens two connections, one connection to a backup server and one connection server to an application server.

The vSwitch is configured with two uplinks.

Connection 1: VM1 > Backup Server (A4120154 Xor A41201A0 = F4) % 2 = 0
Connection 2: VM1 > Application Server (A4120154 Xor A41201C3 = 97) % 2 = 1

IP-Hash treats each connection between a source and destination IP address as a unique route and the vSwitch will distributed each connection across the available uplinks in the vSwitch. However due to the pNIC to vNIC affiliation, any connection is on a per flow basis. A flow can’t overflow to another uplink; this means that a connection is still limited to the speed of a single physical NIC. A real-world user case for IP-hash would be a backup server which requires a lot of bandwidth across multiple connections other than that; there are very few workloads that require bandwidth that can’t be satisfied by a single adapter.

Complexity –In order for IP-hash to function correctly additional configuration at the network layer is required:

EtherChannel: IP-hash needs to be configured on the vSwitch if EtherChannel technology is used at the physical switch layer. With EtherChannel the switch will load balance connections over multiple ports in the EtherChannel. Without IP-hash, the VMkernel only expects to receive information on a specific MAC address on a single vNIC. Resulting in some sessions go through to the virtual machine while other sessions will be dropped. When IP-hash is selected, then the VMkernel will accept inbound mac addresses on both active NICs

EtherChannel configuration: As vSphere does not support dynamic link aggregation (LACP), none of the members can be set up to auto-negotiate membership and therefore physical switches have to be configured with static EtherChannel.

Switch configuration: vSphere supports EtherChannel from one switch to the vSwitch. This switch can be a single switch or a stack of individual switches that act as one, but vSphere does not support EtherChannel from two separate – non stacked – switches, when the EtherChannel connect to the same vSwitch.

Additional overhead – For each connection the VMkernel needs to select the appropriate uplink. If a virtual machine is running a front-end application and communicates 95% of its time to the backend database, the IP-Hash calculation is almost pointless. The VMkernel needs to perform the math for every connection and 95% of the connections will use the same uplink because the Algorithm will always result in the same hash.

Utilization-unaware – It is possible that a second virtual machine is assigned to use the same uplink as the virtual machine that is already saturating the link. Let’s use the first example and introduce a new virtual machine VM3. Due to the backup window, VM3 connects to the backup server.

Connection 3: VM3> Backup Server (A4120156 Xor A41201A0 = F6) % 2 = 0

Due to IP-HASH load balancing policy being unaware of utilization it will not rebalance if the uplink is saturated or if virtual machine are added or removed due to power-on or (DRS) migrations. DRS is unaware of network utilization and does not initiate a rebalance if a virtual machine cannot send or receive packets due to physical NIC saturation. In worst-case scenario DRS can migrate virtual machines to other ESX servers, leaving all the virtual machine that are saturating a NIC while the other virtual machines utilizing the other NICs are migrated. Admitted it’s a little bit of a stretch, but being aware of this behavior allows you to see the true beauty of the Load-Based Teaming team policy.

Possible Denial of Service –Due to the pNIC-to-vNIC affiliation per connection a misbehaving virtual machine generating many connections can cause some sort of denial of service on all uplinks on the vSwitch. If this application would connect to a vSwitch with “Port-ID” or “based on physical load” only one uplink would be affected.

Network failover detection Beacon Probing – Beacon probe does not work correctly if EtherChannel is used. ESX broadcast beacon packets out of all uplinks in a team. The physical switch is expected to forward all packets to other ports. In EtherChannel mode, the physical switch will not send the packets because it’s considered as one link. No beacon packets will be received and can interrupt network connections. Cisco switches will report flapping errors. See KB article 1012819.

Route based on physical NIC Load

VMware vSphere 4.1 introduced a new load-balancing policy available on distributed vSwitches. Route based on physical NIC load, also known as Load Based Teaming (LBT) takes the virtual machine network I/O load into account and tries to avoid congestion by dynamically reassigning and balancing the virtual switch port to physical NIC mappings.

How LBT works
Load Based Teaming maps vNICs to pNICs and remaps the vNIC-to-PNIC affiliation if the load exceeds specific thresholds on an uplink.
LBT uses the same initial port assignment as the “originating port id” load balancing policy, resulting in the first vNIC being affiliated to the first pNIC, the second vNIC to the second pNIC, etc. After initial placement, LBT examines both ingress and egress load of each uplink in the team and will adjust the vNIC to pNIC mapping if an uplink is congested. The NIC team load balancer flags a congestion condition if an uplink experiences a mean utilization of 75% or more over a 30-second period.

Complexity – LBT requires standard Access or Trunk ports. LBT does not support EtherChannels. Because LBT is moving flows among the available uplinks of the vSwitch, it may create packets re-ordering. Even though the reshuffling process is not done often (worst case scenario every 30 seconds) it is recommended to enable PortFast or TrunkFast on the switch ports.

Additional overhead – The VMkernel will examine the congestion condition after each time window, this calculation creates a minor overhead opposed to using the static load-balancing policy “originating port-id”.

Utilization aware – vNIC to pNIC mappings will be adjusted if the VMkernel detects congestion on an uplink. In the previous example both VM1 and VM3 shared the same connection due to the IP-hash calculation. Both connections can share the same physical NIC as long as the utilization stays below the threshold. It is likely that both vNICs are mapped to separate physical NICs.
In the next example a third virtual machine is powered up and is mapped to NIC1. Utilization of NIC1 exceeds the mean utilization of 70% over a period of more than 30 seconds. After identifying congestion LBT remaps VM2 to NIC2 to decrease the utilization of NIC1.

Although LBT is not integrated in DRS it can be viewed as complimentary technology next to DRS. When DRS migrates virtual machines onto a host, it is possible that congestion is introduced on a particular physical NIC. Due to vNIC to pNIC mapping based on actual load, LBT actively tries to avoid congestion at physical NIC level and attempts to reallocate virtual machines. By remapping vNiCs to pNICs it will attempt to make as much bandwidth available to the virtual machine, which ultimately benefits the overall performance of the virtual machine.

Recommendations
When using distributed virtual Switches it is recommended to use Load-Based teaming instead of IP-hash. LBT has no additional requirements on the physical network layer, reduces complexity and is able to adjust to fluctuating workloads. Due to the remapping of vNICs to pNICs based on actual load, LBT attempts to allocate as much bandwidth possible where IP-hash just simply distributes connections across the available physical NICs.

IP-Hash versus LBT is a post from: frankdenneman.nl

Just a quick rehash:

Active\Standby
One NIC (vmnic0) is assigned as active to the management\service console portgroup, the second NIC (vmnic1) is configured as standby. The vMotion portgroup is configured with the first NIC (vmnic0) in standby mode and the second NIC as Active (vmnic1).

Active Standby setup management network vSwitch0

Failback
The Failback setting determines if the VMkernel will return the uplink (NIC) to active duty after recovery of a downed link or failed NIC. If the Failback setting is set to Yes the NIC will return to active duty, when Failback is set to No the failed NIC is assigned the Standby role and the administrator must manually reconfigure the NIC to the active state.

Effect of Failback yes setting on environment
When using the default setting of Failback unexpected behavior can occur during maintenance of a physical switch. Most switches, like those from Cisco, initiate the port after boot, so called Lights on. The port is active but is still unable to receive or transmitting data. The process from Lights-on to forwarding mode can take up to 50 seconds; unfortunately ESX is not able to distinguish between Lights-on status and forwarding mode, there for treating the link as usable and will return the NIC to active status again.

High Availability will proceed to transmit heartbeats and expect to receive heartbeats, after missing 13 seconds of heartbeats HA will try to ping its Isolation Address, due to the specified Isolation respond it will shut down or power-off the virtual machines two seconds later to allow other ESX hosts to power-up the virtual machines. But because it is common – recommended even – to configure each host in the cluster in an identical manner, each active NIC used by the management network of every ESX host connect to the same physical switch. Due to this design, once the switch is booted, a cluster wide Isolation response occurs resulting in a cluster wide outage.

To allow switch maintenance, it’s better to set the vSwitch failback mode to No. Selecting this setting introduces an increase of manual operations after failure or certain maintenance operations, but will reduce the change of “false positives” and cluster-wide isolation responses.

vSwitch Failback and High Availability is a post from: frankdenneman.nl

The three existing load-balancing policies, Port-ID, Mac-Based and IP-hash use a static mapping between virtual switch ports and the connected uplinks. The VMkernel assigns a virtual switch port during the power-on of a virtual machine, this virtual switch port gets assigned to a physical NIC based on either a round-robin- or hashing algorithm, but all algorithms do not take overall utilization of the pNIC into account. This can lead to a scenario where several virtual machines mapped to the same physical adapter saturate the physical NIC and fight for bandwidth while the other adapters are underutilized. LBT solves this by remapping the virtual switch ports to a physical NIC when congestion is detected.

After the initial virtual switch port to physical port assignment is completed, Load Based teaming checks the load on the dvUplinks at a 30 second interval and dynamically reassigns port bindings based on the current network load and the level of saturation of the dvUplinks. The VMkernel indicates the network I/O load as congested if transmit (Tx) or receive (Rx) network traffic is exceeding a 75% mean over a 30 second period. (The mean is the sum of the observations divided by the number of observations).

An interval period of 30 seconds is used to avoid MAC address flapping issues with the physical switches. Although an interval of 30 seconds is used, it is recommended to enable port fast (trunk fast) on the physical switches, all switches must be a part of the same layer 2 domain.

Load Based Teaming is a post from: frankdenneman.nl

A little background first:
When installing the Cisco Nexus 1000V VSM, the VSM uses an extension-key for identification. During the configuration process the VSM spawns a DVS and will configure it with the same extension-key. Due to the matching extension keys (extension session) the VSM owns the DVS essentially.
And only the VSM with the same extension-key as the DVS can delete the DVS.

So to be able to delete a DVS, a VSM must exist registered with the same extension key.

If you deleted the VSM and are stuck with an orphaned DVS, the first thing to do is to install and configure a new VSM. Use a different switch name than the first (deleted) VSM. The new VSM will spawn a new DVS matching the switch name configured within the VSM.
The first step is to remove the new spawned DVS and do this the proper way using commands from within the VSM virtual machine.

Removing DVS with Nexus VSM virtual machine:
Log in VSM
Ping the vCenter to make sure you have a connection.

conf t

svs connection connection_name

The connection_name is created during the configuration of the connection between the VSM and the vCenter server. The default connection name is vCenter.

To query the current connection name:

show svs connections

If the SVS connection output does not show a datacenter name, but the minus (-) sign, you must specify the vCenter datacenter where the DVS is created, with the following command:
(In my case I needed to specify the datacenter even when the datacenter name was listed in the svs connection)

vmware dvs datacenter-name name (case sensitive)

e.g: vCenter datacenter name = DATACenter

vmware dvs datacenter-name DATACenter

Use the following command to remove the DVS:

no vmware dvs

The following warning appears:

This will remove the DVS from the vCenter Server and any associated port-groups

Do you really want to proceed (yes/no)

When selecting Yes, the following output appears in the VSM command prompt:

Note: Command execution in progress, please wait..

Simultaneously the recent task window in vCenter shows two tasks:
Delete folder
Delete vNetwork Distributed Switch

Select Network Inventory view to check if the DVS is deleted.

Removing DVS after destroying the VSM virtual machine.
The first part was the easy part, removing a DVS after the corresponding VSM is removed is a bit trickier.
First we must change the hostname of the VSM to reflect the switch name of the orphaned DVS.

Log in VSM

conf t
hostname
exit
copy run start

After setting the new hostname, the command prompt changes immediately to the “new” hostname.
At this moment, the VSM is using the same switch name, but it still uses a different extension-key as the orphaned DVS.
We need to change the extension key of the VSM to match the extension key of the orphaned DVS. Both old and new extension keys are listed in the vCenter database. First we need to know which extension-key the VSM is currently using. This key is going to be deleted from the vCenter DB.
This is done by using the following command in the VSM:

show vmware vc extension-key

The command prompt returns with an extension-key e.g:

Cisco_Nexus_1000V_1234101238

We need to remove this extension key from the vCenter DB, to do this we are going to use the managed Object Browser or (MOB. The Managed Object Browser is a web-based tool for working with the API. This tool enables you to browse managed objects on vCenter Server.
Open an Internet Explorer window to access the vCenters’ MOB.
Enter https:///mob

See KB 568529 for more information about using the Managed Object Browser operations.
Log in with user with administrator rights in vCenter.
Before we are going to unregister the “new” extension-key, we need to know which extension-key the old VSM used. (the values listed in italic are examples, the values in your environment can be different)

Go to:
ServiceContent: content
rootFolder: group-d8
childEntity: datacenter-76
networkFolder: group-n15
childEntity: group-n3456
childEntity: dvs-3457
DVSConfigInfo: config

At this moment, two keys exists, one key matching the orphaned DVS and one used by the newly spawned DVS during configuration of the second VSM. Copy the old key matching the orphaned DVS to notepad file; we are using that key later on.
Now copy the key matching the new DVS, this key must be removed using the ExtensionManager. For example:
extensionKey: “Cisco_Nexus_1000V_1234101238”

Go to: https://vcenterhostname/mob/?moid=ExtensionManager or follow the following path from the MOB home screen:
ServiceContent: content
extensionManger: ExtensionManager
void: unregisterExtension
extensionKey (required) string: paste key you copied from the DVSConfigInfo
for example:

extensionKey (required) string: Cisco_Nexus_1000V_8321457891
Click on InvokeMethod
This will return with the status: Method invocation result: void

Now return to the VSM command prompt and use the old extension id of the first VSM (saved in notepad file).

vmware vc extension-key

If the following error appears:

Cannot change the extension key while a connection is already enabled

the svs connection is still active and you must disconnect the current SVS connection by entering the following commands:

svs connection vcenter
no connect

Issue the vmware vc extension-key again after closing the svs connection.
At this point the VSM is configured with a matching extension-key as the orphaned DVS, but the VSM must registered within vCenter with this extension-key. To do this, you must use the extension.xml of the VSM, which is available for download on the webpage of the Nexus VSM.
(Before I downloaded the xml file I restarted the vCenter, don’t know if this is necessary, but I just wanted to be sure that the prior configuration settings are saved and committed to the database.)

Register the VSM by importing the xml with the extension key of the orphaned DVS switch:
open internet explorer and enter the ip-address of the VSM.
right click the link to save the cisco_nexus_1000v_extension.xml to your computer
open vcenter
Select Plug-ins
Manage Plug-ins…
Right click on whitespace inside Plug-in Manager and select the option New Plug-in…
click the Browse button and select the saved xml file
click on Ok.

now the Cisco_Nexus_1000v_old extension key code appears in Available Plug-ins section.

At this point the VSM is using the matching switch name and extension key of the orphaned DVS and is registered with the extension key in vCenter.Time to connect the VSM to vCenter. This will spawn a new DVS, this DVS will use the same extension key and switch name as the orphaned DVS and override all the info in the vCenter database. But because a extension session will exist we can remove the newly spawned DVS from the VSM using the commands mentioned in the first part of this article.
Return to the command prompt of the VSM and issue the following commands;

conf t
svs connection vcenter
vmware dvs datacenter_name datacenter

The DVS is created with the same name as the orphaned dvs and overwriting the old configuration. When this is completed check the network inventory view.
return to command promt and issue the command

no vmware dvs

to remove the new-old DVS.

If you will receive an error follow the steps mentioned in the Removing DVS with Nexus VSM virtual machine section.
The orphaned DVS is removed.
More information about the Nexus commands go visit the online nexus command reference

Removing orphaned Nexus DVS is a post from: frankdenneman.nl

First of all, unlike the the default Port-based load balancing policy, IP-Hash has got some technical requirements.

The physical switch must be configured with STATIC LACP (a.k.a. etherchannel, for Cisco switches) Dynamic LACP is not supported by VMware.  KB article 1004048 focuses on the configuration of LACP between the ESX server and physical switches.

Another technical requirement is that all uplinks in the vSwitch need to be active as the physicall switch is not aware of certain status of the uplink inside the vSwitch. David Marotta (VMware) explained the theory in such a way, I cannot forget this setting after hearing the story.

Because the pSwitch thinks it can send traffic to a particular VM down any of the pNICs. So all pNICs should be Active. Otherwise traffic to VM-A could be sent by the pSwitch to pNIC-2, and if the vSwitch thinks pNIC-2 is not Active for the port group used by VM-A, the vSwitch will drop it. Then VM-A will never get the packet. And it will sit in a corner, lonely and depressed, wondering why nobody calls it anymore.

Etherchannels and standby uplinks can introduce “Macflaps”, Duncan Epping (@DuncanYB) has written an excellent article on this as well Active / Standby etherchannels?

Besides the technical requirements of IP-hash, It will not do perfect load-balancing out of the box. Remembering Ken Cline’s (@clinek) the great vSwitch debate series about vSwitches I knew you must dive in to the algorithm used by IP-Hash load balancing and pick specific IP-addresses to make IP-hash load-balancing work.

But how does this IP-Hash algorithm work? As Ken cline so eloquently stated;

Take an exclusive OR of the Least Significant Byte (LSB) of the source and destination IP addresses and then compute the modulo over the number of pNICs.

OK, right! So how can we calculate it so we know if if the environment is balanced? Instead of the following the algorithm stated by Ken, I’ve used the method described in KB article 1007371

Waiver
The ip-addresses which are used in this example are ficticious, the ip-addresses are not based on real-life addresses, if those are in use by some company, it’s purely coincidental.

Step 1: Convert IP address to a HEX value
Use a IP Hex Converter tool to convert the IP addresses to Hex. An online Hex converter tool can be found at http://www.kloth.net/services/iplocate.php

In this example I use a vswitch with 2 uplinks. When using IP-hash, the first uplink has a IP binary representation of 0, the second uplink has a IP binary representation of 1.

The IP-Hash is calculated on source IP address, the VMkernel NFS IP address and the destination address, the NFS array IP address.

The VMknic has in this example the ip address of 145.10.44.10,
The first IP address of the NAS is 145.10.44.80 and the second address is 145.10.44.90

HEX: ESX VMkernel:   145.10.44.10 = 910A2C0A
HEX: NFS address 1:  145.10.44.80 = 910A2C50
HEX: NFS address 2:  145.10.44.90 = 910A2C5A

Step 2 Calculate the binary representation of the HASH
Now lets calculate the binary representation of the uplink’s IP address. I use windows for my desktop, so I use calc.exe for this example.

Open calc.exe and select Programmer, select the option HEX and Qword and paste the HEX value of the VMkernel NIC

Now press Xor , enter the (first) NFS IP address in HEX format (910A2C50) and click on the = button. The result is 5a, press the option Mod (modulo) and use the number of uplinks as value (2). Click on the = button to calculate the modulo.

The result of this calculation is the number 0 (zero). This means that IP-hash chooses the first uplink because the hash and the uplink both have an binary representation of 0.
Now lets calculate the second hash. In short;
HEX value “VMkernel NIC” Xor HEX value “NFS address 2”:
910A2C0A Xor 910A2C5A = 50    MOD 2 =           0

The result of this calculation is also 0 (zero), this means that the VMkernel does not balance traffic and will send traffic to across one uplink.

One ip-address of the NFS array needs to be changed to ensure that the VMkernel wil balance outbound traffic. For this example, IP-address 2: 145.10.44.90 is changed to 145.10.44.81, the HEX value of this address is 910A2C51.

Now lets calculate the binary representation.
910A2C0A Xor 910A2C51 = 5B    MOD 2 =           1.

The result of this calculation is 1 (one) The VMkernel chooses the second uplink because it has the same binary representation of the Hash. Hereby balancing outbound NFS traffic across the two uplinks.

Using IP-Hash to load-balance is a excellent choice, but you do need to fulfill certain technical requirements to get it supported by VMware and plan your IP-address scheme accordingly to get the most out of this load-balancing Policy.

One last thing, because I knew little about NFS, I turned to my primary source of storage related VMware articles Chad Sakac (@sakacc) and he written an excellent article about the use of NFS and VMware together with Vaughn Stewart (@vstewed)  of NetApp. Please read it if you haven’t already.  A “Multivendor Post” to help our mutual NFS customers using VMware. Again it’s truely excellent!

NFS and IP-HASH loadbalancing is a post from: frankdenneman.nl

“What will happen if the VMkernel decides to use that nic to send IO?
Is the Flexnic aware of the standby status of it “native” uplink? Will it send data to the uplink of the VC module it’s connected to or will it send data to the active uplink?
How is this done? Will it send the IO through the midplane or CX-4 cable to the VC module with the active uplink? And if this occurs what will be the added latency of this behavior?
HP describes the standby status as blocked, what does this mean? Will virtual connect discard IO send to the standby IO, will it not accept IO and how will it indicate this?”

The virtualconnect module is not standby, it’s external( X1 thru x6) ports are standby. The blade can and will send I/O to the virtualconnect module. It is the only way, because the blade NIC is hardwired to that VC module. When the virtual connect module receives data, it will transfer the data to the Virtual Connect module with the active ports over its internal X0 port.

The X0 port is also mentioned in the HP documentation as cross connect. HP beefed up the cross connects in theThe Flex10 module, it has 2 x 10 cross connects instead of the 1 x 10 cross connect found in the 1/10 virtual connect module

BL460c G1 NC373i 1GB Nics bandwidth upgrade
The new BL460cG6 blade has a Flex10 LOM adapter, when using G6 blades along G1 blades in a C7000 enclosure It might be nice to swap the 1/10 virtual connect module for Flex10 virtual connect modules.

When a NC373i is connected to a flex10 VC module the link speed of the 1GB module will be upgraded to 2,5 GB. It does not automatically upgrade the bandwidth when installing the Flex10 module, a firmware upgrade of the NC373i nic is needed.

Download the network firmware update tool from the HP support site (date 7 oct 2008, version 2.1.3.1). The updated version of the boot code, 4.4.1 that enables the 2.5 Gigabit support. Windows 2003 will not show the proper uplink speed. I haven’t checked it on a windows 2008 server. But to be certain Check the Flex10 virtual connect module to see which speed the nic is linked to.

Flex10 update is a post from: frankdenneman.nl