frankdenneman Frank Denneman is the Chief Technologist for AI at VMware by Broadcom. He is an author of the vSphere host and clustering deep dive series, as well as a podcast host for the Unexplored Territory podcast. You can follow him on Twitter @frankdenneman

Multi-NIC vMotion – failover order configuration

2 min read

After posting the article “designing your vMotion network” I quickly received the question which failover order configuration is better. Is it better to configure the redundant NIC(s) as standby or as unused? The short answer: always use standby and never unused!
Tomas Fojta posted the comment that it does not make sense to place the NICs into standby mode:

In the scenario as depicted in the diagram I prefer to use active/unused. If you think about it the standby option does not give you anything as when one of the NICs fails both vmknics will be on the same NIC which does not give you anything.

Although it does not provide any performance benefits having vMotion routing the traffic across the same physical NIC during a NIC failure, there are two important reasons for providing redundant connection to each vmknic:

  1. Abstraction layer
  2. Using a default interface for management traffic

Abstraction layer
As mentioned in the previous article, vMotion operations are done at the vmknic level. Due to vMotion focussing on the vmknic layer instead of the physical layer, some details are abstracted from vMotion load balancing logic such as the physical NIC’s link state or health. Due to this abstraction vMotion just selects the appopriate vmknics for load balancing network packets and trusts that there is connectivity between the vmknics of the source and destination host.

Default interface
Although vMotion is able to use multiple vmknics to load balance traffic, vMotion assigns one vmknic as its default interface and prefers to use this for connection management and some trivial management transmissions. *
As such, if you’ve got multiple physical NICs on a host that you plan to use for vMotion traffic, it makes sense to mark them as standby NICs for the other vMotion vmknics on the host. That way, even if you lose a physical NIC, you won’t see vMotion network connectivity issues.
This means that if you have designated three physical NICs for vMotion your vmknic configuration should look as follows:

VMknic Active NIC Standby NIC
vmknic0 NIC1 NIC2, NIC3
vmknic1 NIC2 NIC1, NIC3
vmknic2 NIC3 NIC1, NIC2

By placing the redundant NICs into standby instead of unused you avoid the risk of having an unstable vMotion network. If a NIC fails, you might experience some vMotion performance degradation as the traffic gets routed through the same NIC, but you can trust your vMotion network to correclty migrate all virtual machines off the host in order for to replace the faulty NIC.
* Word to the wise
By writing about the fact that vMotion designates a vmknic to be the default interface I’m aware that this triggers and sparks the interest of some of the creative minds in our community. Please do not attempt to figure out which vmknic is designated as default interface and make that specific vmknic redundant and different from the rest. To paraphrase Albert Einstein: “Simplicity is the root of all genius”. Keep your Multi-NIC consistent and identical within the host and throughout all hosts. This saves you a lot of frustration during troubleshooting. Being able to depend on your vMotion network to migrate the virtual machines safely and correctly is worth its weight in gold.
Part 1 – Designing your vMotion network
Part 3 – Multi-NIC vMotion and NetIOC
Part 4 – Choose link aggregation over Multi-NIC vMotion?
Part 5 – 3 reasons why I use a distributed switch for vMotion networks

frankdenneman Frank Denneman is the Chief Technologist for AI at VMware by Broadcom. He is an author of the vSphere host and clustering deep dive series, as well as a podcast host for the Unexplored Territory podcast. You can follow him on Twitter @frankdenneman

Help Us Make vMotion Even Better

The vMotion product team is looking for input on how to improve vMotion.  vMotion has proven to be a paradigm shift of datacenter management....
21 sec read

Disable vMotion for a single VM

This question pops up regularly on the VMTN forums and reddit. It’s a viable question but the admins who request this feature usually don’t...
1 min read

vSphere 5.1 update 1 release fixes Storage vMotion rename…

vSphere 5.1 update 1 is released today which contains several updates and bug fixes for both ESXi and vCenter Server 5.1. This release contains...
29 sec read

7 Replies to “Multi-NIC vMotion – failover order configuration”

  1. What about Multi-Nic vMotion with IP Hash configured at the vswitch level ? Would it speed the things up to create multiple VMkernel port on this config ?

  2. Nice post Frank. I’ve had this ‘argument’ with some people in the past. Most felt it should be Active/Unused because they are use to that configuration for iSCSI multipathing. I have always used Active/Standby for multi nic vMotion, although never knew the exact reason why that correct…..until now 🙂

  3. Me again 🙂
    Would it make sense to use the same VMkernels for both iSCSI and vMotion ? (please note that this is on Essentials+ edition so very few migrations and manual only).
    I could also create some additionals VMkernels but on same vmnics.
    Network config is : 4 ports in IP Hash for VMs + 4 ports for iSCSI
    How would you deal with that case ?

Comments are closed.