frankdenneman Frank Denneman is a Chief Technologist at VMware, primarily focusing on Machine Learning technology. He is an author of the vSphere host and clustering deep dive series, podcast host for the Unexplored Territory podcast, and you can follow him on Twitter @frankdenneman.

Calculating the bandwidth usage and duration of a vMotion process?

2 min read

Every once in a while I get the question if I have a calculator that can determine the lead-time and the bandwidth consumption of a vMotion process. Unfortunately I haven’t got such a calculator, as there isn’t an easy way to calculate the consumed bandwidth and the duration of a vMotion process.
CPU
vMotion tries to move the used memory blocks as fast as possible. vMotion uses all the available bandwidth depending on the available CPU speed and bandwidth. Depending on the detected line speed, vMotion reserves an X amount of CPU speed at the start of a vMotion process. vMotion computes its desired host vMotion CPU reservation. For every 1GBe vMotion link speed it detects vMotion in vSphere 5.1 reserved 10% of a CPU core with a minimum desired CPU reservation of 30%. This means that if you use a single 1GBe, vMotion reserves 30% of a core, if you use 4 x 1GBe connections, that means vMotion reserves 40% of a core. A 10GBe link is special as vMotion reserves 100% of a single core.
vMotion creates a (system) resource pool and sets the appropriate CPU reservation on the resource pool. It’s important to note that this is being done to the vMotion resource pool, which means that the reservation is shared across all vMotions happening on the host.
vMotion system Resource Pool
Warning: DO NOT CHANGE the default settings of the system vMotion resource pool. This is set dynamically by the kernel depending on its memory state, manually adjusting this setting will likely hurt performance. Please do not attempt to be smarter then the kernel, many have tried, very few have succeeded.
DRS
When DRS is enabled, it can decide to migrate virtual machines as well. It might happen that this occurs at the same time your vMotion process is running. All vMotions will be placed into the vMotion resource pool contesting for the resources acquired by the resource pool of vMotion. If high priority for the manual vMotion is selected (User interface uses the term: Reserve CPU for optimal vMotion performance) then the vMotion process receives a higher priority within the vMotion resource pool. In which case the high priority vMotion will have double the relative CPU shares, and as a result probably complete more quickly than their lower priority counterparts. However it still need to share the resources claimed by the vMotion resource pool. Although it has a higher priority over DRS vMotions, sharing resources still may have an effect on the duration of the vMotion process.
Memory
vMotion copies only the used memory blocks, a virtual machine doesn’t always have to use all of its memory. Therefor its not easy to determine the required bandwidth. To make it more complex, as we are migrating a live virtual machine, the virtual machine can dirty (re-use) memory blocks that are already copied over, those blocks have to be sent again. Prolonging the duration of the process and the used bandwidth.
Swap file
If the swap file is located on a non-shared datastore and pages has been stored in the swap file, those pages are copied over to the new swap file on a location accessible by the destination host. This will increase the demand for bandwidth and increases the duration of the vMotion process. For more information about the impact of non-shared swap files, please read the following articles: (Alternative) VM swap file locations Q&A and (Alternative) VM swap file locations Q&A – part 2.
Conclusion
As you can see, it’s very difficult to determine the duration of a vMotion process and the actual bandwidth it consumes.

frankdenneman Frank Denneman is a Chief Technologist at VMware, primarily focusing on Machine Learning technology. He is an author of the vSphere host and clustering deep dive series, podcast host for the Unexplored Territory podcast, and you can follow him on Twitter @frankdenneman.

Help Us Make vMotion Even Better

The vMotion product team is looking for input on how to improve vMotion.  vMotion has proven to be a paradigm shift of datacenter management....
frankdenneman
21 sec read

Disable vMotion for a single VM

This question pops up regularly on the VMTN forums and reddit. It’s a viable question but the admins who request this feature usually don’t...
frankdenneman
1 min read

vSphere 5.1 update 1 release fixes Storage vMotion rename…

vSphere 5.1 update 1 is released today which contains several updates and bug fixes for both ESXi and vCenter Server 5.1. This release contains...
frankdenneman
29 sec read