When enabling datastore maintenance mode, Storage DRS will move virtual machines out of the datastore as fast as it can. The number of virtual machines that can be migrated in or out of a datastore is 8. This is related to the concurrent migration limits of hosts, network and datastores. To manage and limit the number of concurrent migrations, either by vMotion or Storage vMotion, a cost and limit factor is applied. Although the term limit is used, a better description of limit is maximum cost.
In order for a migration operation to be able to start, the cost cannot exceed the max cost (limit). A vMotion and Storage vMotion are considered operations. The ESXi host, network and datastore are considered resources. A resource has both a max cost and an in-use cost. When an operation is started, the in-use cost and the new operation cost cannot exceed the max cost.
The operation cost of a storage vMotion on a host is “4”, the max cost of a host is “8”. If one Storage vMotion operation is running, the in-use cost of the host resource is “4”, allowing one more Storage vMotion process to start without exceeding the host limit.
As a storage vMotion operation also hits the storage resource cost, the max cost and
in-use cost of the datastore needs to be factored in as well. The operation cost of a Storage vMotion for datastores is set to 16, the max cost of a datastore is 128. This means that 8 concurrent Storage vMotion operations can be executed on a datastore. These operations can be started on multiple hosts, not more than 2 storage vMotion from the same host due to the max cost of a Storage vMotion operation on the host level.
How to throttle the number of Storage vMotion operations?
To throttle the number of storage vMotion operations to reduce the IO hit on a datastore during maintenance mode, it preferable to reduce the max cost for provisioning operations to the datastore. Adjusting host costs is strongly discouraged. Host costs are defined as they are due to host resource limitation issues, adjusting host costs can impact other host functionality, unrelated to vMotion or Storage vMotion processes.
Adjusting the max cost per datastore can be done by editing the vpxd.cfg or via the advanced settings of the vCenter Server Settings in the administration view.
If done via the vpxd.cfg, the value vpxd.ResourceManager.MaxCostPerEsx41Ds is added as follows:
< config >
< vpxd >
< ResourceManager >
< MaxCostPerEsx41Ds > new value < /MaxCostPerEsx41Ds >
< /ResourceManager >
< /vpxd >
< /config >
As the max cost have not been increased since ESX 4.1, the value-name
Please remember to leave some room for vMotion when resizing the max cost of a datastore. The vMotion process has a datastore cost as well. During the stun/unstun of a virtual machine the vMotion process hits the datastore, the cost involved in this process is 1.
For example, Changing the
Please note that cost and max values are applied to each migration process, impact normal day to day DRS and Storage DRS load balancing operations as well as the manual vMotion and Storage vMotion operations occuring in the virtual infrastructure managed by the vCenter server.
As mentioned before adjusting the cost at the host side can be tricky as the costs of operation and limits are relative to each other and can even harm other host processes unrelated to migration processes. If you still have the urge to change the cost on the host, consider the impact on DRS! When increasing the cost of a Storage vMotion operation on the host, the available “slots” for vMotion operations are reduced. This might impact DRS load balancing efficiency when a storage vMotion process is active and should be avoided at all times.
Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman
Thanks very much for this deep dive Frank!
Regards,
Mario
I see a need to limit the number of storage vMotions but not on a per host basis. I need to limit them per datastore, datastore cluster or per storage controller. Is there an option to do this that I haven’t found yet?
If I do multiple storage vMotions on the same NetApp filer head today, my storage guys don’t seem to be happy…
Frank,
I’m not able to get this to limit the storage vMotions. I set my cost to 34 and can still migrate over 2 at a time. Tried restarting the vCenter service to make sure the config took. Has anything changed since your write up? I’m on 5.0 U1a.
Josh, I tried to do this as well and ran into problems. It turns out the capitalization of the variable name in this article is incorrect and capitalization matters for the variable names. I don’t know if it was an auto-correct or what that caused the problem but the correct variable to set is vpxd.ResourceManager.maxCostPerEsx41Ds
Anyone know if this works in 5.0?
Yes, the variable I posted works on 5.0.
Hi Bill,
As my article stated: As the max cost have not been increased since ESX 4.1, the value-name remains the same and is valid for all ESX 4.1+ hosts.
This means that any version from 4.1 and up use this variable. I.e vSphere 5.0 and vSphere 5.1.
Thanks Frank. Is there a list out there of other operations and their associated costs? What I’m curious about is if I want, for example, a maximum of 4 storage vmotions at a time so I set my max datastore cost to 64 what other operations will I be preventing from occurring if four storage vmotions should happen at once? You state in your article that a Vmotion will have a cost of 1 on the datastore. What else could add costs? If I want to ensure a maximum of four does it make more sense to set my max to 79 (4 * 16 for the storage vmotions + 15 for additional without allowing a 5th vmotion).
Thanks!