Frank Denneman - Chief Technologist AI at VMware

Calculating the bandwidth usage and duration of a vMotion process?

December 4, 2012 by frankdenneman

Every once in a while I get the question if I have a calculator that can determine the lead-time and the bandwidth consumption of a vMotion process. Unfortunately I haven’t got such a calculator, as there isn’t an easy way to calculate the consumed bandwidth and the duration of a vMotion process.
CPU
vMotion tries to move the used memory blocks as fast as possible. vMotion uses all the available bandwidth depending on the available CPU speed and bandwidth. Depending on the detected line speed, vMotion reserves an X amount of CPU speed at the start of a vMotion process. vMotion computes its desired host vMotion CPU reservation. For every 1GBe vMotion link speed it detects vMotion in vSphere 5.1 reserved 10% of a CPU core with a minimum desired CPU reservation of 30%. This means that if you use a single 1GBe, vMotion reserves 30% of a core, if you use 4 x 1GBe connections, that means vMotion reserves 40% of a core. A 10GBe link is special as vMotion reserves 100% of a single core.
vMotion creates a (system) resource pool and sets the appropriate CPU reservation on the resource pool. It’s important to note that this is being done to the vMotion resource pool, which means that the reservation is shared across all vMotions happening on the host.

Warning: DO NOT CHANGE the default settings of the system vMotion resource pool. This is set dynamically by the kernel depending on its memory state, manually adjusting this setting will likely hurt performance. Please do not attempt to be smarter then the kernel, many have tried, very few have succeeded.
DRS
When DRS is enabled, it can decide to migrate virtual machines as well. It might happen that this occurs at the same time your vMotion process is running. All vMotions will be placed into the vMotion resource pool contesting for the resources acquired by the resource pool of vMotion. If high priority for the manual vMotion is selected (User interface uses the term: Reserve CPU for optimal vMotion performance) then the vMotion process receives a higher priority within the vMotion resource pool. In which case the high priority vMotion will have double the relative CPU shares, and as a result probably complete more quickly than their lower priority counterparts. However it still need to share the resources claimed by the vMotion resource pool. Although it has a higher priority over DRS vMotions, sharing resources still may have an effect on the duration of the vMotion process.
Memory
vMotion copies only the used memory blocks, a virtual machine doesn’t always have to use all of its memory. Therefor its not easy to determine the required bandwidth. To make it more complex, as we are migrating a live virtual machine, the virtual machine can dirty (re-use) memory blocks that are already copied over, those blocks have to be sent again. Prolonging the duration of the process and the used bandwidth.
Swap file
If the swap file is located on a non-shared datastore and pages has been stored in the swap file, those pages are copied over to the new swap file on a location accessible by the destination host. This will increase the demand for bandwidth and increases the duration of the vMotion process. For more information about the impact of non-shared swap files, please read the following articles: (Alternative) VM swap file locations Q&A and (Alternative) VM swap file locations Q&A – part 2.
Conclusion
As you can see, it’s very difficult to determine the duration of a vMotion process and the actual bandwidth it consumes.

How to create a "New Storage DRS recommendation generated" alarm

November 30, 2012 by frankdenneman

It is recommended to configure Storage DRS in manual mode when you are new to Storage DRS. This way you become familiar with the decision matrix Storage DRS uses and you are able to review the recommendations it provides. One of the drawbacks of manual mode is the need to monitor the datastore cluster on a regular basis to discover if new recommendations are generated. As Storage DRS is generated every 8 hours and doesn’t provide insights when the next invocation run is scheduled, it’s becomes a bit of a guessing game when the next load balancing operation has occurred.
To solve this problem, it is recommended to create a custom alarm and configure the alarm to send a notification email when new Storage DRS recommendations are generated. Here’s how you do it:
Step 1: Select the object where the alarm object resides
If you want to create a custom rule for a specific datastore cluster, select the datastore cluster otherwise select the Datacenter object to apply this rule to each datastore cluster. In this example, I’m defining the rule on the datastore cluster object.
Step 2: Go to Manage and select Alarm Definitions
Click on the green + icon to open the New Alarm Definition wizard

Step 3: General Alarm options
Provide the name of the alarm as this name will be used by vCenter as the subject of the email. Provide an adequate description so that other administrators understand the purpose of this alarm.
In the Monitor drop-down box select the option “Datastore Cluster” and select the option “specific event occurring on this object, for example VM Power On”. Click on Next.

Step 4: Triggers
Click on the green + icon to select the event this alarm should be triggered by. Select “New Storage DRS recommendation generated”. The other fields can be left blank, as they are not applicable for this alarm. Click on next.

Step 5: Actions
Click on the green plus icon to create a new action. You can select “Run a Command”, “Send a notification email” and “Send a notification trap”. For this exercise I have selected “Send a notification email”. Specify the email address that will receive the messages containing the warning that Storage DRS has generated a migration recommendation. Configure the alarm so that it will send a mail once when the state changes from green to yellow and yellow to red. Click on Finish.

The custom alarm is now listed between the pre-defined alarms. As I chose to define the alarm on this particular datastore cluster, vCenter list that the alarm is defined on “this Object”. This particular alarm is therefor not displayed at Datacenter level and cannot be applied to other datastore clusters in this vCenter Datacenter.

Please note that you must configure a Mail server when using the option “send a notification email” and configure an valid SNMP receiver when using the option “Send a notification trap”. To configure a mail or SNMP server, select the vCenter server option in the inventory list, select manage, settings and click on edit. Go to Mail and provide a valid mail server address and an optional mail sender.

To test the alarm, I moved a couple of files onto a datastore to violate the datastore cluster space utilization threshold. Storage DRS ran and displayed the following notifications on the datastore cluster summary screen and at the “triggered alarm” view:

The moment Storage DRS generated a migration recommendation I received the following email:

As depicted in the screenshot above, the subject of the email generated by vCenter contains the name of the alarm you specified (notice the exclamation mark), the event itself – New Storage DRS recommendation generated” and the datastore cluster in which the event occurred.

How to create a "New Storage DRS recommendation generated" alarm

November 30, 2012 by frankdenneman

vSphere 5.1 Clustering deepdive Cyber Monday deal

November 26, 2012 by frankdenneman

We are long time fascinated by the whole Black Friday and Cyber Monday craze in the USA. Unfortunately we do not celebrate Thanksgiving in the Netherlands and none of the shops are participating in something similar as Black Friday.
Similar to last year, we thought it was a great idea to participate in some form and what better than to offer our vSphere 5.1 Clustering Technical Deepdive book for a price you cannot resist. We just changed the price of the vSphere 5.1 Clustering Technical Deepdive to $17.95, Amazon Deutschland is offering the book for 16.00 EURO, while Amazon UK is selling the book for 11.01 Pounds sterling.
The book has some amazing reviews, here is one we like to share with you:

The book contains information critical to VMware administrators. Clustering is a critical technology, and the book covers the underlying concepts as well as the practical issues surrounding implementation. The information contained within will be important far past vSphere 5.1; the principles will apply for decades, and even the details of implementation are unlikely to change dramatically over the next few generations of the product. As with any good “deep dive,” the fundamental concepts discussed will ultimately help you in any clustering situation, even with non-VMware products.
From a reading comprehension standpoint, the book is easy to grok. The information flows quickly and you can read the entire work cover to cover with relative ease. This is a must have for any systems administrator.

What better way than recover from the madness of Black Friday and just sit back and relax reading this amazing piece of work? This is most definitely the deal of the year for all virtualization fanatics!
Duncan and Frank

(Alternative) VM swap file locations Q&A – part 2

November 19, 2012 by frankdenneman

After writing the article “(Alternative) VM swap file locations Q&A” I received a lot of questions about the destination of the swapped pages and reading back my article I didn’t do a good job clarifying that part of the process.
Which network is used for copying swapped pages?
As mentioned in the previous post the swap file itself is not copied over to the destination host, but only the swapped pages itself. Raphael Schitz (@hypervisor_fr) was the first to ask, which network is used to copy over the swapped pages? The answer is vMotion network.
The reason why the vMotion network is used, is that the source host running the active virtual machine, pulls the swapped pages back in to memory when migrating the memory pages to the destination host.
Are swapped pages on the source host swapped out on the destination host?
As the pages are copied out from the swap file to the destination host, swapped pages are copied into the stream of the in-memory pages from the source host to the destination host. That means that the destination host is not aware which pages orginate from swap file and which pages come from in-memory, they are just memory pages that need to be stored and made available to the new virtual machine.

To describe the behavior in a different way, the source host pulls the swapped pages from disk before sending them over, therefor the destination host sees a continues stream of memory pages, unmarked, all equal and are therefor stored in memory by the destination host.
What if the destination host is experiencing contention?
Well it’s up to the destination host to decide which pages to swap out to disk. During a vMotion process, the destination VM starts out with a clean slate, meaning that the memory target is not determined by the source host but by the destination host. Memory targets are local memory schedule metrics and thus not shared. The source host shares the percentage of active pages but it’s the destination hosts’ memory scheduler that determines the appropriate swap target for the new virtual machine. It can possibly push out memory pages back to its swap file as needed. The pages could be the same as the pages on the old host, but they can also completely different pages.
What about compressed pages?
For every rule there is an exception and the exception is compressed pages. During a vMotion process the destination host will maintain the compressed pages by keeping them compressed. This behavior occurs even with an unshared swap migration.
Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman