Resource entitlement
Before we dive into the old and the new chart, clarification of the term “resource entitlement” can be helpful. The charts show the resource entitlement of virtual machines running inside the cluster, but what are resource entitlements and what are they used for? To quote Minwen a VMware R&D engineer

Based on various stats and some estimation techniques, DRS determines each VM’s *demand* for CPU & memory resources. It then computes each VM’s cpu & mem *entitlement* (how many resources it should get) based on resource settings (shares, limits, reservations) and the degree of resource contention there is. If there are enough resources in the cluster to satisfy every VM’s demand (and assuming no limits), then entitlement is equal to demand, meaning every VM gets as much CPU & memory as it wants.

The old chart
Let’s begin with the vCenter 2.5 chart and take a look how the old chart showed the info. I always disliked the way the old chart displayed the number of hosts. The chart used in this example belongs to a cluster with 3 hosts. At the top of the Y-axis a number 2 is shown and in the middle of the Y axis a horizontal line is displayed. This horizontal line in this example depicts 50% of two hosts, in other words: one host. The chart shows multiple bars, the leftmost bars represent the resource utilization of 2 hosts, which means that two hosts have a CPU and memory utilization between 0 and 10 percent. The orange bar in the 20-30 column stretches to the horizontal line on the Y-axis and that means that the memory of the third host is utilized between 20 and 30 percent, in this example the same third host has a CPU utilization of 90-100 percent.
If the cluster is balanced, all orange or blue bars are closed to each other, the closer the blue and orange bars are to each other, the more balanced the cluster is.

The bottom chart “percent of entitled resources delivered” uses a different scale in this example. Because all three hosts deliver 90-100 percent of the memory resources the top of the Y axis represents 3 hosts. 2 hosts deliver 90-100 percent of the CPU resources and the bar stretches to 66% of the Y axis. Because of the current statistics, the horizontal bar in the middle of the Y-axis cannot be translated into one host. The Y-axis has a dynamic nature due to the relation of the scale to the load; this behavior does not contribute to legibility of the charts.

DRS Resource Distribution chart 2.0
The chart in vCenter 2.5 showed the CPU and Memory utilization of the host inside the cluster, load distribution across host and if the delivery of the resource entitlement of the virtual machines. The current chart used in vCenter 4 shows seven levels of information:

Cluster info:
• Relative balance of hosts X
• Approximate spare capacity left in the cluster

Host info:
• Resource usage for each host X
• Relative resource consumption of each VM on each host X

VM info:
• Resource usage for each VM X
• Host assignment for each VM X
• Percent resource entitlement for each VM X

VMware chose not to combine the CPU and Memory into one chart, but used a toggle view. Let’s take a look at the new view:

For this example I used the CPU chart, because of the pretty colors. You can choose between % or MHz view. I used the % view in all the examples for no specific reasons, it can be easily be substituted with the MHz view, both views are equally informative.

The new chart displays the host on the Y-axis and bar along the X-axis shows the CPU (or memory) utilization. The Y-axis lists the hosts of the cluster with its hostname. This way the resource usage of a specific host and the relative balance between hosts inside the cluster is instantly recognizable. In this example all the hosts are utilized below the 25%, it shows that we have an enormous amount of spare CPU capacity left in the cluster.

The horizontal bar behind the hostname shows the current resource usage of the host and is subdivided by the resource utilization of each virtual machine on that host. Each block in the bar stands for a specific virtual machine, the size of the block represent the relative resource consumption of the virtual machine on the host. When hovering over a block, the statistics of that specific virtual machine running on the host is displayed.

The info block shows the consumed resources, the active resources and the resource entitlement of the virtual machine.

Color vs Gray scale
The CPU view uses a color scheme. At the bottom of the chart view a gradient bar displays the percentage of the entitle resources delivered. The VM block color corresponds to the amount of entitled resources that are delivered by the host to the virtual machine.

The memory view uses a gray scale; actually it uses just one tone, just gray.

I don’t have an authoritative answer why the memory chart view doesn’t use a color gradient, but I have an educated guess. I already sent the product manager of DRS some questions; hopefully I can soon update this article with a definitive answer.
Stay tuned!

There are options to avoid creation of large slot sizes. Such as not setting reservations, disabling strict admission control, using vSphere new admission control policy “percentage of cluster resources reserved” or creating a custom slot size by altering the advanced settings das.vmMemoryMinMB.

But what if you are still using ESX 3.5, must guarantee memory resources for that specific VM, do not want to disable strict admission control or don’t like tinkering with the custom slot size setting? Maybe using the resource pool workaround can be an option.

Resource pool workaround
During a conversation with my colleague Craig Risinger, author of the very interesting article “The resource pool priority pie paradox”, we discussed the lack of relation between resource pools reservation settings and High Availability. As Craig so eloquently put it:

“RP reservations will not muck around with HA slot sizes”

High Availability ignores resource pools reservation settings when calculating the slot size, so if a single VM is placed in a resource pools with memory reservation configured, it will have the same effect on resource allocation as per VM memory reservation, but does not affect the HA slot size.

By creating a resource pool with a substantial memory setting you can avoid decreasing the consolidation ratio of the cluster and still guarantee the virtual machine its resources. Publishing this article does not automatically mean that I’m advocating using this workaround on a regular basis. I recommend implementing this workaround very sparingly as creating a RP for each VM creates a lot of administrative overhead and makes the host and cluster view a very unpleasant environment to work in.

A possible scenario to use this workaround can be when implementing MS Exchange 2010 mailbox servers. These mailbox servers are notorious for demanding a huge amount of memory and listed by many organizations as mission critical servers.

To emphasize it once more, this is not a best practice! But it might be useful in certain scenarios to avoid large slots and therefore low consolidation ratios.

VM swap file
Lets start with some basics, by default a VM swap file is created when a virtual machine starts, the formula to calculate the swap file size is: configured memory – memory reservation = swap file. For example a virtual machine configured with 2GB and a 1GB memory reservation will have a 1GB swap file.

Reservations will guarantee that the specified amount of virtual machine memory is (always) backed by ESX machine memory. Swap space must be reserved on the ESX host for the virtual machine memory that is not guaranteed to be backed by ESX machine memory. For more information on memory management of the ESX host, please the article on the impact of memory reservation.

During start up of the virtual machine, the VMkernel will pre-allocate the swap file blocks to ensure that all pages can be swapped out safely. A VM swap file is a static file and will not grow or shrink not matter how much memory is paged. If there is not enough disk space to create the swap file, the host admission control will not allow the VM to be powered up.

This rule also applies when migrating a VM configured with a host-local VM swap file as the swap file needs to be created on the local VMFS volume of the destination host. Besides creating a new swap file, the swapped out pages must be copied out to the destination host. It’s not uncommon that a VM has pages swapped out, even if there is not memory pressure at that moment. ESX does not proactively return swapped pages back into machine memory. Swapped pages always stays swapped, the VM needs to actively access the page in the swap file to be transferred back to machine memory but this only occurs if the ESX host is not under memory pressure (more than 6% free physical memory).

Copying host-swap local pages between source- and destination host is a disk-to-disk copy process, this is one of the reasons why VMotion takes longer when host-local swap is used.

Real-life scenario
A customer of mine was not aware of this behavior and had discarded the multiple warnings of full local VMFS datastores on some of their ESX hosts. All the virtual machines were up and running and all seemed well. Certain ESX servers seemed to be low on resource utilization and had a few active VMs, while other hosts were highly utilized. DRS was active on all the clusters, fully automated and a default (3 stars) migration threshold. It looked like we had a major DRS problem.

DRS
If DRS decide to rebalance the cluster, it will migrate virtual machines to low utilized hosts. VMkernel tries to create a new swap file on the destination host during the VMotion process. In my scenario the host did not contain any free space in the VMFS datastore and DRS could not VMotion any virtual machine to that host because the lack of free space. But the host CPU active and host memory active metrics were still monitored by DRS to calculate the load standard deviation used for its recommendations to balance the cluster. (More info about the DRS algorithm can be found on the DRS deepdive page). The lack of disk space on the local VMFS datastores influenced the effectiveness of DRS and limited the options for DRS to balance the cluster.

High availability failover
The same applies when a HA isolation response occurs, when not enough space is available to create the virtual machine swap files, no virtual machines are started on the host. If a host fails, the virtual machines will only power-up on host containing enough free space on their local VMFS datastores. It might be possible that virtual machines will not power-up at-all if not enough free disk space is available.

Failover capacity planning
When using host local swap setting to store the VM swap files, the following factors must be considered.
• Amount of ESX hosts inside cluster.
• HA configured host failover capacity.
• Amount of active virtual machines inside cluster.
• Consolidation ratio (VM per host).
• Average swap file size.
• Free disk space local VMFS datastores.

For the sake of simplicity, let’s assume that DRS balanced the cluster load and that all (identical) virtual machines are spread evenly across every host.

In case of a host failure, 27 VMs will be restarted on the remaining 5 hosts inside the cluster, HA will start 5.4 virtual machines per host, as it is impossible to start 0.4 VM, some ESX hosts will start 6 virtual machines, while other hosts will start 5 VM’s.
The average swap file size is 4GB, this requires at least 24 GB of free space to be available on the local VMFS datastores to start the VM’s. Besides the 24GB, enough free space needs to be available to for DRS to move multiple VMs around to rebalance the load across the cluster.

If the design of the virtual infrastructure incorporates site failover as well, enough free disk space on all the ESX hosts must be reserved to power-up all the affected virtual machines from the failed site.

Closing remarks
Using host local swap can be a valid option for some environments, but additional calculation of the factors mentioned above is necessary to ensure sustained HA and DRS functionality.

• Design defend session (75 minutes)
• Design session (30 minutes)
• Troubleshooting session (15 minutes)

During the design defend session you are required to present your design, I used a twelve deck slide presentation and included all blueprints\Visio drawings as appendix. This helped me a lot, as I am not a native English speaker using diagrams helped me to explain the layout.
There is no time limit on the duration of the presentation, but it is wise to keep it as brief as possible. During the session, the panel will try to address a number of sections and if they cannot address these sections this can impact your score.

The design and troubleshooting session you need to show you are able to think on your feet. One of the goals is to understand your though process. Thinking out loud and using the whiteboard will help you a lot.
So how was my experience? After meeting my panel members I started to get really nervous as one of the storage guru’s within VMware was on my panel. The other two panel members have an extreme good track record inside the company as well, so basically I was being judged by an all-star panel. I thought my presentation went well, but word of advice; read your submitted documentation on a regular basis before entering the defend panel as the smallest details can be asked.

After completing the design defend pane, I was asked to step outside. After the short break the design session and troubleshooting scenarios were next. I did not solve the design and troubleshooting scenarios, but that is really not the goal of those sections.

Thinking out loud in English can be challenging for non-native English speakers, so my advice is to try to practice this as much as possible. I did a test presentation for a couple of friends and discovered some areas to focus on before doing the defend part of the program.

After completing my defend panel, I was scheduled to participate as an observer on the remaining defend panel sessions the rest of the week. After multiple sessions as an observer and receiving the news that I passed the VCDX defend panel, I participated as a panel member on a defend session. Hopefully I will be on a lot more panels in the upcoming year, because sitting on the other side of the table is so much better than standing in front of it sweating like a pig.

Figure 1: Local and Remote memory access

Accessing remote memory will increase latency, the key is to avoid this as much as possible. How can you ensure memory locality as much as possible?



VM sizing pitfall #1, vCPU sizing and Initial placement.

ESX is NUMA aware and will use the NUMA CPU scheduler when detecting a NUMA system. On non-NUMA systems the ESX CPU scheduler spreads load across all sockets in a round robin manner. This approach improves performance by utilizing as much as cache as possible. When using a vSMP virtual machine in a non-NUMA system, each vCPU is scheduled on a separate socket.
On NUMA systems, the NUMA CPU scheduler kicks in and use the NUMA optimizations to assigns each VM to a NUMA node, the scheduler tries to keep the vCPU and memory located in the same node. When a VM has multiple CPUs, all the vCPUs will be assigned to the same node and will reside in the same socket, this is to support memory locality as much as possible.

Figure 2: NON-NUMA vCPU placement

Figure 3: NUMA vCPU placement

At this moment, AMD and Intel offer Quad Core CPU’s, but what if the customer decides to configure an 8-vCPU virtual machine? If a VM cannot fit inside one NUMA node, the vCPUs are scheduled in the traditional way again and are spread across the CPU’s in the system. The VM will not benefit from the local memory optimization and it’s possible that the memory will not reside locally, creating added latency by crossing the intersocket connection to access the memory.



VM sizing pitfall #2: VM configured memory sizing and node local memory size
NUMA will assign all vCPU’s to a NUMA node, but what if the configured memory of the VM is greater than the assigned local memory of the NUMA node? Not aligning the VM configured memory with the local memory size will stop the ESX kernel of using NUMA optimizations for this VM. You can end up with all the VM’s memory scattered all over the server.
So how do you know how much memory every NUMA node contains? Typically each socket will get assigned the same amount of memory; the physical memory (minus service console memory) is divided between the sockets. For example 16GB will be assigned to each NUMA node on a two socket server with 32GB total physical. A quick way to confirm the local memory configuration of the NUMA nodes is firing up esxtop. Esxtop will only display NUMA statistics if ESX is running on a NUMA server. The first number list the total amount of machine memory in the NUMA node that is managed by ESX, the statistic displayed within the round brackets is the amount of machine memory in the node that is currently free.

Figure 4: esxtop memory totals

Let’s explore NUMA statistics in esxtop a little bit more based on this example. This system is a HP BL 460c with two Nehalem quad cores with 64GB memory. As shown, each NUMA node is assigned roughly 32GB. The first node has 13GB free; the second node has 372 MB free. It looks it will run out of memory space soon, luckily the VMs on that node still can get access remote memory. When a VM has a certain amount of memory located remote, the ESX scheduler migrates the VM to another node to improve locality. It’s not documented what threshold must be exceeded to trigger the migration, but its considered poor memory locality when a VM has less than 80% mapped locally, so my “educated” guess is that it will be migrated when the VM hit a number below the 80%. Esxtop memory NUMA statistics show the memory location of each VM. Start esxtop, press m for memory view, press f for customizing esxtop and press f to select the NUMA Statistics.

Figure 5: Customizing esxtop

Figure 6 shows the NUMA statistics of the same ESX server with a fully loaded NUMA node, the N%L field shows the percentage of mapped local memory (memory locality) of the virtual machines.

Figure 6: esxtop NUMA statistics

It shows that a few VMs access remote memory. The man pages of esxtop explain all the statistics:

Transparent page sharing and memory locality.
So how about transparent page sharing (TPS), this can increase latency if the VM on node 0 will share its page with a VM on node 1. Luckily VMware thought of that and TPS across nodes is disabled by default to ensure memory locality. TPS still works, but will share identical pages only inside nodes. The performance hit of accessing remote memory does not outweigh the saving of shared pages system wide.

Figure 7: NUMA TPS boundaries

This behavior can be changed by altering the setting VMkernel.Boot.sharePerNode. As most default settings in ESX, only change this setting if you are sure that it will benefit your environment, 99.99% of all environments will benefit from the default setting.



Take away
With the introduction of vSphere ESX 4, the software layer surpasses some abilities current hardware techniques can offer. ESX is NUMA aware and tries to ensure memory locality, but when a VM is configured outside the NUMA node limits, ESX will not apply NUMA node optimizations. While a VM still run correctly without NUMA optimizations, it can experience slower memory access. While the actually % of performance decrease depends on the workload, avoiding performance decrease if possible must always be on the agenda of any administrator.

To quote the resource management guide:

The NUMA scheduling and memory placement policies in VMware ESX Server can manage all VM transparently, so that administrators do not need to address the complexity of balancing virtual machines between nodes explicitly.

While this is true, administrators must not treat the ESX server as a black box; with this knowledge administrators can make informed decisions about their resource policies. This information can help to adopt a scale-out policy (multiple smaller VMs) for some virtual machines instead of a scale up policy (creating large VMs) if possible.

Beside the preference for scale up or scale out policy, a virtual environment will profit when administrator choose to keep the VMs as agile as possible. My advice to each customer is to configure the VM reflecting its current and near future workload and actively monitor its habits. Creating the VM with a configuration which might be suitable for the workload somewhere in its lifetime can have a negative effect on performance.

Congratulations to all other people mentioned in the list and I would like to congratulate Duncan Epping specifically for taking the number 1 place again this year. The top 25 as published by Eric Siebert on vSphere-land:

A new home
Being voted one of the top 25 bloggers, puts a lot of pressure on one. I hope to continue blogging articles people find interesting. And to make a good start, frankdenneman.wordpress.com moved to www.frankdenneman.nl.

To my suprise, Eric decided to nominate my blog as well. I’m really honored to be a nominee amongst the best virtualization bloggers out there. Unfortunately the list Eric is longer that the 10 votes one can cast, so good luck picking the ones who stand out above the excellent crowd.

This is my top 10 blogs;

Duncan Epping
Chad Sakac
Kenneth van Ditmarsch
Alan Renouf
Scott Lowe
Scott Drummonds
Hypervizor (Hany Michael)
Arnim van Lieshout
Arne Fokkema
Gabe Virtual world.

Go vote now before it’s too late!
http://vsphere-land.com/news/time-to-vote-for-your-favorite-bloggers.html

Trying to create in-depth articles is an excellent way to learn stuff. Most of the time describing a certain subject somehow challenged my current knowledge of that topic and ended up spending ridiculous amounts of time researching that particular subject. More often than not coming across very interesting material not really related to the subject but similarly interesting, consuming even more time. Some articles are more popular than others; these are the top five visited articles this year;

1. Increasing the queue depth
2. Lefthand SAN – Lessons learned
3. HP Continuous Access and the use of LUN balancing scripts
4. Impact of memory reservations
5. NFS and IP-HASH Load-Balancing

Lately I’m running into a few limitations of the free wordpress blog themes, that’s why I’ve decided to move to another site, stay tuned for the URL.
I’m aiming to release the new site at the beginning of next year.

Begin August I got an e-mail from Duncan Epping If I would like to do some contractors work for VMware. As you can imagine, it didn’t take me long to respond with a Font size 72 YES. (Do you know you can make text blink in word?)
After completing a few project VMware offered me a permanent job, being a contractor for 9 years made the decision a bit tougher, but getting such a job offer is something you can hardly refuse.

Working with the best of the business, being able to access internal information and getting exposed to all the new stuff VMware is creating is just plain awesome. So on the 4th of January I will be joining VMware as the new Senior PSO Consultant.

un•put•down•a•ble
Pronunciation: (un”poot-dou’nu-bul), [key]
—adj. Informal.
Adjective meaning consistently and irresistibly interesting. Typically refers to a book that is so well written and entertaining as to be difficult to (literally) put down and pause away from.

Normally a term used to describe novels, but the vSphere Quick Start Guide certainly fits the definition. Last month I was finishing three major projects and needed to write my VCDX application in one week, but somehow it kept ending up in my hands. So what’s so special about this book and how does it distinguish itself from the competition?

The book central theme is providing tips and ‘how to’s’ and it does this rather well. The book handles the traditional subjects, such like vCenter, Host, Virtual Machines, Networking and Storage. Besides the concise, easy to follow and non-ambiguous way the tips are written, I really like the minimal use of screenshots. This allowed using the (limited) space to contain as much content as possible.

Besides describing how to change settings via the Service Console CLI and the GUI, most tips also list PowerCLI and RemoteCLI example scripts. Incorporating PowerCLI scripts allows this book to be of value to the more experienced administrator who is using PowerCLI or RemoteCLI to manage its environment. The examples certainly increased my interest of picking up PowerCLI.

But what really makes this book shine is the short in-depth text accompanying most of the tips and how to’s. The text contains valuable information on how certain mechanism works, what impact changing a setting can have and field experience of using certain settings. Added bonus is addressing the possibility of using third-party tools such as Dell expart, EMC powerpath VE, vwire and many others, confirming that this book is written by authors with true field experience.

I really recommend this book to anyone who is using VMware ESX. It doesn’t matter if you are a novice administrator or a seasoned consulting architect, you WILL learn something new by reading this book. During the ESX 2.5 era, anyone who was serious about his job owned the Advanced Technical Design Guide, in the current vSphere era it’s clear that this book must be on your desk.