DRS RESOURCE DISTRIBUTION CHART
A customer of mine wanted more information about the new DRS Resource Distribution Chart in vCenter 4.0, so I thought after writing the text for the customer, why not share this? The DRS Resource Distribution Chart was overhauled in vCenter 4.0 and is quite an improvement over the resource distribution chart featured in vCenter 2.5. Not only does it use a better format, the new charts produce more in-depth information.
RESOURCE POOLS AND AVOIDING HA SLOT SIZING
Virtual machines configured with large amounts of memory (16GB+) are not uncommon these days. Most of the time these “heavy hitters” run mission critical applications so it’s not unusual setting memory reservations to guarantee the availability of memory resources. If such a virtual machine is placed in a HA cluster, these significant memory reservations can lead to a very conservative consolidation ratio, due to the impact on HA slot size calculation. (For more information about slot size calculation, please review the HA deep dive page on yellow-bricks.com.) There are options to avoid creation of large slot sizes. Such as not setting reservations, disabling strict admission control, using vSphere new admission control policy “percentage of cluster resources reserved” or creating a custom slot size by altering the advanced settings das.vmMemoryMinMB. But what if you are still using ESX 3.5, must guarantee memory resources for that specific VM, do not want to disable strict admission control or don’t like tinkering with the custom slot size setting? Maybe using the resource pool workaround can be an option. Resource pool workaround During a conversation with my colleague Craig Risinger, author of the very interesting article “The resource pool priority pie paradox”, we discussed the lack of relation between resource pools reservation settings and High Availability. As Craig so eloquently put it:
IMPACT OF HOST LOCAL VM SWAP ON HA AND DRS
On a regular basis I come across NFS based environments where the decision is made to store the virtual machine swap files on local VMFS datastores. Using host-local swap can affect DRS load balancing and HA failover in certain situations. So when designing an environment using host-local swap, some areas must be focused on to guarantee HA and DRS functionality. VM swap file Lets start with some basics, by default a VM swap file is created when a virtual machine starts, the formula to calculate the swap file size is: configured memory – memory reservation = swap file. For example a virtual machine configured with 2GB and a 1GB memory reservation will have a 1GB swap file. Reservations will guarantee that the specified amount of virtual machine memory is (always) backed by ESX machine memory. Swap space must be reserved on the ESX host for the virtual machine memory that is not guaranteed to be backed by ESX machine memory. For more information on memory management of the ESX host, please the article on the impact of memory reservation. During start up of the virtual machine, the VMkernel will pre-allocate the swap file blocks to ensure that all pages can be swapped out safely. A VM swap file is a static file and will not grow or shrink not matter how much memory is paged. If there is not enough disk space to create the swap file, the host admission control will not allow the VM to be powered up. Note: If the local VMFS does not have enough space, the VMkernel tries to store the VM swap file in the working directory of the virtual machine. You need to ensure enough free space is available in the working directory otherwise the VM is still not allowed to be powered up. Let alone ignoring the fact that you initially didn’t want the VM swap stored on the shared storage in the first place. This rule also applies when migrating a VM configured with a host-local VM swap file as the swap file needs to be created on the local VMFS volume of the destination host. Besides creating a new swap file, the swapped out pages must be copied out to the destination host. It’s not uncommon that a VM has pages swapped out, even if there is not memory pressure at that moment. ESX does not proactively return swapped pages back into machine memory. Swapped pages always stays swapped, the VM needs to actively access the page in the swap file to be transferred back to machine memory but this only occurs if the ESX host is not under memory pressure (more than 6% free physical memory). Copying host-swap local pages between source- and destination host is a disk-to-disk copy process, this is one of the reasons why VMotion takes longer when host-local swap is used. Real-life scenario A customer of mine was not aware of this behavior and had discarded the multiple warnings of full local VMFS datastores on some of their ESX hosts. All the virtual machines were up and running and all seemed well. Certain ESX servers seemed to be low on resource utilization and had a few active VMs, while other hosts were highly utilized. DRS was active on all the clusters, fully automated and a default (3 stars) migration threshold. It looked like we had a major DRS problem. DRS If DRS decide to rebalance the cluster, it will migrate virtual machines to low utilized hosts. VMkernel tries to create a new swap file on the destination host during the VMotion process. In my scenario the host did not contain any free space in the VMFS datastore and DRS could not VMotion any virtual machine to that host because the lack of free space. But the host CPU active and host memory active metrics were still monitored by DRS to calculate the load standard deviation used for its recommendations to balance the cluster. (More info about the DRS algorithm can be found on the DRS deepdive page). The lack of disk space on the local VMFS datastores influenced the effectiveness of DRS and limited the options for DRS to balance the cluster. High availability failover The same applies when a HA isolation response occurs, when not enough space is available to create the virtual machine swap files, no virtual machines are started on the host. If a host fails, the virtual machines will only power-up on host containing enough free space on their local VMFS datastores. It might be possible that virtual machines will not power-up at-all if not enough free disk space is available. Failover capacity planning When using host local swap setting to store the VM swap files, the following factors must be considered. • Amount of ESX hosts inside cluster. • HA configured host failover capacity. • Amount of active virtual machines inside cluster. • Consolidation ratio (VM per host). • Average swap file size. • Free disk space local VMFS datastores.
VCDX NUMBER 029
Monday 8th of February I was scheduled to participate in the defend session of the VCDX panel at Las Vegas. For people not familiar with the VCDX program, the defend panel is the final part of the extensive VCDX program. My defend session was the first session of the week, so my panel members where fresh and eager to get started. Besides the three panel members, an observer and a facilitator where also present in the room. The session consisted out of three parts; • Design defend session (75 minutes) • Design session (30 minutes) • Troubleshooting session (15 minutes) During the design defend session you are required to present your design, I used a twelve deck slide presentation and included all blueprints\Visio drawings as appendix. This helped me a lot, as I am not a native English speaker using diagrams helped me to explain the layout. There is no time limit on the duration of the presentation, but it is wise to keep it as brief as possible. During the session, the panel will try to address a number of sections and if they cannot address these sections this can impact your score. The design and troubleshooting session you need to show you are able to think on your feet. One of the goals is to understand your though process. Thinking out loud and using the whiteboard will help you a lot. So how was my experience? After meeting my panel members I started to get really nervous as one of the storage guru’s within VMware was on my panel. The other two panel members have an extreme good track record inside the company as well, so basically I was being judged by an all-star panel. I thought my presentation went well, but word of advice; read your submitted documentation on a regular basis before entering the defend panel as the smallest details can be asked. After completing the design defend pane, I was asked to step outside. After the short break the design session and troubleshooting scenarios were next. I did not solve the design and troubleshooting scenarios, but that is really not the goal of those sections. Thinking out loud in English can be challenging for non-native English speakers, so my advice is to try to practice this as much as possible. I did a test presentation for a couple of friends and discovered some areas to focus on before doing the defend part of the program. After completing my defend panel, I was scheduled to participate as an observer on the remaining defend panel sessions the rest of the week. After multiple sessions as an observer and receiving the news that I passed the VCDX defend panel, I participated as a panel member on a defend session. Hopefully I will be on a lot more panels in the upcoming year, because sitting on the other side of the table is so much better than standing in front of it sweating like a pig. :)
SIZING VMS AND NUMA NODES
Note: This article describes NUMA scheduling on ESX 3.5 and ESX 4.0 platform, vSphere 4.1 introduced wide NUMA nodes, information about this can be found in my new article: ESX4.1 NUMA scheduling With the introduction of vSphere, VM configurations with 8 CPUs and 255 GB of memory are possible. While I haven’t seen that much VM’s with more than 32GB, I receive a lot of questions about 8-way virtual machines. With today’s CPU architecture, VMs with more than 4 vCPUs can experience a decrease in memory performance when used on NUMA enabled systems. While the actually % of performance decrease depends on the workload, avoiding performance decrease must always be on the agenda of any administrator.
TOP 25 VIRTUALIZATION BLOGGERS
This week Eric Siebert processed all the votes and published this year’s Top 25 VMware/Virtualization Blogger list. Over 700 people voted, each casting 10 votes. I can only imagine the work involved that is put in to producing this list, so a big thank you goes out to Eric for voluntary organizing this! Awesome! This year a lot of new names entered the top 25 including my blog. I never ever expected to see my name published in the top 25. I’m truly honored to make it to the list, let alone be voted number #14 so I would like to thank everyone for voting for me! I really appreciate it! Congratulations to all other people mentioned in the list and I would like to congratulate Duncan Epping specifically for taking the number 1 place again this year. The top 25 as published by Eric Siebert on vSphere-land:
VOTING CLOSES AT FRIDAY!
Eric Siebert of vsphere-land.com started a new election of the best 20 bloggers in the VMware and Virtualization scene. Because even more top blogs got started in 2009, Eric decided to expand the top 20 to the top 25. To my suprise, Eric decided to nominate my blog as well. I’m really honored to be a nominee amongst the best virtualization bloggers out there. Unfortunately the list Eric is longer that the 10 votes one can cast, so good luck picking the ones who stand out above the excellent crowd. This is my top 10 blogs; Duncan Epping Chad Sakac Kenneth van Ditmarsch Alan Renouf Scott Lowe Scott Drummonds Hypervizor (Hany Michael) Arnim van Lieshout Arne Fokkema Gabe Virtual world. Go vote now before it’s too late! http://vsphere-land.com/news/time-to-vote-for-your-favorite-bloggers.html
JOINING VMWARE
The year 2009 has been an interesting year. After leaving a long-term position, I participated in some really awesome projects, got to fiddle around with the cutting edge technology and got to work with some really excellent and inspiring people. Begin August I got an e-mail from Duncan Epping If I would like to do some contractors work for VMware. As you can imagine, it didn’t take me long to respond with a Font size 72 YES. (Do you know you can make text blink in word?) After completing a few project VMware offered me a permanent job, being a contractor for 9 years made the decision a bit tougher, but getting such a job offer is something you can hardly refuse. Working with the best of the business, being able to access internal information and getting exposed to all the new stuff VMware is creating is just plain awesome. So on the 4th of January I will be joining VMware as the new Senior PSO Consultant.
WRAP-UP 2009
Daniel Easons blog post inspired me to write a wrap-up of 2009 myself. Beside the career move described in the previous post, 2009 was a year of finding two new addictions. Blogging and twitter(@frankdenneman). Beginning of February I started blogging and the first article was received pretty well. The article got mentioned on Yellow Bricks the same day. Now 10 months later, more than 37.000 people visited the site. It cannot hold a candle to the great blogs out there, but it’s nice start. Some of my articles appeared in a few Top 5 Planetv12n lists, got mentioned on Yellow Bricks and were featured in Scott Lowe’s virtualization short takes. Trying to create in-depth articles is an excellent way to learn stuff. Most of the time describing a certain subject somehow challenged my current knowledge of that topic and ended up spending ridiculous amounts of time researching that particular subject. More often than not coming across very interesting material not really related to the subject but similarly interesting, consuming even more time. Some articles are more popular than others; these are the top five visited articles this year; 1. Increasing the queue depth 2. Lefthand SAN – Lessons learned 3. HP Continuous Access and the use of LUN balancing scripts 4. Impact of memory reservations 5. NFS and IP-HASH Load-Balancing Lately I’m running into a few limitations of the free wordpress blog themes, that’s why I’ve decided to move to another site, stay tuned for the URL. I’m aiming to release the new site at the beginning of next year.
VSPHERE 4.0 QUICK START GUIDE REVIEW
un•put•down•a•ble Pronunciation: (un"poot-dou’nu-bul), [key] —adj. Informal. Adjective meaning consistently and irresistibly interesting. Typically refers to a book that is so well written and entertaining as to be difficult to (literally) put down and pause away from. Normally a term used to describe novels, but the vSphere Quick Start Guide certainly fits the definition. Last month I was finishing three major projects and needed to write my VCDX application in one week, but somehow it kept ending up in my hands. So what’s so special about this book and how does it distinguish itself from the competition? The book central theme is providing tips and ‘how to’s’ and it does this rather well. The book handles the traditional subjects, such like vCenter, Host, Virtual Machines, Networking and Storage. Besides the concise, easy to follow and non-ambiguous way the tips are written, I really like the minimal use of screenshots. This allowed using the (limited) space to contain as much content as possible. Besides describing how to change settings via the Service Console CLI and the GUI, most tips also list PowerCLI and RemoteCLI example scripts. Incorporating PowerCLI scripts allows this book to be of value to the more experienced administrator who is using PowerCLI or RemoteCLI to manage its environment. The examples certainly increased my interest of picking up PowerCLI. But what really makes this book shine is the short in-depth text accompanying most of the tips and how to’s. The text contains valuable information on how certain mechanism works, what impact changing a setting can have and field experience of using certain settings. Added bonus is addressing the possibility of using third-party tools such as Dell expart, EMC powerpath VE, vwire and many others, confirming that this book is written by authors with true field experience. I really recommend this book to anyone who is using VMware ESX. It doesn’t matter if you are a novice administrator or a seasoned consulting architect, you WILL learn something new by reading this book. During the ESX 2.5 era, anyone who was serious about his job owned the Advanced Technical Design Guide, in the current vSphere era it’s clear that this book must be on your desk.