A big part of resource management is sizing of the virtual machines. Right-sizing the virtual machines allows IT teams to optimize the resource utilization of the virtual machines. Right sizing has become a tactical tool for enterprise IT-teams to ensure maximum workload performance and efficient use of the physical infrastructure. Another big part of resource management is keeping track of resource utilization, some of these processes are a part of the daily operation tasks performed by specialized monitoring teams or the administrators themselves. Service Providers usually cannot influence the right sizing element, therefor they focus more on the monitoring part. What is almost universal across virtual infrastructure owners is the incidental nature of tracking down ‘noisy-neighbors’ VMs . Noisy neighbor VMs generate workload in such a way that it monopolizes resources and have negative impact on the performance of other virtual machines. Service Providers and enterprise IT teams have to deal with these consumer outliers in order to meet the SLAs of existing workloads and being able to satisfy the SLA requirements of new workloads.
It’s interesting that noisy neighbor tracking is an incidental activity as it can be so detrimental to the performance of the virtual datacenter. Tools such as vSphere Storage IO Control (short term focus) and vSphere Storage DRS (long term focus) assist to alleviate the infrastructure from the burden of noisy neighbors, but attacking this problem structurally is necessary to ensure consistent and predictable performance from your infrastructure. At long term, noisy neighbor VMs impact the projected consolidation ratio, which in turn influences the growth rate of the infrastructure. I’ve seen plenty of knee jerk reactions, creating a server and storage infrastructure sprawl due to introduction of these outlier workloads.
Identifying noisy neighbors can become a valuable tool in both strategic and tactical playbooks of the IT organization. Having insight of which VMs are monopolizing the resources allow IT teams to act appropriately. Similar to real life the behavior of noisy neighbor can be changed often, but sometimes that’s the nature of the beast and you just have to live with it. In that situation noisy neighbors become outliers of conduct and one ha to make external adjustments. This insight allows IT teams to respond along the entire vertical axis of the virtual datacenter, from application to infrastructure choice. By having the correct analysis, the IT team can provide insights to the application owner, allowing them to adjust accordingly. It helps the IT team to understand whether the environment can handle the workload and make adjustment to the infrastructure necessary. Sometimes the intensity of the workload is just what it is and hosting that workload is necessary to support the business. In that case the IT team has to understand whether the infrastructure is suitable to support the application. As most IT organization have access to multiple platforms, the accurate insight of characteristics (and requirements) of the workload allows them to identify the correct platform.
Virtual Datacenters are difficult to monitor. They are comprised of a disparate stack of components. Every component logs and presents data differently. Different granularity of information, different time frames, and different output formats make it extremely difficult to correlate data. In addition you need to be able to correctly identify the workload characteristics and interpret the impact it has on the shared environment. We do not live in a world anymore where we have to deal with isolated technology stacks. Applications typically do not run anymore on a single box, connected to a single and isolated raid array. Today everything within the infrastructure is shared, the level of hardware resource distribution is diluting with each introduction of new hardware. Where we used to run a single application in a VM on top of server with ten other VMs, sharing a couple of NICs and HBA’s, we slowly moved towards converged network platforms. In the last 10 years, we shared and shared more, the only monolith remaining is the application in the VM and that is rapidly changing as well with the popularity of containers and micro services. Yet most of our testing mechanisms and monitoring efforts are still based on the architecture we left behind 10 years ago. Virtual Datacenters require continuous analytics that fully comprehends the context of the environment, with the ability to zoom in and focus on outliers if necessary.
In the upcoming series I’m going to focus on how to explore cluster level workloads and progressively zooming into specific workloads based on IOPS, block size, throughput and unaligned IOs.
Category: Uncategorized
-
Tracking down noisy neighbors
-
Let Cloudphysics help rid yourself of Heartbleed
Unfortunately the Open SSL Heartbleed bug (CVE-2014-0224) is present in the ESXi and vCenter 5.5 builds. VMware responded by incorporating a patch to solve the OpenSSL vulnerability in the OpenSSL 1.0.1 library. For more info about the ESXI 5.5 patch read KB 2076665, VMware issued two releases for vCenter 5.5, read KB 2076692.
Unfortunately some NFS environments experienced connection loss after applying the ESXi 5.5 patch, VMware responded by releasing patch 2077360 and more recently vCenter update 1b. The coverage on the NFS problems and the amount of ESX and vCenter update releases to fix a bunch of problems may left organizations in the dark whether they patched the Heartbleed vulnerability. Cloudphysics released a free Heartbleed analytic card in their card store that helps identify which hosts in your environment are unprotected.
Check out the recent article of Cloudphysics CTO, Irfan Ahmad about their recently released Heartbleed analytic package. I would recommend to run the card and rid yourself of this nasty bug. -
Homelab – Power-on your Supermicro system by SSH'ing into IPMI
Just a short article, recently I discovered you can access Supermicro IPMI via SSH and power on the system by using the command:
start /system1/pwrmgtsvc1
A nice short command that saves you a lot of time by eliminating the need to log in the webUI and wait until the app responds. -
My lab and the birth of the portable Ikea lack 19” datacenter rack
Currently, topics about labs are hot, and when meeting people at the VMUGs or other tech conferences, I get asked a lot about my lab configuration. I’m a big fan of labs, and I think everybody who works in IT needs a lab, whether it’s at home or in a centralized location.
At PernixData, we have two major labs. One on the east coast and one on the west coast of the U.S. Both these labs are shared, so you cannot do everything you like. However, sometimes you want to break stuff. You want to pull cables and disks and kill an entire server or array. To see what happens. For these reasons having a lab that is 4000 miles away doesn’t work. Enough reasons to build a small lab at home.
Currently topic about labs are hot and when meeting people at the VMUGs or other tech conferences, I get asked a lot about my lab configuration. I’m a big fan of labs and I think everybody who works in IT needs a lab, whether it’s at home or in a centralised location.
At PernixData we have two major labs. One at the east coast and one at the west coast of the U.S of A. Both these labs are shared, that means you cannot do everything you like. However sometimes you want to break stuff, you want to pull cables, disks, kill an entire server or array. Just to see what happens. For these reasons having a lab that is 4000 miles away doesn’t really work, enough reasons to build a small lab at home.Nested or physical hardware?
To nest or not to nest, that’s not even the question. Nesting is amazing, and VMware spends a lot of energy and time on nested environments (think HOL). Recently the fling VMware tools for Nested ESXi was released, and I assume more nested ESXi flings will follow after seeing the attention it received from the community.But to run nested ESXi, you need to have physical hardware. Thanks to a generous donation, I received 6 Dell r610s, which covered my compute level requirements. But sometimes, you only want to test the software, and in those cases, you do not need to fire up an incredibly loud semi-datacenter rig. For those situations, I created an ESXi host that is near silent when running full speed. This ESXi server also hosts a nested ESXi environment and is just a white box with a simple ASUS mobo, 24GB, and the Intel 1GB Ethernet port. Once this machine is due for renewal, a white box following the baby dragon design will replace it.
To test the software at the enterprise level, you require multiple levels of bandwidth, sometimes the bare minimum and sometimes copious amounts of it. The R610 sports 4 x 1GB Ethernet connections, allowing me to test scenarios that can happen in a bandwidth-constrained environment. Usually, compelling cases happen when you have a lot of restrictions to deal with, and these 1GB NICs are perfect for this. 10GB connections are on my wish list, but to have a nice setup, you still need to invest more than 1000 bucks in testing it adequately.
A little bit over the top for my home lab, but the community came to the rescue and provided me with a solution; the Infiniband hack. A special thanks go out to Raphael Schitz and Eric Bussink for providing me the software and the information to run my lab at 10Gbps and being able to provide incredibly low latencies to my virtual machines. With the InfiniBand setup, I can test scenarios where bandwidth is not a restriction and investigate specific setups and configurations. For more info, listen to the vBrownbag tech talk where Erik Bussink dives into the topic “InfiniBand in the Lab“
The storage layer is provided by some virtual storage appliances, each backed by a collection of different SSD disks and WD Black Caviar 750GB disks. Multiple solutions allow me to test various scenarios such as all-flash arrays, hybrid, and all magnetic disk arrays. If I need to understand the specific dynamics of an array, I log in to one of the two US-based labs.
Home office
My home office is designed to be an office and not a data center. So where do you place 19″ rack servers without ruining the esthetics of your minimalistic designed home office ;). Well, you create a 19″ rack on wheels so you can roll it out of sight and place it wherever you want it. Introducing the portable Ikea lack 19″ datacenter rack.
Regular readers of my blog or Twitter followers know I’m a big fan of hacking IKEA furniture. I created a whiteboard desk that got the attention of multiple sites and ikeahackers.net provided me with a lot of ideas on how to hack the famous lack table side table.I bought two lack tables, a couple of L-shaped brackets, four wheels, nuts, and bolts. The first lack table provides the base platform. Only the tabletop is used. The legs are discarded and act as a backup if I make a mistake during the drilling.
I didn’t test the center of the tabletop, but the corners of the tabletop are solid and can be used to install wheels. I used heavy-duty ball-bearing wheels with an offset swivel caster design that permits ease of directional movement. Simple 5mm nuts and bots keep the L shape brackets in place, but beware, the table legs are not made of solid wood. They are hollow! Only a few centimeters of the top of the leg is solid. This to hold the screw that connects the table and leg. To avoid having the server pull the screw through the leg due to its weight, I used washers to keep them in place
What’s next?
From a hardware perspective, 10GbE is still high on my wishlist. When looking at the software layer, I want to create a more automated way of deploying and testing PernixData FVP software. One of the things I’m looking into is using and incorporating Auto Deploy in the lab. But that’s another blog post.t. -
Only two days left to sign up for your VMworld speaker shirt!
During our advisory call with CloudPhysics, a great idea was born. Why not provide all the speakers at VMworld a cool speaker shirt? Unfortunately in 2011 VMworld made the decision to stop providing speaker shirts to the people on stage, so most people started wearing older speaker shirt or even RUN DRS shirts ☺. For most speakers this move by VMworld was disappointing as the speaker shirts made them more recognizable but it also served as a cool badge of honor.
I think CloudPhysics stepped up big time and gave us back that cool badge of honor. If you are a speaker this year, go register here before tomorrow evening as the deadline is Tuesday August 13th end of day(pst). -
Have you seen the new video introduction to PernixData?
I know PernixData FVP is cool, you know it’s cool, now how can you tell you all your friends and colleagues about it? Going around and telling them one by one doesn’t scale, just show them this video!
http://www.youtube.com/watch?v=_t5b61xtT4Q
I think Jeff sums it up nicely:
-
Want to have my former job?
My old technical marketing team of VMware is looking for someone to cover resource management. If you have a passion for resource management of virtual infrastructures and like to help VMware’s field personnel, partners and customers understand the technology then this job might be something for you.
A large part of my role was bridging between engineering, product management, product marketing and the field / customers. Provide information to the R&D side of VMware how the products are used and what features customers are requesting. You create collateral in every way or form to help the customer and field personnel understand and adopt the features.
I always enjoyed working with the different teams at VMware. The cloud resource management and vMotion team are an awesome group to work with. Be prepared to deep dive with these guys Marianas trench style. Having a customer facing background helps you provide the team valuable information to align the features to the customer wishes.
In this role you assist product marketing and product management in achieving their tactical and strategic plans.
Besides working with the responsible engineering and product marketing teams you collaborate with your technical marketing colleagues. You have the ability to interact with guys such as Ken Werneberg, Cormac Hogan, Mike Foley, William Lam, Alan Renouf or Rawlinson Rivera on a daily basis.
If you have thorough understanding of the vMotion features, DRS, Storage DRS, SIOC and DPM and love to help customers adopt these features, then apply now!
http://jobs.vmware.com/job/Palo-Alto-Sr_-Technical-Marketing-Manager-Resource-Management-Job-CA-94301/2593496/
Please note
Be aware that this is my former role and that I no longer work for VMware. Therefor I cannot answer any further inquiries. Please contact the VMware career team. -
Cloudphysics webinar: Expert Tips for Managing Datastore Space in a vSphere Environment
Tonight I will join Erik Haus and Krishna Raj Raja of CloudPhysics to talk about Datastore space management in a virtual infrastructure.
During the webinar Krishna will show you how the new CloudPhysics “Datastore Space” card and “Snapshots Gone Wild” card help you to identify and resolve space problems.
Join us!
The event will start at 6:00 pm Amsterdam Time on May 9, 2013. (9:00 AM PDT)
Go to the CloudPhysics site to register! -
Migrating VMs between DRS clusters in an elastic vDC
In the article “Migrating datastore clusters by changing storage profiles in a vCloud“ I closed with the remark that vCD is not providing an option to migrate virtual machines between compute clusters that are part of an elastic vDC. Fortunately my statement was not correct. Tomas Fojta pointed out that vCD does provide this functionality. Unfortunately this feature is not exposed in the vCloud organization portal but in the system portal of the vCloud infrastructure itself. In other words, to be able to use this functionality you need to have system administrator privileges.
In the previous article, I created the scenario where you want to move virtual machines between two sites. Site 1 contains compute cluster “vCloud-Cluster1” and datastore cluster “ DSC-Site-1”. Site 2 contains “vCloud-Cluster2” and datastore cluster “DSC-Site-2” . By changing the VM storage profile from Site-1 to Site-2, we have vCD instruct vSphere to storage vMotion the virtual machine disk files from one datastore cluster to another. Now at this point we need to migrate the compute state of the virtual machine.
Migrate virtual machine between clusters
Please note that vCD refers to clusters as resource pools. To migrate the virtual machine between clusters, log into the vCloud director and select the system tab. Go to the vSphere resources and select Resource Pools menu option.
The UI displays the clusters that are a part of the Provide vDC. Select the cluster a.k.a. resource pool in which the virtual machine resides. Select the virtual machine to migrate, right click the virtual machine to have vCD display the submenu and select the option “Migrate to…”
The user interface allows you to choose how you want to select the destination resource pool for the virtual machine: Either automatic and let vCD select the resource pool for you, or select the appropriate resource pool manually. When selecting automatic vCD selects the cluster with the most unreserved resources available. If the virtual machine happens to be in the cluster with the most unreserved resources available vCD might not move the virtual machine. In this case we want to place the virtual machine in site 2 so that means we need to select the appropriate cluster. We select vCloud-Cluster2 and click on OK to start the migration process.
vCD instructs vSphere to migrate the virtual machine between clusters with the use of vMotion. In order to use vMotion, both clusters need to have access to the datastore on which the virtual machine files reside. vCD does not use “enhanced’ vMotion where it can live migrate between host without being connected shared storage. Hopefully we see this enhancement in the future. When we log into vSphere we can verify if the life migration of the virtual machine was completed.
Select the destination cluster, in this case that would be vCloud-Cluster2, go to menu option Monitor, select tasks and click on the entry “Migrate virtual machine”
In the lower part of the screen, you get more detailed information of the Migrate-virtua-machine entry. As you can seem the virtual machine W2K8_RS_SP1 is migrated between servers 10.27.51.155 and 10.27.51.152. As we do not change anything to the storage configuration, the virtual machine files remains untouched and stay on the same datastore.
To determine if vCD has updated the current location of the virtual machine, log into vCD again, go to the menu option “Resource Pools” and select the cluster chosen as destination as the previously org cluster.
-
Hello world! Again
During my holiday, frankdenneman.nl got some unwanted attention.
I’m currently in the process of rebuilding the site.
Stay tuned for new updates!
