Comments on: Sizing VMs and NUMA nodes

By: Benjamin Athawes

Benjamin Athawes — Wed, 09 Nov 2011 22:42:07 +0000

Hi Frank,

Just to let you know, I referred to “Figure 1: Local and Remote memory access” on my own blog over at http://www.benjaminathawes.com/blog/Lists/Posts/Post.aspx?ID=28.

I hope you don’t mind.
Ben

By: VCAP-DCA Study notes – 3.1 Tune and Optimize vSphere Performance | www.vExperienced.co.uk

Sat, 16 Apr 2011 11:34:02 +0000

[...] be monitored using esxtop (Frank Denneman’s post shows [...]

By: VMware vSphere: Manage for Performance Course Experience « TheSaffaGeek

VMware vSphere: Manage for Performance Course Experience « TheSaffaGeek — Mon, 21 Feb 2011 09:58:01 +0000

[...] your virtual machines due to NUMA. Frank Denneman has done two brilliant postings all about the sizing of VM’s and NUMA Nodes and ESX 4.1 NUMA Scheduling which covers pretty much everything you need to know about this feature [...]

By: La fête de NUMA ou pourquoi il faut réfléchir un peu avant de créer une VM

La fête de NUMA ou pourquoi il faut réfléchir un peu avant de créer une VM — Thu, 17 Feb 2011 06:31:57 +0000

[...] lire : http://frankdenneman.nl/2010/02/sizing-vms-and-numa-nodes/ & http://blogs.msdn.com/b/virtual_pc_guy/archive/2007/05/16/numa-and-virtual-server.aspx [...]

By: Shambaboy

Shambaboy — Wed, 16 Feb 2011 17:02:55 +0000

Thanks for this article,

you wrote

“At this moment, AMD and Intel offer Quad Core CPU’s…”

Do you have some experiances to share about hexa core CPUs please ?

My Best

By: Michael

Michael — Thu, 11 Nov 2010 04:58:01 +0000

There are some situations where the vSphere scheduler doesn’t schedule optimally on NUMA systems. To improve the scheduling we’ve recommended disabling NUMA Round Robin in some cases (esxcfg-advcfg -s 0 /Numa/RoundRobin), and in Brandon’s case also recommended sizing VM’s so the vCPU’s are easily divisible by the physical core count. So on a Hex Core system that would be 1,2,3, or 6 vCPU’s per VM only, on a quad core system 1, 2 and 4 vCPU per VM. Doing both resulted in greatly improved CPU ready times. Best practice for performance is as you say, make sure VM Memory size is less than NUMA node size. But without some changes there are some situations where node migration doesn’t occur when it should.

By: Brandon

Brandon — Tue, 02 Nov 2010 17:09:56 +0000

@Frank, so for slow reply, ive moved on from the AMD/NUMA site and am currenlty at an Intel/NON NUMA site. However given all new Intel are NUMA I would like to see an answer.

The VM’s are a mixtures of large SQL (8 x 12-20GB RAM), medium (30 x 4GB citrix), and rats and mice (100 x 500mb – 2gb RAM).

In my un-educated guess, it would seem that numa rebalance DOESNT look at CPU ready time, and to be honest, this is the only metric that is under pressure. EG plenty of RAM and MHZ left in every node.

As mentioned its only when I run out of physical memory in a node, that a new node gets used…..

By: acai berry optimum

acai berry optimum — Fri, 22 Oct 2010 01:50:48 +0000

I admire someone that takes the pride you have and with your projecton of information. oSo when i actually do sit down to read material, I appreciate well written and organized blogs like this one. I have it bookmarked and will be back. Thanks.

By: acai berry optimum

acai berry optimum — Fri, 22 Oct 2010 01:47:36 +0000

Writing a paper for my summer class this helped lots will credit the site thanks!

By: Frank Denneman

Frank Denneman — Tue, 19 Oct 2010 18:35:19 +0000

Alex, you are aware that you disabled NUMA with this setting right? What might help, is to adjust the Numa.MigImbalanceThreshold setting.
The Numa.MigImbalanceThreshold setting controls the minimum load imbalance between the NUMA nodes needed to trigger a virtual machine migration.
Default this advanced parameter is set to 10 percent, but in your case you might try a lower percentage.

Be aware that the CPU scheduler takes several metrics into account when calculating the CPU imbalance. Metrics such as the CPU entitlement and usage.
Which makes me wonder if both you and Brandon set CPU reservations on the virtual machines?

As always, please use the default settings as much as possible, only change advanced settings when absolutely necessary!