Free vSphere 6.5 Host Resources Deep Dive E-Book

In June of this year, Niels and I published the vSphere 6.5 Host Resources Deep Dive, and the community was buzzing. Twitter exploded, and many community members provided rave reviews.

This excitement caught Rubriks attention, and they decided to support the community by giving away 2000 free copies of the printed version at VMworld. The interest was overwhelming, before the end of the second signing session in Barcelona we ran out of books.

A lot of people reached out to Rubrik and us to find out if they could get a free book as well. This gave us an idea, and we sat down with Rubrik and the VMUG organization to determine how to cater the community.

We are proud to announce that you can download the e-book version (PDF only) for free at rubrik.com. Just sign up and download your full e-book copy here.

Spread the word! And if you like, thank @Rubrik and @myVMUG for their efforts to help the VMware community advance.

What if the VM Memory Config Exceeds the Memory Capacity of the Physical NUMA Node?

This week I had the pleasure to talk to a customer about NUMA use-cases and a very interesting config came up. They have a VM with a particular memory configuration that exceeds the ESXi host NUMA node memory configuration. This scenario is covered in the vSphere 6.5 Host Resources Deep Dive, excerpt below.

Memory Configuration
The scenario described happens in multi-socket systems that are used to host monster-VMs. Extreme memory footprint VMs are getting more common by the day. The system is equipped with two CPU packages. Each CPU package contains twelve cores. The system has a memory configuration of 128 GB in total. The NUMA nodes are symmetrically configured and contain 64 GB of memory each.

However, if the VM requires 96 GB of memory, a maximum of 64 GB can be obtained from a single NUMA node. This means that 32 GB of memory could become remote if the vCPUs of that VM can fit inside one NUMA node. In this case, the VM is configured with 8 vCPUs.

The VM fits from a vCPU perspective inside one NUMA node, and therefore the NUMA scheduler configures for this VM a single virtual proximity domain (VPD) and a single a load-balancing group which is internally referred to as a physical proximity domain (PPD).

Example Workload
Running a SQL DB on this machine resulted in the following local and remote memory consumption. The VM consumes nearly 64 GB on its local NUMA node (clientID shows the location of the vCPUs) while it consumes 31 GB of remote memory.

In this scenario, it could be beneficial to the performance of the VM to rely on the NUMA optimizations that exist in the guest OS and application. The VM advanced setting numa.consolidate = FALSE instructs the NUMA scheduler to distribute the VM configuration across as many NUMA nodes as possible.

In this scenario, the NUMA scheduler creates 2 load-balancing domains (PPDs) and allows for a more symmetrical configuration of 4 vCPUs per node.

Please note that a single VPD (VPD0) is created and as a result, the guest OS and the application only detect a single NUMA node. Local and remote memory optimizations are (only) applied by the NUMA scheduler in the hypervisor.

Whether or not the application can benefit from this configuration depends on its design. If it’s a multi-threaded application, the NUMA scheduler can allocate memory closes to the CPU operation. However, if the VM is running a single-threaded application, you still might end up with a lot of remote memory access, as the physical NUMA node hosting the vCPU is unable to provide the memory demand by itself.

Test the behavior of your application before making the change to create a baseline. As always, use advanced settings only if necessary!

A vSphere Focused Guide to the Intel Xeon Scalable Family – Memory Subsystem

The Intel Xeon Scalable Family introduces a new platform (Purley). The most prominent change regarding system design is the memory subsystem.

More Memory Bandwidth and Consistency in Speed
The new memory subsystem supports the same number of DIMMs per CPU as the previous models. However, it’s wider and less deep. What I mean by that is that the last platform (Grantley) supported up to three DIMMs per channel (DPC) and made use of four channels. In total, the Grantley platform supported up to twelve DIMMs per CPU. Purley increases the number of channels from four to six but reduces the numbers of supported DIMMs per channel from three to two. Although this sounds like a potato, potato; tomato, tomato discussion it provides a significant increase in bandwidth while ensuring consistency in speed during a scaling up exercise. Let’s take a closer look.

DIMMs per Memory Channel
Depending on the DIMM slot configuration of the server board, multiple DIMMs are supported per channel. The E5-2600 V-series supports up to 3 DIMMs per channel (3 DPC). Using more DIMMs per channel provides the largest capacity, but unfortunately, it impacts the operational speed of memory.

A DIMM groups memory chips into ranks. DIMMs come in three rank configurations; single-rank, dual-rank or quad-rank configuration, ranks are denoted as (xR). With the addition of each rank, the electrical load on the channel increases. And as more ranks are used in a memory channel, memory speed drops restricting the use of additional memory. Therefore in certain configurations, DIMMs will run slower than their listed maximum speeds. This reduction in speed occurs when 3 DIMMs per channel is used.

RDIMM 1 DPC 2 DPC 3 DPC LRDIMM 1 DPC 2 DPC 3 DPC Source
Cisco 2400 MHz 2400 MHz 1866 MHz 2400 MHz 2400 MHz 2133 MHz Cisco PDF
Dell 2400 MHz 2400 MHz 1866 MHz 2400 MHz 2400 MHz 2133 MHz Dell.com
Fujitsu 2400 MHz 2400 MHz 1866 MHz 2400 MHz 2400 MHz 1866 MHz Fujitsu PDF
HP 2400 MHz 2400 MHz 1866 MHz 2400 MHz 2400 MHz 2400 MHz* HP PDF
Performance Drop 0 0 28% 0 0 12%/28%

* HP claims no reduction of speed due to proprietary memory technology. I have not tested this.

Moving to 2 DIMMs per Channel Configuration
The Purley platform avoids this pitfall by reducing the supported number of DIMMs per channel. It supports up to 2 DIMMs per channel, maintaining the same performance regardless the number of DIMMs per channel. However, reducing the number of DIMMs supported per channel severely impacts the total supported memory capacity per CPU. Intel solves this by adding more channels to the memory controller. For some organizations, this change can result in dropping the requirement of obtaining LRDIMMs. Some organizations avoid the steep performance reduction by purchasing the (more) expensive LRDIMMs, 2 DPC configurations will not affect the performance characteristics of the memory modules.

Six Memory Channel Support
The Purley platform supports up to six channels per CPU. As a result, the bandwidth increases and the support for high capacity memory systems remains. By default, the new Xeon CPU supports up to 768 GB. Please review part 1 of this series in which covers the high memory capacity optimization option (M-suffix).

If all six channels are populated with DIMMs, the CPU interleaves memory access across the multiple memory channels. When creating a 1 DIMM per channel (1 DPC) configuration, the CPU forms one region (Region 0) and interleaves the memory access. Theoretically, this multiplies the data rate by exactly the number of channels present. A 2666 MT/s DIMM has a theoretical peak transfer rate of 21,300 MB/s. If populating all six DIMM slots, the memory controller accesses each module sequentially. Instead of writing all the data to a single DIMM, the data is written across the modules in one region in an alternating pattern, leveraging each channel bandwidth separately. That means that the memory controllers of a single Xeon CPU have access to a combined bandwidth of 127,800 MB/s. In theory, that means that a dual Xeon system has access to 256 GB per second (21,300 MB/s x 6 channels x 2 sockets). In theory!

This all depends on the type of workload and the compute power that drives the workload. The Xeon’s cores have direct access to the six channels in the CPU package. One thread can never obtain 256 GB due to the interconnect and the raw power it can produce to feed the channels. Anandtech has an excellent write-up about this behavior.

Memory Configuration
As a result of an increase of channels and the design consideration of populating every DIMM slot to create a 1 DPC or 2 DPC configuration, a new vSphere system will likely have a different memory capacity configuration than your previous systems (Inform your standards commission). Please note that the table lists the memory configuration of a single NUMA node.

6 x DIMM 16 GB 32 GB 64 GB 128 GB
1 DPC 96 GB 192 GB 384 GB 768 GB
2 DPC 192 GB 384 GB 768 GB 1536 GB *

* M-suffix Xeon CPU required

Please note that the table lists the memory configuration of a single NUMA node. Dual CPU systems is the most common configuration for vSphere servers. That means that you can expect the major system integrators such as DELL and HP to offer the following configurations:

Dual CPU Socket 16 GB 32 GB 64 GB 128 GB
1 DPC 192 GB 384 GB 768 GB 1536 GB
2 DPC 384 GB 768 GB 1536 GB 3072 GB *

For completeness sake, the next table shows the configuration of a v4 system with a maximum of 2-DPC. Very familiar configuration numbers, I guess we just need to get used to the new configuration standards such as 384, 768 and 1536 GB per system.

Dual CPU Socket v4 16 GB 32 GB 64 GB 128 GB
1 DPC 128 GB 256 GB 512 GB 1024 GB
2 DPC 256 GB 512 GB 1024 GB 2048 GB

* M-suffix Xeon CPU required.

vSphere 6.5 supports up to 12 TB per host. As a result, the entire range of Intel Xeon Scalable CPU with the extended memory feature is fully supported (8 CPUs x 1536 GB). Interesting data point, the vSphere 6.5 Configurations Maximum Guide started to list a maximum number of NUMA nodes per system. This limit is set to 16. The Intel Scalable Xeon supports sub-NUMA clustering (similar to Cluster-on-Die functionality), splitting up the CPU package into two NUMA nodes. As a result, vSphere 6.5 would support a system equipped with 8 Intel Xeon Platinum 8176M Processors each fully loaded with 1.5 TB of memory and configured with sub-NUMA clustering. This setup would create one system, offering 16 NUMA nodes each fitted with 768 GB of local memory.

Take Caution of 8 DIMM System Board Designs
The introduction of Purley forces system integrators to redesign the system boards to support the new functionality. To support the full possibilities of the memory subsystem, system boards should be equipped with either 6 or 12 DIMM sockets. Some entry-level systems are designed with 8 DIMM slots. The Intel Xeon is designed to use the six channels when creating a region, this results in an unbalanced region design of 6+2. Region 0 consists of 6 DIMM slots, offering a theoretical peak transfer rate of 127,800 MB/s (when using 2666 GT/s), while region 1 offers
42,600 MB/s. This will result in inconsistent performance, something to definitely to avoid. Thus it’s recommended to equip these systems with the six-channel configuration in mind, order these systems and only populate the first six DIMM slots per CPU.

Interconnect
The performance of a dual CPU system can be impacted by the interconnect between the CPU packages if you span VMs across the NUMA nodes (Wide-VMs). Purley introduces a new interconnect called the UltraPath Interconnect (UPI) and replaces the QuickPath Interconnect. The next article in this series provides an in-depth look at the UPI.

A vSphere Focused Guide to the Intel Xeon Scalable Family

Intel released the much-anticipated Skylake Server CPU this year. Moving away from the E5-2600-v moniker, Intel names the new iteration of its server CPU the Intel Xeon Scalable Family. On top of this it uses precious metal categories such as Platinum and Gold to identify different types and abilities.

Upholding the tradition, the new Xeon family contains more cores than the previous Xeon version. The new top-of-the-line CPU offers 28 cores on a single processor die, memory speeds are now supported up to 2666 MHz. However, the biggest appeal for vSphere datacenters is the new “Purley” platform and its focus on increasing bandwidth between possibly every component possible. In this series, we are going to look at the new Intel Xeon Scalable family microarchitecture and which functions help to advance vSphere datacenters.

NUMA and vNUMA Focus
Instead of solely listing the speeds and feeds of the new architecture, I will be reviewing the new functionality with considerations of today’s vSphere VM configuration landscape. In modern vSphere datacenters, small VMs and large VMs co-exists with a single server. Many VMs consume the interconnect between physical CPUs. Some VMs span multiple NUMA nodes (Wide-VMs) while others fit inside a single physical NUMA node. The VMkernel NUMA scheduler attempts to optimize local memory consumption as much as possible. Sometimes remote memory is unavoidable. Consolidation ratios increase each year, hence the focus on the interconnect. Yet, single-threaded applications are still prevalent in many DC’s. Therefore single-core improvements will not be ignored in this series. Designing a system that is bound to run a high consolidation with a mix of small and large VMs is not an easy task.

Rebranding
The Xeon Scalable family introduces a new naming scheme. Gone are the names such as the E5-2630 v4, E5-4660 v4 or E7-8894 v4. Now Bronze, Silver, Gold, and Platinum class indicate the range of overall performance, Bronze representing the entry-level class CPU comparable to the previous E3 series, while Platinum class CPUs provide you the highest levels of scalability and most cores possible. As of today, Intel offers 58 different CPU types within the Xeon Scalable family, i.e., 2 Bronze CPUs, 8 Silver, 6 Gold 51xx, 26 Gold 61xx and 16 Platinum CPUs.

Bronze Silver Gold 51xx Gold 61xx Platinum
Scalability 2 2 2 2-4 2-8
Max Cores 8 12 14 22 28
Max Base Frequency (GHz) 1.7 2.6 3.6 3.5 3.6
Max Memory Speed (MHz) 2133 2400 2400 2666 2666
UPI* Links 2 2 2 3 3
UPI Speed (GT/s) 9.6 9.6 10.4 10.4 10.4

* The Xeon Scalable family introduces a new processor interconnect called the UltraPath Interconnect (UPI) and replaces the QuickPath Interconnect. The next article in this series provides an in-depth look at the UPI.

Integrations and Optimizations
Intel uses suffixes to indicate particular integrations or optimizations.

Suffix Function Integration | Optimization Availability
F Fabric Integrated Intel® Omni-Path Architecture Gold 61xx, Platinum
M Memory Capacity 1.5 TB Support per Socket Gold 61xx, Platinum
T High Tcase Extended Reliability (10-Year Use) Silver,Gold, Platinum

Omni-Path Architecture
The new Xeon family offers on-die Omni-Path Architecture that allows for 100 Gbps connectivity. In-line with the industry effort to remove as much “moving parts or components as possible the new architecture the signal is not being routed through the socket and motherboard but provides a direct connection to the processor.

Image by ServeTheHome.com

The always excellent Serve The Home has published a nice article about the F-type Xeons. Unfortunately, the current vSphere version does not support the Omni-Path Architecture.

Total Addressable Memory
Intel hard-coded the addressable memory capacity on the CPU. As a result, non-M CPUs will not function if more than 768 GB of RAM is present in the DIMM sockets connected to its memory controllers. If you tend to scale-up your servers during their lifecycle, consider this limitation. If you are planning to run monster VMs that require more than 768 GB of RAM and want to avoid spanning it across NUMA nodes, consider obtaining “M” designation CPUs.

Why wouldn’t you just buy M designated CPUs in the first place, you might wonder? Well, the M badge comes with a near-3K USD price hike. Comparing, 6142 (16 cores at 2.6 GHz) and 6140 (18 cores at 2.3 GHz) the list price for the 6142 is $2946, while the 6142M is $5949. The similar price difference for the 6140, vanilla style 6140 costs $2445, M-badge $5448. But with current RAM prices, we are talking about a minimum of $100.000 price tag for 1.5 TB of memory PER socket!

Extended Reliability
For specific use-cases, Intel provides CPUs with an extended reliability of up to 10 years. As you can imagine, these CPUs do not operate at top speeds. The fastest T enabled CPU runs at 2.6 GHz base frequency.

Similar, but not identical CPU packaging
When reviewing the spectrum of available CPUs, one noticeable thing is the availability of identical core count CPUs across the precious metals. For example, the 12 core CPU package. It’s available in Silver, Gold 51xx, Gold 61xx and Platinum. It’s available with an extended Tcase (Intel Xeon Silver 4116), and the Intel Xeon Gold 6126 is also available as 6126T and 6126F. One has to dig a little bit further to determine the added benefits of selecting a Gold version over a Silver version.

Processor Silver 4116 Gold 5118 Gold 6126 Gold 6136 Gold 6146 Platinum 8158
Cores 12 12 12 12 12 12
Base Frequency (GHz) 2.10 2.30 2.60 3.00 3.20 3.00
Max Turbo Frequency (GHz) 3.00 3.20 3.70 3.70 4.20 3.70
TDP (W) 85 105 125 150 165 150
L3 Cache (MB) 16.5 16.5 19.25 24.75 24.75 24.75
# of UPI Links 2 2 3 3 3 3
Scalability 2S 4S 4S 4S 4S 8S
# of AVX-512 FMA Units 1 1 2 2 2 2
Max Memory Size (GB) 768 768 768 768 768 768
Max Memory Speed (MHz) 2400 2400 2666 2666 2666 2666
Memory Channels 6 6 6 6 6 6

Part 2: Memory Subsystem available

VMware Cloud on AWS Technical Overview

Yesterday we launched the VMware Cloud on AWS service. VMware Cloud on AWS allows you to run your applications across private, public, and hybrid cloud environments based on VMware vSphere, with optimized access to AWS services. The Cloud SDDC consists of vSphere, NSX and vSAN technology to provide you a familiar environment which can be managed an operated with your current tool and skill set. By leveraging bare-metal AWS infrastructure the Cloud SDDC can scale in an unprecedented way.

VMware Cloud on AWS is a service and that means that we will not using product versions when we refer to the service. Instead we will be calling the first release the initial availability of the service. Any release after is referred to as future release. VMware Cloud on AWS is operated by VMware. In short that means that VMware is responsible for providing infrastructure resources, the customer is responsible for consuming the resources. This article explores the resource capacity of the Cloud SDDC at initial availability.

Compute in VMware Cloud on AWS
At initial availability, the VMware Cloud on AWS base cluster configuration contains four hosts. Each host is configured with 512GB of memory and contains dual CPUs. These CPUs are custom-built Intel Xeon Processor E5-2686 v4 CPU. Each CPU contains 18 cores running at 2.3GHz, resulting in a physical cluster core count of 144. Hyper-threading is enabled, allowing the VMs to consume 288 logical processors in the default Cloud SDDC configuration. Please note, that VMware Cloud on AWS uses a single, fixed host configuration; the option to add components to the host configuration is not offered at this time. However, the scale-out model enables expansion to up to 16 hosts, resulting in 576 CPU cores and 8TB of memory.

vSphere DRS and vSphere HA are enabled and are configured to provide the best availability and resource utilization. vSphere DRS is full automated and the migration threshold is set to the default vSphere DRS level to avoid excessive vSphere vMotion operations. High availability of cluster resources is provided by vSphere HA and Auto remediation hardware.

vSphere High Availability is used to guarantee enough resources for restarting VMs during an ESXi host failure. The ESXi hosts are monitored and in the event of a failure, the VMs on a failed host are restarted on alternative ESXi hosts in the cluster. To maximize productivity while minimizing overhead, the vSphere HA settings of the cluster is configured to tolerate the equivalent of one ESXi host failure (25% percentage-based admission control policy). The host isolation response is set to power off and restart the VMs.

Host failures remediation is the responsibility of VMware. If a host fails permanently, VMware replaces this ESXi host without user intervention. Automatic remediation of failed hardware eliminates the impact of long-term resource reduction of a permanent host failure. The Cloud SDDC is configured with two DRS resource pools. One resource pool contains the management VMs to operate the Cloud SDDC, while the other top-level resource pool is created to manage customer workloads. Customers have the option to create child resource pools.

Storage in VMware Cloud on AWS
The SDDC cluster includes a vSAN all-flash array and each host provides a total of 10TB of raw capacity for VMs to consume. A default Cloud SDDC cluster provides 40TB of raw capacity. The capacity consumption of the VM depends on the configured storage policy. By default, a RAID-1 Fault Tolerance Method is applied, but customers can create storage profiles that provide less overhead, such as RAID-5 or RAID-6 Failure Tolerance Method. Please note that for using RAID-6 Failure Tolerance Method a minimum of 6 hosts are required inside the Cloud SDDC cluster.

Each ESXi host contains 8 NVMe devices. These 8 devices are distributed across two vSAN disk groups. Within a disk group, the write-caching tier leverages one NVMe device with 1.7TB of storage; the storage capacity tier leverages the other three NVMe devices with a combined 5.1TB of storage.

Storage Encryption
Datastore-level encryption with vSAN encryption, or VM-level encryption with vSphere VM encryption, is not available at initial availability of VMware Cloud on AWS. To provide data security, all local storage NVMe devices are encrypted at the firmware level by AWS. The encryption keys are managed by AWS and are not exposed to or controlled by VMware or VMware Cloud on AWS customers.

Cloud SDDC Configuration
At initial availability, the Cloud SDDC is restricted to a single AWS region and availability zone (AZ). Failed hardware can be automatically detected, and automated remediation enables failed host to be automatically replaced by other ESXi hosts. If necessary the VSAN datastore is automatically rebuilt without user intervention.

In future VMware Cloud on AWS releases, through the partnership of VMware and AWS, multi-AZ availability will be possible for the first time ever, by stretching the cluster across two AZs in the same region. With this groundbreaking offering, refactoring of traditional applications will no longer be required to obtain high availability on the AWS infrastructure. Instead, synchronous write replication will be leveraged across AZs, resulting in a recovery point objective (RPO) of zero and a recovery time objective (RTO) that depends on the vSphere HA restart.

Networking in VMware Cloud on AWS
VMware Cloud on AWS is built around NSX. It’s optimized to provide VM networking in the Cloud SDDC, while abstracting the Amazon Virtual Private Cloud (VPC) networks. It enables ease of management by providing logical networks to VMs and automatically connecting new hosts to logical and VMkernel networks as clusters are scaled out. At initial availability, users connect to VMware Cloud on AWS via a layer 3 VPN connection. Future releases of VMware Cloud on AWS, however, will support AWS Direct Connect and allow cross-cloud vSphere vMotion operations.

An IPsec layer 3 VPN is set up to securely connect the on-premises vCenter Server instance with the management components running on the in-cloud SDDC cluster. A separate IPsec layer 3 VPN is set up to create connectivity between the on-premises workloads and the VMs running inside the in-cloud SDDC cluster. NSX is used for all networking and security and is decoupled from Amazon VPC networking. The compute gateway and DLR are pre-configured as part of the prescriptive network topology and cannot be changed by the customer. Customers provide only their own subnets and IP ranges.

VMware Cloud on AWS ready for your workload
VMware Cloud on AWS provides you cloud resources that can be consumed by using your current skill set and tool set. Each cloud SDDC provides state-of-the-art resources that can run the most demanding applications of today. The best enterprise software combined with the best cloud operator in the world allows you to run and scale your data center in an unprecedented way.

For more information, go to https://cloud.vmware.com/vmc-aws/resources

« Older posts

© 2017 frankdenneman.nl

Theme by Anders NorenUp ↑