Author: Frank Denneman (page 2 of 69)

A vSphere Focused Guide to the Intel Xeon Scalable Family

Intel released the much-anticipated Skylake Server CPU this year. Moving away from the E5-2600-v moniker, Intel names the new iteration of its server CPU the Intel Xeon Scalable Family. On top of this it uses precious metal categories such as Platinum and Gold to identify different types and abilities.

Upholding the tradition, the new Xeon family contains more cores than the previous Xeon version. The new top-of-the-line CPU offers 28 cores on a single processor die, memory speeds are now supported up to 2666 MHz. However, the biggest appeal for vSphere datacenters is the new “Purley” platform and its focus on increasing bandwidth between possibly every component possible. In this series, we are going to look at the new Intel Xeon Scalable family microarchitecture and which functions help to advance vSphere datacenters.

NUMA and vNUMA Focus
Instead of solely listing the speeds and feeds of the new architecture, I will be reviewing the new functionality with considerations of today’s vSphere VM configuration landscape. In modern vSphere datacenters, small VMs and large VMs co-exists with a single server. Many VMs consume the interconnect between physical CPUs. Some VMs span multiple NUMA nodes (Wide-VMs) while others fit inside a single physical NUMA node. The VMkernel NUMA scheduler attempts to optimize local memory consumption as much as possible. Sometimes remote memory is unavoidable. Consolidation ratios increase each year, hence the focus on the interconnect. Yet, single-threaded applications are still prevalent in many DC’s. Therefore single-core improvements will not be ignored in this series. Designing a system that is bound to run a high consolidation with a mix of small and large VMs is not an easy task.

The Xeon Scalable family introduces a new naming scheme. Gone are the names such as the E5-2630 v4, E5-4660 v4 or E7-8894 v4. Now Bronze, Silver, Gold, and Platinum class indicate the range of overall performance, Bronze representing the entry-level class CPU comparable to the previous E3 series, while Platinum class CPUs provide you the highest levels of scalability and most cores possible. As of today, Intel offers 58 different CPU types within the Xeon Scalable family, i.e., 2 Bronze CPUs, 8 Silver, 6 Gold 51xx, 26 Gold 61xx and 16 Platinum CPUs.

Bronze Silver Gold 51xx Gold 61xx Platinum
Scalability 2 2 2 2-4 2-8
Max Cores 8 12 14 22 28
Max Base Frequency (GHz) 1.7 2.6 3.6 3.5 3.6
Max Memory Speed (MHz) 2133 2400 2400 2666 2666
UPI* Links 2 2 2 3 3
UPI Speed (GT/s) 9.6 9.6 10.4 10.4 10.4

* The Xeon Scalable family introduces a new processor interconnect called the UltraPath Interconnect (UPI) and replaces the QuickPath Interconnect. The next article in this series provides an in-depth look at the UPI.

Integrations and Optimizations
Intel uses suffixes to indicate particular integrations or optimizations.

Suffix Function Integration | Optimization Availability
F Fabric Integrated Intel® Omni-Path Architecture Gold 61xx, Platinum
M Memory Capacity 1.5 TB Support per Socket Gold 61xx, Platinum
T High Tcase Extended Reliability (10-Year Use) Silver,Gold, Platinum

Omni-Path Architecture
The new Xeon family offers on-die Omni-Path Architecture that allows for 100 Gbps connectivity. In-line with the industry effort to remove as much “moving parts or components as possible the new architecture the signal is not being routed through the socket and motherboard but provides a direct connection to the processor.

Image by

The always excellent Serve The Home has published a nice article about the F-type Xeons. Unfortunately, the current vSphere version does not support the Omni-Path Architecture.

Total Addressable Memory
Intel hard-coded the addressable memory capacity on the CPU. As a result, non-M CPUs will not function if more than 768 GB of RAM is present in the DIMM sockets connected to its memory controllers. If you tend to scale-up your servers during their lifecycle, consider this limitation. If you are planning to run monster VMs that require more than 768 GB of RAM and want to avoid spanning it across NUMA nodes, consider obtaining “M” designation CPUs.

Why wouldn’t you just buy M designated CPUs in the first place, you might wonder? Well, the M badge comes with a near-3K USD price hike. Comparing, 6142 (16 cores at 2.6 GHz) and 6140 (18 cores at 2.3 GHz) the list price for the 6142 is $2946, while the 6142M is $5949. The similar price difference for the 6140, vanilla style 6140 costs $2445, M-badge $5448. But with current RAM prices, we are talking about a minimum of $100.000 price tag for 1.5 TB of memory PER socket!

Extended Reliability
For specific use-cases, Intel provides CPUs with an extended reliability of up to 10 years. As you can imagine, these CPUs do not operate at top speeds. The fastest T enabled CPU runs at 2.6 GHz base frequency.

Similar, but not identical CPU packaging
When reviewing the spectrum of available CPUs, one noticeable thing is the availability of identical core count CPUs across the precious metals. For example, the 12 core CPU package. It’s available in Silver, Gold 51xx, Gold 61xx and Platinum. It’s available with an extended Tcase (Intel Xeon Silver 4116), and the Intel Xeon Gold 6126 is also available as 6126T and 6126F. One has to dig a little bit further to determine the added benefits of selecting a Gold version over a Silver version.

Processor Silver 4116 Gold 5118 Gold 6126 Gold 6136 Gold 6146 Platinum 8158
Cores 12 12 12 12 12 12
Base Frequency (GHz) 2.10 2.30 2.60 3.00 3.20 3.00
Max Turbo Frequency (GHz) 3.00 3.20 3.70 3.70 4.20 3.70
TDP (W) 85 105 125 150 165 150
L3 Cache (MB) 16.5 16.5 19.25 24.75 24.75 24.75
# of UPI Links 2 2 3 3 3 3
Scalability 2S 4S 4S 4S 4S 8S
# of AVX-512 FMA Units 1 1 2 2 2 2
Max Memory Size (GB) 768 768 768 768 768 768
Max Memory Speed (MHz) 2400 2400 2666 2666 2666 2666
Memory Channels 6 6 6 6 6 6

Part 2: Memory Subsystem available

VMware Cloud on AWS Technical Overview

Yesterday we launched the VMware Cloud on AWS service. VMware Cloud on AWS allows you to run your applications across private, public, and hybrid cloud environments based on VMware vSphere, with optimized access to AWS services. The Cloud SDDC consists of vSphere, NSX and vSAN technology to provide you a familiar environment which can be managed an operated with your current tool and skill set. By leveraging bare-metal AWS infrastructure the Cloud SDDC can scale in an unprecedented way.

VMware Cloud on AWS is a service and that means that we will not using product versions when we refer to the service. Instead we will be calling the first release the initial availability of the service. Any release after is referred to as future release. VMware Cloud on AWS is operated by VMware. In short that means that VMware is responsible for providing infrastructure resources, the customer is responsible for consuming the resources. This article explores the resource capacity of the Cloud SDDC at initial availability.

Compute in VMware Cloud on AWS
At initial availability, the VMware Cloud on AWS base cluster configuration contains four hosts. Each host is configured with 512GB of memory and contains dual CPUs. These CPUs are custom-built Intel Xeon Processor E5-2686 v4 CPU. Each CPU contains 18 cores running at 2.3GHz, resulting in a physical cluster core count of 144. Hyper-threading is enabled, allowing the VMs to consume 288 logical processors in the default Cloud SDDC configuration. Please note, that VMware Cloud on AWS uses a single, fixed host configuration; the option to add components to the host configuration is not offered at this time. However, the scale-out model enables expansion to up to 16 hosts, resulting in 576 CPU cores and 8TB of memory.

vSphere DRS and vSphere HA are enabled and are configured to provide the best availability and resource utilization. vSphere DRS is full automated and the migration threshold is set to the default vSphere DRS level to avoid excessive vSphere vMotion operations. High availability of cluster resources is provided by vSphere HA and Auto remediation hardware.

vSphere High Availability is used to guarantee enough resources for restarting VMs during an ESXi host failure. The ESXi hosts are monitored and in the event of a failure, the VMs on a failed host are restarted on alternative ESXi hosts in the cluster. To maximize productivity while minimizing overhead, the vSphere HA settings of the cluster is configured to tolerate the equivalent of one ESXi host failure (25% percentage-based admission control policy). The host isolation response is set to power off and restart the VMs.

Host failures remediation is the responsibility of VMware. If a host fails permanently, VMware replaces this ESXi host without user intervention. Automatic remediation of failed hardware eliminates the impact of long-term resource reduction of a permanent host failure. The Cloud SDDC is configured with two DRS resource pools. One resource pool contains the management VMs to operate the Cloud SDDC, while the other top-level resource pool is created to manage customer workloads. Customers have the option to create child resource pools.

Storage in VMware Cloud on AWS
The SDDC cluster includes a vSAN all-flash array and each host provides a total of 10TB of raw capacity for VMs to consume. A default Cloud SDDC cluster provides 40TB of raw capacity. The capacity consumption of the VM depends on the configured storage policy. By default, a RAID-1 Fault Tolerance Method is applied, but customers can create storage profiles that provide less overhead, such as RAID-5 or RAID-6 Failure Tolerance Method. Please note that for using RAID-6 Failure Tolerance Method a minimum of 6 hosts are required inside the Cloud SDDC cluster.

Each ESXi host contains 8 NVMe devices. These 8 devices are distributed across two vSAN disk groups. Within a disk group, the write-caching tier leverages one NVMe device with 1.7TB of storage; the storage capacity tier leverages the other three NVMe devices with a combined 5.1TB of storage.

Storage Encryption
Datastore-level encryption with vSAN encryption, or VM-level encryption with vSphere VM encryption, is not available at initial availability of VMware Cloud on AWS. To provide data security, all local storage NVMe devices are encrypted at the firmware level by AWS. The encryption keys are managed by AWS and are not exposed to or controlled by VMware or VMware Cloud on AWS customers.

Cloud SDDC Configuration
At initial availability, the Cloud SDDC is restricted to a single AWS region and availability zone (AZ). Failed hardware can be automatically detected, and automated remediation enables failed host to be automatically replaced by other ESXi hosts. If necessary the VSAN datastore is automatically rebuilt without user intervention.

In future VMware Cloud on AWS releases, through the partnership of VMware and AWS, multi-AZ availability will be possible for the first time ever, by stretching the cluster across two AZs in the same region. With this groundbreaking offering, refactoring of traditional applications will no longer be required to obtain high availability on the AWS infrastructure. Instead, synchronous write replication will be leveraged across AZs, resulting in a recovery point objective (RPO) of zero and a recovery time objective (RTO) that depends on the vSphere HA restart.

Networking in VMware Cloud on AWS
VMware Cloud on AWS is built around NSX. It’s optimized to provide VM networking in the Cloud SDDC, while abstracting the Amazon Virtual Private Cloud (VPC) networks. It enables ease of management by providing logical networks to VMs and automatically connecting new hosts to logical and VMkernel networks as clusters are scaled out. At initial availability, users connect to VMware Cloud on AWS via a layer 3 VPN connection. Future releases of VMware Cloud on AWS, however, will support AWS Direct Connect and allow cross-cloud vSphere vMotion operations.

An IPsec layer 3 VPN is set up to securely connect the on-premises vCenter Server instance with the management components running on the in-cloud SDDC cluster. A separate IPsec layer 3 VPN is set up to create connectivity between the on-premises workloads and the VMs running inside the in-cloud SDDC cluster. NSX is used for all networking and security and is decoupled from Amazon VPC networking. The compute gateway and DLR are pre-configured as part of the prescriptive network topology and cannot be changed by the customer. Customers provide only their own subnets and IP ranges.

VMware Cloud on AWS ready for your workload
VMware Cloud on AWS provides you cloud resources that can be consumed by using your current skill set and tool set. Each cloud SDDC provides state-of-the-art resources that can run the most demanding applications of today. The best enterprise software combined with the best cloud operator in the world allows you to run and scale your data center in an unprecedented way.

For more information, go to

Get your Free Book at VMworld

At VMworld, the presenters of the following sessions will be giving away free copies of the Host Deep Dive book to the audience.

Performance Bootcamp
Mark Achtemichuk
Saturday, Aug 26, 8:00 a.m. – 5:00 p.m.
More information about pre-VMworld Performance Bootcamp

An Introduction to VMware Software-Defined Storage [STO2138QU]
Lee Dilworth, Principal Systems Engineer, VMware
Sunday, Aug 27, 4:00 p.m. – 4:30 p.m. | Oceanside C, Level 2

A Deep Dive into vSphere 6.5 Core Storage Features and Functionality [SER1143BU]
Cody Hosterman, Technical Director–VMware Solutions, Pure Storage
Cormac Hogan, Director – Chief Technologist, VMware
Monday, Aug 28, 11:30 a.m. – 12:30 p.m. | Mandalay Bay Ballroom G, Level 2

Extreme Performance Series: Benchmarking 101 [SER2723BUR]
Joshua Schnee, Senior Staff Engineer @ VMware Performance, VMware
Mark Achtemichuk, Staff Engineer, Performance, VMware
Monday, Aug 28, 4:00 p.m. – 5:00 p.m. | Mandalay Bay Ballroom B, Level 2

Maximum Performance with Mark Achtemichuk [VIRT2368GU]
Mark Achtemichuk, Staff Engineer, Performance, VMware
Monday, Aug 28, 5:30 p.m. – 6:30 p.m. | Reef E, Level 2

The Top 10 Things to Know About vSAN [STO1264BU]
Duncan Epping, Chief Technologist, VMware
Cormac Hogan, Director – Chief Technologist, VMware
Monday, Aug 28, 5:30 p.m. – 6:30 p.m. | Mandalay Bay Ballroom H, Level 2

VMware vSAN: From 2 Nodes to 64 Nodes, Architecting and Operating vSAN Like a VCDX for Scalability and Simplicity [STO2114BU]
Greg Mulholland, Principal Systems Engineer, VMware
Jeff Wong, Customer Success Architect, VMware
Monday, Aug 28, 5:30 p.m. – 6:30 p.m. | Surf E, Level 2

Extreme Performance Series: Performance Best Practices [SER2724BU]
Reza Taheri, Principal Engineer, VMware
Mark Achtemichuk, Staff Engineer, Performance, VMware
Tuesday, Aug 29, 2:30 p.m. – 3:30 p.m. | Oceanside D, Level 2

vSphere 6.5 Host Resources Deep Dive: Part 2 [SER1872BU]
Frank Denneman, Senior Staff Architect, VMware
Niels Hagoort, Owner, HIC (Hagoort ICT Consultancy)
Wednesday, Aug 30, 8:30 a.m. – 9:30 a.m. | Breakers E, Level 2

Extreme Performance Series: Benchmarking 101 [SER2723BUR]
Joshua Schnee, Senior Staff Engineer @ VMware Performance, VMware
Mark Achtemichuk, Staff Engineer, Performance, VMware
Wednesday, Aug 30, 8:30 a.m. – 9:30 a.m. | Lagoon L, Level 2

vSAN Networking and Design Best Practices [STO3276GU]
John Nicholson, Senior Technical Marketing Manager, VMware
Wednesday, Aug 30, 11:30 a.m. – 12:30 p.m. | Reef C, Level 2

vSAN Hardware Deep Dive Panel [STO1540PU]
Ed Goggin, Staff Engineer 2, VMware
David Edwards, Principal Engineer, Director Solutions, Resurgent Technology
Ken Werneburg, Group Manager Technical Marketing, VMware
Jeffrey Taylor, Technical Director, VMware
Ron Scott-Adams, Hyper-Converged Systems Engineer, VMware
Wednesday, Aug 30, 1:00 p.m. – 2:00 p.m. | Mandalay Bay Ballroom D, Level 2

A Closer Look at vSAN Networking Design and Configuration Considerations [STO1193BU]
Cormac Hogan, Director – Chief Technologist, VMware
Andreas Scherr, Senior Solution Architect, VMware
Wednesday, Aug 30, 4:00 p.m. – 5:00 p.m. | Mandalay Bay Ballroom G, Level 2

Virtual Volumes Technical Deep Dive [STO2446BU]
Patrick Dirks, Sr. Manager, VMware
Pete Flecha, Sr Technical Marketing Architect, VMware
Thursday, Aug 31, 10:30 a.m. – 11:30 a.m. | Oceanside B, Level 2

Book Signing
We will be doing two book signing sessions as well.
At the Rubrik booth #412 on Monday, Aug 28, 2:00 p.m. – 3:00 p.m.
At the VMworld Book store on Tuesday, Aug 29, 11:30 a.m. – 12:00 p.m.
Or just feel free to approach us when you see us walking by.

Register now for VMware Cloud on AWS Technical Deep Dive Session

I noticed that our technical deep dive session on VMware Cloud on AWS was added to the content catalog of VMworld. In this session, Ray Budavari and I will cover the VMware Cloud on AWS infrastructure in detail.

For the first time, we are allowed to uncover details about the host configuration, the vSAN infrastructure and of course network topology. We explore advanced features such as Elastic DRS and Autoremediation HA. The last 15 minutes of our session allows you to ask questions about VMC. Please register if you don’t want to miss this session. Both Ray and I have a full schedule, therefore we are unable to schedule a repeat of this session during VMworld US.

Session details:
VMware Cloud on AWS: A Technical Deep Dive [LHC2384BU]
Frank Denneman, Senior Staff Architect, VMware
Ray Budavari, Senior Staff Technical Product Manager, VMware

Tuesday, Aug 29, 3:30 p.m. – 4:30 p.m.
Session Type: Breakout Session
Track: Integrate Public Clouds
Integrate Public Clouds: Leverage Hybrid Clouds
Product and Topics: NSX, vCenter, vSAN, vSphere
Technical Level: Technical – Advanced
Session Hashtag: #LHC2384BU

VMware Cloud on AWS – Predictable Capacity Provisioning

In preparation for the VMworld Session LHC2971BU – Managing Your Hybrid Cloud with VMware Cloud on AWS which I’m co-presenting with Emad Younis, I asked the following question on Twitter:

And the number of answers were overwhelming. The stories were a bit underwhelming. Funny to see that we strive to automate every single step in the process. Guys like Alan, Luc, and William help the community to create scripted installs and configuration of the ESXi host. Creating a consistent, human-error free, rapid process. Shaving off valuable time of the time-consuming server provisioning process. Some organizations incorporate the vRealize suite to create a consistent user experience for the entire IT services portfolio. Interestingly enough, the overall lead time seems mostly impacted by internal acquisition processes. To give a few examples:

And the list goes on and on. In most organizations, the procurement process is rigid, well-defined process. However, the lead time of the acquisition process is either unpredictable and inconsistent. The overall message is that it cripples the agility of the IT organization.

IT organizations need to react fast to the business needs. Resource management of current workload is difficult enough, figuring out what to expect in the upcoming months is challenging. Unfortunately, the introduction of new workload does not follow a linear demand curve. To cater the (possible) future needs of the customer, the order is either doubled in size, or onboarding of new workloads is gated. Either impacting the bottom-line of the company or the ability to facilitate IT services properly.

In essence, the CAPEX element of server resource acquisition massively impacts or hinders the execution ability of the IT organization. Strategizing CAPEX\OPEX is not a part of the core focus of many admins and architects, it does affect their means of execution. As demonstrated by the many tweets. With VMware Cloud on AWS, the host resource acquisition process shifts from CAPEX to OPEX. Removing the inconsistent and unpredictable procurement process, allowing for a faster, consistent and predictable method of providing compute, storage and networking resources.

VMware Cloud on AWS (VMC) makes my resource-management heart beat faster. By leveraging the AWS operation model, the SDDC cluster running on the AWS infrastructure is resizable by a click of a button. Right-click on the cluster and select resize.

Just select the number of hosts you want to add and within moments you will get new dedicated physical hardware added to your cluster. Ready to provide the resources your new workloads require. Resize means you can remove the resources as well, which in result your costs will go down as well.

Due to the combined fleet management of AWS and VMC, the new ESXi hosts are fully configured and ready to welcome new workload. All VMkernel and logical networks are automatically configured and made available. The vSAN datastore is automatically expanded with the host-local NVMe flash devices provided by the new hosts. DRS detects the new physical resources and automatically rebalances the cluster, provided the most optimal resource availability.

Elastic DRS and Autoremedation HA allows for an automatize method or adding and removing dedicated hardware resources, but these topics will be covered in a different article.

From a resource management perspective, a mindset shift will happen. VMC allows you to reduce the time spent on infrastructure configuration and management and allows you to focus more on resource consumption. What cluster configuration is required in the upcoming months? What is my burst strategy?

Unfortunately, I can’t go into detail as the service is not released yet. VMworld boasts an exciting line up of VMware Cloud on AWS sessions. I will be hosting a meet the expert on resource management at both VMworlds, sign up if you want to talk more about this exciting new technology

Older posts Newer posts

© 2018

Theme by Anders NorenUp ↑