Inside PernixData Engineering: Write back acceleration deep dive

FVP is well known for its incredible user experience design, the ease of use is unparalleled. When accelerating a virtual machine the only thing the user needs to decide on is whether to select the write through or write back policy and FVP takes care of the rest. Some of the things may not be obvious but there are so much that goes under the covers that I asked Mahesh Patil, engineer in the data path team, to give us more insight into the cluster, recovery and fault tolerance functionality of write back acceleration. Go check it out

Inside PernixData Engineering: Clustered acceleration deep dive

The last day of VMworld is always an interesting one, the buzz slowly dies done and everything is done at a more relaxed pace. However instead of enjoying this laid back moment, I decided to go back to the office and meet up with some of the engineers to shoot some videos.

In this video Woon and I take a deepdive in to clustered acceleration. Woon Jung, one of the core data path engineers, provides background about the challenges of write back. He gives insight about the clustering functions such as the cache coherency protocol and the support of remote flash access after a vMotion. During the interview I ask him to give me some examples of the surprises they encountered during the development of the product. This leads to some interesting information of the world of flash and kernel scheduling overhead.

Check it out:

Improve public speaking by reading a book?

Although it sounds like an oxymoron I do have the feeling that books about this topic can help you become a better public speaker, or in matter of fact more skillful in driving home your message.

After our talk at VMworld a lot of friends complimented not only on the talk itself but also on the improvements I’ve made when it comes to public speaking. My first public speaking engagement was VMworld 2010 at Vegas, 8 o’clock Monday morning for 1200 person. Talk about a challenge. Since then I have been slowly improving my skills. Last year I’ve done more talks than the previous 3 years before combined. Although Malcolm Gladwell’s 10.000 –hour rule is heavily debated nowadays, I do believe that practice is by far the best way to improve your skill. By itself getting 10.000 hours of public speaking time is rather a challenge and just by going through the motions alone will be very inefficient. To maximize efficiency I started to dive into the theory behind public speaking or even more broadly theory about communicating. Over the year I read a decent stack of books but these four stood out the most.

1: Confessions of a public speaker by Scott Berkun
Funny and highly practical. If you want to buy only one book, this one should be it. The book helps you with the act of public speaking; How to deal with stage fright, how to work a tough room, what are the things I need to take care of to make my talk go smoothly.

2: Made to Stick by Chip and Dan Heath
This books helps you structure the message you public-speaking-books
want to convey. It helps you to dive into the core of your message and communicate them in a memorable way. It’s a great book to read, lots of interesting stories and it’s one of those books that you should read multiple times to keep on refining your skillset.

3: Talk like Ted by Carmine Gallo
To some extent a combination of the two first books. The interesting part is the focus on the listener experience and its capability to focus for 18 minutes. In addition it gives you insights on some of the greatest TED talks.

4 Pitch Perfect by Bill and Alisa Bowman
This book helps you to enhance communication skills. It dives deeper in the act verbal and non-verbal language. It helps you to become cognizant of some of the mistakes everyone makes, yet can be avoided quite easily. The book helps you to drive your point in a more confident, persuasive and certain manner.

The beauty of these books is that you can use them, learn from them even if you are not a public speaker. In everyday life we all need to communicate, we all want our idea to be heard and possible get a buy in from others. I believe these books will help you achieve this. If you have found other books useful and interesting please leave a comment.

Storage I/O requirements of SAP HANA on vSphere 5.5

During VMworld a lot of attention was going to the support for SAP HANA on vSphere 5.5 for Production Environments. SAP HANA is an In-memory database platform that allows running real-time analytics and real-time apps.

In its white paper “Best practices and recommendations for Scale-Up Deployments of SAP HANA on VMware vSphere” VMware states that vMotion, DRS and HA is supported for virtualized SAP HANA systems. This is amazing and very exciting. Being able to run these types of database platform virtualized is a big deal. You can finally leverage the mobility and isolation benefits provided by the virtual infrastructure and get rid of the rigid physical landscapes that are costly to maintain and a pain to support.

When digging deeper in the architecture of the SAP HANA platform you discover that SAP HANA has to write to disk even though it’s an In-memory database platform. Writing to disk allows HANA to provide ACID guarantees for the database. ACID stands for (Atomicity, Consistency, Isolation, Durability) and these properties guarantee that database transactions are processed reliably.

On a side note, the release of SAP HANA support triggered me in to diving into database structures and architectures, luckily for me our director of Products has an impressive track record on DB designs so I spend a few hours with him to learn more about this. This information will be shared in a short series soon. But I digress.

The document “SAP HANA – Storage Requirements” available at provides detailed insight in storage I/O behavior of the platform. On page 4 the following statement is made: SAP HANA uses storage for several purposes:

Data: SAP HANA persists a copy of the in-memory data, by writing changed data in the form of so-called save point blocks to free file positions, using I/O operations from 4 KB to 16 MB (up to 64 MB when considering super blocks) depending on the data usage type and number of free blocks. Each SAP HANA service (process) separately writes to its own save point files, every five minutes by default.

Redo Log: To ensure the recovery of the database with zero data loss in case of faults, SAP HANA records each transaction in the form of a so-called redo log entry. Each SAP HANA service separately writes its own redo-log files. Typical block-write sizes range from 4KB to 1MB

So it makes sense to use a fast storage platform that can process various types of block sizes real fast. That means low latency and high throughput, which server-side resources can provide very easily.

In the document “SAP HANA Guidelines for being virtualized with VMware vSphere” available on the following statement is issued in the section 4.5.3. Storage Requirement:

SAP and VMware recommend to following the VMware Best Practices for SAP HANA virtualized with VMware vSphere with regards to technical storage configuration in VMware. Especially virtual disks created for log volumes of SAP HANA should reside on local SSD or PCI adapter flash devices if present. Central enterprise storage may be used in terms of the SAP HANA tailored data
center intergation approach.

It is an interesting standpoint. SAP recommends using flash and it makes sense because what is the point of running such a platform in memory when your storage platform is slow. When using local flash storage you will introduce a static workload in your virtual infrastructure again. SAP-HANA supports the use of enhanced vMotion, migrating a VM between two hosts and two datastores simultaneously, however at time of writing, DRS does not leverage enhanced vMotion for load-balancing operations. This results in the loss of automatic load balancing and potentially reducing the ability of virtual machine recovery by vSphere High Availability.

Instead of introducing rigid and silo’ed architectures, its makes sense to use PernixData FVP. FVP, Supported by VMware, allows for clustered and fault tolerant I/O acceleration by using flash or (soon) memory. By virtualizing these acceleration resources into one seamless pool, VMs can seamlessly migrate to any host in the cluster while being able to retrieve data throughout the cluster.

SAP HANA accelerates instructions by keeping it in memory, while FVP accelerates the writes by leveraging the available acceleration resources. In vSphere 5.5 SAP HANA is limited to 1TB of memory due to the maximum virtual machine configuration, however vSphere 5.5 supports a host memory configuration of 4TB. With the soon to be released FVP 2.0 with memory support, FVP allows you to leverage the remaining memory to accelerates the writes as well, making it a true in-memory platform.

Re: The Rack Endgame: A New Storage Architecture For the Data Center

Recently Stephen posted an excellent article about the future developments of enterprise storage architecture. His analysis about the different roles of storage is spot on, i.e. long-term capacity and short-term retention. Connecting high performance resources through high speed interconnects is an interesting idea, but I do have some comments I would like to share.

Recently Dave’s (McCrory) article about data gravity has been referred to more and more and it does make sense when you look at the industry shift gravitating resources closer to workloads. Intels’ move on increasing PCI bandwidth in their new CPU architecture, increased memory support per CPU (1.5TB) and supporting the NVMe specifications for their NAND devices indicates the move towards an abundance of ultra fast, low latency server side resources. VMware announced 6TB host memory support for their upcoming vSphere 6 platform and we expect support of NVMe drivers on the vSphere 5.5 platform soon.

In my view this seems logical, especially with the Software Defined Datacenter move. To many SDDC is the holy glue of datacenters, the primordial soup. I’m referring to the term introduced by Oparin, not Teenage Mutant Ninja Turtles although some datacenters do have most characteristics of a mutant. In reality many virtual infrastructures rest on a disparate set of hardware components. This equipment, typically multi-vendor, is expected to provide deterministic performance levels. Now it’s expected that SDDC will be this soup that transforms into a more mature form. And yet I don’t see this happening soon.

Why? No proper development cycle coordination, no proper joint agenda, no clear business value. Let’s use VAAI as an example. VAAI initiated by VMware working with partners implementing vendor specific commands, all based on T10 standards. VMware started to work in VAAI back in 2008 and today almost all storage systems ship with VAAI compliant firmware. 6 years later. What about the future, and now I must note that this is speculation on my part, VVOLS. VMware object based storage project relies heavily on VASA (vSphere API for Storage Awareness). It provides its partners with a set of commands, lets see how many commands they managed to include in the upcoming release. Don’t get me wrong, I’m thrilled about these initiatives, however it’s usually a myriad of reasons other than technical challenges that interfere with combined cross industry solution architectures.

And yet we do have a solution and that solution is staring us right in the face. The Hypervisor! The hypervisor is rich with information, including a collection of tightly knit resource schedulers. It is the perfect place to introduce policy-based management engines. The hypervisor becomes a single control plane that manages both the resource as well as the demand. A single construct to automate instructions in a single language providing a correct Quality of Service model at application granularity levels. You can control resource demand and distribution from one single pane of management. No need to wait on the completion of the development cycles from each vendor.

By allowing the hypervisor to transform into the architecture control plane the next step is to get the resource as close to the application as possible. And we all try to optimize memory and CPU resources for some time now. Think about NUMA, Early 2010 I published one of my first articles on NUMA and see how many content has been published since. For years we have been focusing in removing as much bottlenecks as possible and moving into territories of optimizing resources at Nano speeds. Moving towards an architecture that wilfully uses interconnects looks like its not the most optimal play here.

Leverage the optimization and increase of server side resources, leverage and build out the control plane of the virtualized infrastructure called the hypervisor and you can build out an application aware storage platform that is scalable in many ways.