Although FVP relies on the crossbar architecture of the network, the fast point-to-point connection we noticed that in some scenarios the network performance could be detrimental to the storage performance. This was especially true if a lot of virtual machines ran in write back with two replicas. Particularly 1Gb networks are susceptible for this when a lot of replication traffic was introduced.
The diagram provided by Pete Koehler (vmpete.com) is showing this behavior, on the left the virtual machine is leveraging the power of the flash device. The application generates a workload of more than 5000 IOPS and the flash device (orange curve) happily provides that storage performance to the application. Once fault tolerance is enabled, sometimes the network becomes the bottleneck when the data is written to the remote flash device. The green curve, the network performance, overlays the orange (flash) curve and dictates the IO performance.
Introducing Adaptive Network Compression
What we wanted to do was to make sure that the bottleneck simply did not move from storage to network. Therefor we developed network compression, providing the ability to compress redundant write data before sending it over the network to remote flash devices. This allows FVP to consume the least amount of bandwidth possible; a byproduct of compression is the reduction of latency compared to sending uncompressed bigger blocks.
The thing that comes to the mind immediately, when thinking about compression is CPU cost. There is a CPU cost associated with compressing data and there is a CPU cost associated with decompressing data at the peer end. And then you have to figure out which cost is more detrimental to performance, CPU cost or network bandwidth cost? The graph shown below is the CPU compression cost introduced by FVP adaptive network compression. It shows that the cost incurred is minimal. And that is awesome because now you are able to get saving on network bandwidth without taking cost in terms of CPU.
The blue curve is the CPU cost the virtual machine incurs when using write back without replication. The orange curve is the CPU cost seen by the virtual machine with compression enabled. As you see the blue and orange curves are very close together, indicating that the CPU cost are minor.
To keep the CPU cost on the source host as low as possible is to use an adaptive algorithm that measures costs and benefit. Adaptive network compression makes sure that data can be compressed and that the cost of the compression does not exceed the benefit of bandwidth saved. Funny thing is that sometimes small blocks increase in size when compressing data, therefor it will review every bit of data and make sure compression provides benefits to the virtualized datacenter.
An interesting fact is that we do not decompress the data when it is written to the remote flash device. This provides two benefits. No CPU overhead involved and a reduction of remote flash footprint. The reason why we keep it in a compressed state is very simple. Typically environments perform well, meaning that the majority of time your environment should be up and running. Outages should be very infrequent. When a failure occurs and the task of writing data to the storage array falls on the peer host, we will incur the CPU cost of decompression. During normal operations, redundant write data lands on the flash device in a compressed state and will be moved out of the flash device once the data is written to the storage array. This process should impact the peer host as little as possible and keeping it in a compressed state accomplish that requirement.
The chart above illustrates the before and after state of write back with replication using a 1GbE connection. The performance on the left shows an average of 2700 IOPS, once replication is enabled the performance dropped to 1700 writes per second. The flat line perfectly shows the bottleneck introduced by bandwidth constraints. When running the same workload on a 10GbE network, it shows that the network provides compatible speed and bandwidth.
The same workload test was done on a 1GbE network with network compression enabled. The compression showed performance to near native flash only performance results.
To illustrate the benefit of adaptive network compression, the engineers disabled and enabled the feature. In FVP 2.0 adaptive network compression is enabled automatically when a 1GbE network connection is detected. No user intervention is required and for the curious minds, we do not offer a U.I. or Powershell option to enable or disable it. The adaptive nature will provide you the best performance with the lowest amount of overhead.
One of the cool things on the FVP cluster summary are the hero numbers, this shows the data saved from the storage array, the bandwidth saved from the storage area network and in FVP 2.0 you can monitor the amount of network bandwidth saved by Adaptive Network Compression. In order to make the screenshot I had to change my FVP network configuration from using a 10GbE to a 1GbE network. With FVP you can do this on the fly without impacting any uptime of hosts or virtual machines. I ran a test very quickly to generate some numbers hence the underwhelming total. In reality when using a 1GbE network with real application workload you will see quite impressive numbers.