Write-Back and Write-Through policies in FVP

Fault tolerant write acceleration is one of the key features of FVP. Once you select a virtual machine to be accelerated by FVP you can choose to apply a write through policy or write back policy with a x number of replicas of the virtual machine. Before expanding upon replicas, let’s review the basics of write through and write back policies.

Write policies
Please note that the flash device itself does not act as a persistent datastore, it’s a repository of active I/O. Therefore when the application issues a write I/O, the data is committed to the flash device however this data must always be written to the storage system. The timing of the write operation to the storage system is controlled by the write policy.

Write through policy
When a virtual machine issues an I/O operation FVP determines if it can serve it from flash. When it’s a write I/O operation the write goes straight to the storage system and the data is copied to the flash device. FVP acknowledges the completion of the write operation to the application after it receives the acknowledge from the storage system.

1-wt

Write I/O operations are not accelerated by the flash devices when selecting this write policy, but all subsequent read operations on that particular data are served from flash. Still write operations will eventually benefit because read operations on the flash footprint do not traverse across the network to hit the storage system. This reduces the number of request hitting the storage system and lowering the bandwidth consumption, therefor lower latencies can be expected in virtual infrastructures due to the mixed workload, the exception to this story is if you primarily run write intensive workloads.

First reads / false writes
Not all data is written by a virtual machine first before reading it, read operations can happen before a write operation, such as loading an operating system or opening a file. This operation is called a false write. A false write itself cannot be accelerated by any of two write policies and the access time of this first read is subject to the performance of the storage system. However FVP copies all incoming I/O from the storage system to the flash device providing acceleration on the subsequent reads.

2-fw

Write back policy
Write back policy accelerates both read and write operations. When an application issues a write I/O operation FVP forwards the command to the flash device. The flash devices acknowledges the write to the application first and completes the write operation to the storage system in the background. This results in the application seeing flash-level latencies while FVP deals with latency and throughput performance levels of the storage system.

3-wb

Flash replicas to protect delayed writes
When using Write back, delayed writes occurs. A delayed write is data that resides on the flash but that has not been written (destaged) to the storage system. In the time window between committing data to the flash and destaging the data to the storage system a risk exists where that the ESXi host goes down or the flash device fails. The problem is that when the virtual machine is started again on another host it expects the data to be available as before the outage it received acknowledgements on all previous writes

To protect against these failures and to provide an environment where all data is correct, FVP provides flash replicas. The write back policy provides the option to have 0, 1 or 2 flash replicas of a given virtual machine. Please note that FVP allows you to configure a write policy per virtual machine, you can have an environment where virtual machines run in write-through mode, while others have 0, 1 or 2 replicas.

When a virtual machine is configure with a write back policy with 1 replica, FVP forwards the write to the local flash device and a remote flash device. Once they local flash and the remote flash device acknowledge, FVP acknowledges the write completion to the application.

4-replica

It is the responsibility of the source host running the virtual machine to destage the delayed writes to the array. If the flash device, connection to the datastore or the complete source host goes down then one of hosts containing a replica will take over the job of destaging the delayed writes. In case you are wondering FVP leverages the vMotion network to transfer data to the replica host.

Clustering technology is necessary
Although it might not be obvious but accelerating writes in a virtual infrastructure is a big challenge. The solution needs to provide data acceleration on behalf of the virtual machines in front of a clustered file system where data can be modified from any host connected to the datastore. In addition the solution is subject to clustered operations such as vMotion while making sure that the data is always correct and available. To solve this FVP is a full clustered solution allowing the virtual machine flash footprint available to all participating hosts. Remote flash footprint and the FVP clustering technology will be the focus of the next article.

Frank Denneman

Follow me on Twitter, visit the facebook fanpage of FrankDenneman.nl, or add me to your Linkedin network

6 Responses

  1. Erik Lenten says:

    Nice read and easy to understand. Now the big question is: When you bought into Pernix and you reduced your footprint into backend storage what will happen when your delayed writes cannot be send to backend storage fast enough (ie Pernix is acknowledging very fast but the flash fills up, at some point you need to slow down in accepting writes which get's you back to orginal speeds? Might be a bit theoretical but you need to do smart things to the write operation before writing them to the backend storage system. (dedup, compression, sequentially writing them etc).
    /Erik

  2. Peter says:

    Great post Frank. One thing:"When using Write through, delayed writes occurs." - I think you meant to say write-back. Write-back certainly poses challenges that FVP appears to solve. Data integrity is key so this is a feature we can't live without if we want to accelerate writes.

  3. Pete says:

    Great post Frank. With regards to the Write-back policy, you state “The flash devices acknowledges the write to the application first and completes the write operation to the storage system in the background.” Can you provide more detail on that? Is the flush of that IO to the physical device occurring almost immediately afterward? Or is it more lazy/opportunistic based on the availability? And also, how does the total cache size come into play here? Is there a threshold relative to the total size of the cache in which the FVP starts evacuating the data? I’m also interested to know the pattern that one should see when working in write-back mode with consistently very heavy writes (total IOPS), or at least a heavily weighted write bias (say, 80% writes)

  4. Matt says:

    One minor correct, maybe... "When using Write through, delayed writes occurs."... I think you meant to say write back?

    Is there any information on the increased risk of data corruption when using write back with replicas? Can/does Pernix provide statistics on what benefit the workload (in terms of estimated reduction in write latency) would see if write-back was enabled or would we need to leverage something like Cloud Physics?

  5. Josh Coen says:

    Nice article Frank. The first sentence under the 'Flash replicas to protect delayed writes' heading, you say "When using Write through, delayed writes occurs" -- I believe you meant "When using Write back", correct?

    Are there impacts related to the speed in which FVP is able to destage delayed writes? More to the point, is there benefit to having a disk subsystem that also has write cache which is configured for write back?

    -Josh

  6. chethan says:

    Many of you have posted this comment - "When using Write through, delayed writes occurs". I can answer this on Frank's behalf (by virtue of being his colleague ;-)). Yes, he meant Write Back.

    -Chethan