The controller in an AAA array who can issue commands is called the managing controller, paths to the LUN via ports of this controller are called optimized paths. IO sent to a port of the non-owning controller must be transferred to the owning controller internally and increases latency and have impact on the performance of the array. Due to this, paths leading to the LUN via the non-managing controller are called non-optimized paths.

Pluggable Storage Architecture
The default Native Multipathing Plugin (NMP) used by the storage stack in ESX4 uses two sub-plugins, the Storage Array Type Plug-in and Path Selection Plugins. Storage Array Type Plugins is used for path handling and the Path Selection Plugins is used for Path selection.
It is interesting to know that the SATP is associated with all the physical paths to the array and is configured globally per array, but the PSP can be configured per LUN.

Storage Array Type Plugins
A default Storage Array Type Plugins (SATP) is included for each supported array as well as a generic SATP for non-specified storage arrays. The two SATP available for the HP EVA are:

• VMW_SATP_EVA (array specific)
• VMW_SATP_ALUA (generic SATP)

The problem with the array specific VMW_SATP_EVA is that it doesn’t use Target Port Group Support (TPGS), Funny thing is that TPGS is critical when it comes to determining optimized and non-optimized paths. Therefore the VMware HCL list the generic VMW_SATP_ALUA as the supported SATP on EVA Active-Active arrays.

Target Port Group Support
ALUA uses Target Port Group Support to determine the access characteristics of the path to a LUN. A port on a storage processor is called a target port and belongs to a target port group (TPG). All target ports belonging to the same TPG are always in the same Asymmetric Access State (AAS) relative to a LUN. There are a multiple of access states for a Target port:

• Active/Optimized
• Active/Non-optimized
• Standby
• Unavailable
• Transitioning

Note the grouping of Target Ports per TPG per Controller are vendor specific!

Active/Optimized
A target port reporting the active/optimized state belongs to the managing owner of the LUN and communicates directly with the LUN.

Active/Non-Optimized
A target port reporting the active/non-optimized state belongs to the non-managing owner of the LUN and the controller must send the IO via the multilink-ports to the managing controller of the LUN.

Transitioning
If a target port is in a Transitioning state, the ownership of the LUN is being transferred between controllers, this can occur if a hardware failure happened on the managing controller or when the threshold of proxy IO’s for the specific LUN is reached, this is called an Implicit Lun Transfer (ILT)

For example, a ESX host with two HBA’s connected to a EVA 8100, SP A is the managing controller of LUN1. TPGS will report that the Active Optimized paths are HBA1:1:1 and HBA2:1:1. HBA1:2:1 and HBA2:2:1 are listed as the non-optimized paths.

Path Selection Plugin
It’s up to the Path Selection Plugin (PSP) to actually select the path to be used by the ESX host to communicate with the LUN. There are three PSP’s are available:

• Most Recently Used Path Selection
• VMW_PSP_RR Round Robin Path Selection
• VMW_PSP_FIXED Fixed Path Selection

The Fibre Channel SAN Configuration Guide list the following:

MRU: Selects the path the ESX/ESXi host used most recently to access the given device.
If this path becomes unavailable, the host switches to an alternative path and
continues to use the new path while it is available.

Fixed: Uses the designated preferred path, if it has been configured. Otherwise, it uses
the first working path discovered at system boot time. If the host cannot use
the preferred path, it selects a random alternative available path. The host
automatically reverts back to the preferred path as soon as that path becomes
available.

Round Robin (RR):Uses a path selection algorithm that rotates through all available paths enabling load balancing across the paths.

ALUA aware PSPs
Both MRU and Round Robin PSPs are ALUA aware.MRU selects the first working optimized path discovered at system boot time and uses non-optimized paths when all optimized paths are dead. When an optimized path becomes available again, MRU will switch back to the optimized path. This differs from traditional MRU behavoir where a administrator must manually fail back to a path. Per default Round Robin will issue IO across all optimized paths and will use non-optimized paths only if no optimized paths are available.

Selecting the Fixed PSP on an ALUA enabled array defeats the purpose of ALUA support of the NMP architecture. If the preferred path is configured to use a non-optimized path, ESX will use this path to issue IO, even though optimized paths might be available. My advice is to stick with MRU or Round Robin when using a ALUA aware SATP.

Note! When if you have a mixed environment of ESX 3.5 and ESX 4 set the preferred paths to the LUNs according to the ALUA optimized paths listed in ESX4. If preferred paths are configured to use the non-optimized paths, repeatedly
Implicit LUN transfers can occur

VMware list MRU as the default supported PSP in the HCL, I asked VMware Global Support Services if Round Robin is supported even if it is not listed in the HCL.

If the partner recommends RR for their certified arrays on ESX 4.0, then they provide their customers with the directions for configuring it and they support them. We will support it but not list it in the HCL

MRU or RR
So the question arises, which PSP should I select?

If you use MSCS configurations, using Round Robin is unsupported on the LUNs that are part of MSCS VMs.

By selecting MRU ESX will only use the first optimized path discovered at boot, where Round Robin uses all optimized paths. MRU will use the first optimized path and does not load balance across HBA’s. ie. HBA1->LUN1, HBA2-LUN2, HBA1->LUN3, as where Round Robin will utilize all optimized paths of both HBA’s to send IO. Almost every vendor list Round Robin as best practice when using an ALUA array, generating better load-balance and aggregation of the available uplinks.

Another point of interest is the IOOperationlimit, this setting specifies how much IO commands will be sent across one path before switching over to the next available optimized path. Before following best practise of your vendor, my advice is to read Chad’s post first: vSphere 4 NMP RR IOoperationsLimit bug and workaround.

Continuous Access
So how about the impact of Continuous Access on the path load balancing in ESX 4?
When a LUN is being replicated it becomes a member of a CA Data Replication Group (DR Group). The DR Group is a Virtual Controller Software (VCS) construct and enforces the member LUNs to have the following behavior:

• Disk in DR Group will fail over together
• To preserve write order within the DR Group
• Share a log disk
• Replicate to the same destination array

To meet these requirements, CA will assign one controller as managing controller for all the member LUNS in the DR Group. Because all the target ports of controller belong to the same TPG, these target ports will report to ESX as the Active/Optimized paths to the LUNs of the DR Group. This behavior will impact the design of the DR Groups, how many LUNs will become a part of one DR Group? This will impact the load across both controllers, quite easily an imbalance can occur when “DR-Group managed controller affiliation” is not taken into consideration.

Please be aware of the limitation of the amount of DR-Groups and the use of DR tunnels. The path between the source and destination controller is called a DR Tunnel (DRT), to ensure IO consistency CA uses only one path per DR Group. A DR group is allocated a set of write and copy resources per DR Tunnel and these resources are not infinite. When heavy traffic occurs, DR groups can temporarily run out of allocated write resources. Running out of write resources will have impact on the write performance of the virtual machine.

For these reasons designing a balanced DR Group is something you must do together with your SAN-administrators, because the ratio DR Groups\DR Tunnels can have impact of your virtual environment as well.

Please read the following great posts as well:
“what is ALUA exactly” and “a couple of important (ALUA and SRM) Notes”

ESX4 ALUA, TPGS and HP CA is a post from: frankdenneman.nl

If you want to learn about threshold of certain metrics in ESXtop, please check out the ESXtop metric bible featured on Yellow-bricks.com.

ESXtop is a great tool to view and measure certain criteria in real time, but sometimes you want to collect metrics for later reference. If this is the case, the tool vscsiStats might be helpful. vscsiStats is a tool to profile your storage environment and collects info such as outstanding IO, seekdistance and many many more. Check out Duncan’s excellent article on how to use vscsiStats.

Because vscsiStats will collect data in a .csv file you can create diagrams, Gabe written an article how to convert the vscsiStats data into excel charts.

Identify storage performance issues is a post from: frankdenneman.nl

Lefthand, not the tradional Head-Shelf configuration
HP lefthand SANs are based on iSCSI and are formed by Storage nodes. In traditional storage architectures, a controller manages arrays of disk drives. A Lefthand SAN is composed of storage modules. A Network Storage Module 2120 G2 (NSM node) is basically a HP DL185 server with 12 SAS or SATA drivers running SAN/iQ software.  

This architecture enables the aggregation of multiple storage nodes to create a storage cluster and this solves one of the toughest design questions when sizing a SAN.  Instead of estimating growth and buying a storage array to “grow into”, you can add storage nodes to the cluster when needed. This concludes the sales pitch.

But this technique of aggregating separate NSM nodes into a cluster raises some questions.  Questions such as;

• Where will the blocks of a single LUN be stored; all on one node, or across nodes?
• How are LUNs managed?
• How is datapath load managed?
• What is the impact of failure of a NSM node ?

Block placement and Replication level
The placement of blocks of a LUN depends on the configured replication level. Replication level is a feature called Network RAID Level. Network RAID stripes and mirrors multiple copies of data across a cluster of storage nodes. Up to four levels of synchronous replication at LUN level can be configured;

• None
• 2-way
• 3-way
• 4-way

Blocks will be stored on storage nodes according to replication level. If a LUN is created with the default replication level of 2-way, two authorative blocks are written at the same time to two different nodes. If a 3-way replication level is configured, blocks are stored on 3 nodes. 4-way = 4 nodes. (Replication cannot exceed the number of nodes in the cluster)
SAN IQ will always start to write the next block to the second node containing the previous block. A picture is worth a thousand words.

Node order
The data in which blocks are written to the LUN is determined not by node hostname but by the order in which the nodes are added to the cluster.  The order of the placement of the nodes is extremely important if the SAN will span two locations. More information on this design issue later.

Virtual IP and VIP Load Balancing
When setting up a Lefthand Cluster, a Virtual IP (VIP) needs to be configured. A VIP is required for iSCSI load balancing and fault tolerance. One NSM node will act as the VIP for the cluster, if this node fails, the VIP function will automatically failover to another node in the cluster.

The VIP will function as the iSCSI portal, ESX servers use the VIP for discovery and to log in to the volumes.  ESX servers can connect to volumes two ways. Using the VIP and using the VIP with the option load balancing (VIPLB) enabled.  When enabling VIPLB on LUNs, the SAN/iQ software will balance connections to different nodes of the cluster.

Configure the ESX iscsi inititiator with the VIP as destination address.  The VIP will supply the ESX servers with a target address for each LUN.  VIPLB will transfer initial communication to the gateway connection of the LUN.  Running the vmkiscsi-util command shows the VIP as portal and another ip address as target address of the LUN

root@esxhost00 vmfs]# vmkiscsi-util -i -t 58 -l
***************************************************************************
      Cisco iSCSI Driver Version … 3.6.3 (27-Jun-2005 )
***************************************************************************
TARGET NAME             : iqn.2003-10.com.lefthandnetworks:lefthandcluster:1542:volume3
TARGET ALIAS            :
HOST NO                 : 0
BUS NO                  : 0
TARGET ID               : 58
TARGET ADDRESS          : 148.11.18.60:3260
SESSION STATUS          : ESTABLISHED AT Fri Sep 11 14:51:13 2009
NUMBER OF PORTALS          : 1
PORTAL ADDRESS 1        : 148.11.18.9:3260,1
SESSION ID              : ISID 00023d000001 TSIH 3b1

Gateway Connection
This target address is what Lefthand calls a gateway connection. The gateway connection is described in the Lefthand SAN User Manual (page 561) as follows;

Use iSCSI load balancing to improve iSCSI performance and scalability by distributing iSCSI sessions for different volumes evenly across storage nodes in a cluster. ISCSI load balancing uses iSCSI Login-Redirect.  Only initiators that support Login-Redirect should be used. When using VIP and load balancing, one iSCSI session acts as the gateway session. All I/O goes through this iSCSI session. You can determine which iSCSI session is the gateway by selecting the cluster, then clicking the iSCSI Sessions tab. The Gateway Connection column displays the IP address of the storage node hosting the load balancing iSCSI session.

SAN/IQ will designate a node to act as gateway connection for the LUN, the VIP will send the IP address of this node as target address to all the ESX host. This means every host that use the LUN will connect to that specific node and this storage node will handle all IO for this LUN.

This leads to the question, how will the GC handle IO for blocks not locally stored on that node?  When a block is requested that is stored on another node, the GC will fetch this block. All nodes are aware which block is stored on which node. The GC node will fetch this block of one of the nodes it’s stored and will send the results back to the ESX host.

Gateway Connection failover
Most Clusters will host more LUNs than it has available nodes. This means that each node will host the gateway connection role of multiple LUNs. If a node fails, the GC role will be transferred to the other nodes in the cluster. But when a NSM node returns back online, the VIP will not failback the GC roles. This will create an unbalance it the cluster, which needs to be solved as quickly as possible.  This can be done by issuing the RebalanceVIP for the volume from the cli.

Ken Cline asked me the question:

How do I know when I need to use this command? Is there a status indicator to tell me?

Well actually there isn’t and that is exactly the problem!
After a node failure, you need to be aware of this behaviour and you will have to rebalance a volume yourself by running the RebalanceVIP command.
The Lefthand CMC does not offer this option or some sort of alert.
 
Network Interface Bonds
How about the available bandwidth? Lefthand nodes come standard with two 1GB nics. The two nics can be placed in a bond. A NSM node has three NIC bond configurations;

1. Active – Passive
2. Link Aggregation (802.3 ad)
3. Adaptive Load Balancing

The most interesting is the Adaptive Load Balancing (ALB).  Adaptive Load Balancing combines the benefits of the increased bandwidth of 802.3ad with the network redundancy of Active-Passive. Both NICS are made active and they can be connected to different switches, no additional configuration on physicall switch level is needed.  

When an ALB bond is configured, it creates an interface. This interface balances traffic through both nics. But how will this work with the iSCSI protocol? In RFC 3270 (http://www.ietf.org/rfc/rfc3720.txt) iSCSI uses command connection allegiance;

For any iSCSI request issued over a TCP connection, the corresponding response and/or other related PDU(s) MUST be sent over the same connection.  We call this “connection allegiance”. 

This means that the NSM node must use the same MAC address to send the IO back. How will this affect the bandwidth? As stated in the ISCSI SAN configuration guide; “ESX Server‐based iSCSI initiators establish only one connection to each target.”.

It looks like ESX will communicate with the gateway connection of the LUN with only NIC. I asked Calvin Zito (http://twitter.com/HPstorageGuy) to educate me on ALB and how it handles connection allegiance.

When you create a bond on an NSM, the bond becomes the ‘interface’ and the MAC address of one of the NICs becomes the MAC address for the bond.  The individual NICs become ‘slaves’ at that point.  I/O will be sent to and from the ‘interface’ which is the bond and the bonding logic figures out how to manage the 2 slaves behind the scenes.  So with ALB, for transmitting packets, it will use both NICs or slaves, but they will be associated with the MAC of the bond interface, not the slave device.

The bond uses the same IP and MAC address of the first onboard NIC. This means the node will uses both interfaces to transmit data, but only one to receive.

Chad Sakac (EMC), Andy Banta (VMware) and various other folks has written a multivendor post explaining how ESX and vSphere handles iSCSI traffic. A must read!
http://virtualgeek.typepad.com/virtual_geek/2009/01/a-multivendor-post-to-help-our-mutual-iscsi-customers-using-vmware.html#more

Design issues;
When designing a Lefthand SAN, these points are worth considering;

Network RAID level Write performance
When 2-way replication is selected, blocks will be written on two nodes simultaneously, if a LUN is configured with 3-way replication, then blocks must be replicated to three nodes . Acknowledgements are given when blocks are written in cache on all the participating nodes. When selecting the replication level, keep in mind that higher protection levels leads to less write performance.

Raid Levels NSM
Network RAID offers protection for storage node failure, but it does not protect against diskfailure within a storage node. Disk RAID levels need to be configured at Storage Node level, unlike most traditional arrays where raid level can be configured per LUN level. It is possible to mix storage nodes with different configurations of  RAID within a cluster, but the this can lead a lower useable capacity.

For example, the cluster exists of 12 TB nodes running RAID 10. Each node will provide 6TB in usable storage. When adding two 12TB nodes running RAID 5, each provides 10 TB of usable storage. However, due to the restrictions of how the cluster uses capacity, the NSM nodes running RAID 5 will still be limited to 6 TB per storage node. This restriction is because the cluster operates at the smallest usable per-storage node capacity.
The RAID level of the storage node must first be set before it can join a Cluster. Check RAID level of clusternodes before configuring the new node, because you cannot change the RAID configuration without deleting data.

Combining Replication Levels with RAID levels
RAID levels will ensure data redudancy inside the storage node, while Replication levels will ensure data redundancy on storage node level. Both higher RAID levels and Replication levels offer greater data reduncancy, but will have impact on capacity and performance. RAID5 with 2-way replication seems to be the sweet spot for most implementations, but when high available data protection is needed, Lefthand recommends 3-way replication with raid 5, ensuring triple mirroring with 3 parity blocks available.

I would not suggest RAID 0 with replication, because rebuilding a RAID set will always be quicker that copying an entire storage node over the network.

Node placement
Mentioned previously, the data in which blocks are written to the LUN is determined by the order in which the nodes are added to the cluster.  When using 2-way replication, blocks are written to two consecutive nodes.  When designing a cluster the order of the placement of the nodes is extremely important if the SAN will be placed in two separate racks or even better span two locations.

Because the 2-way replication writes blocks on two consecutive nodes, adding the storage nodes to the cluster in alternating order will ensure that data is written to each rack or site.

 
When nodes are added in the incorrect order or if a node is replaced, the general setting tab of the cluster properties allows you to “promote” or “demote” a storage node in the logical order. This list is the leading for the “write” order of the nodes.  

Management Group and Managers
In addition to setting up data replication, it is important to setup managers. Managers play an important role in controlling data flow and access of clusters. Managers run inside a management group. Several storage nodes must be designated to run the manager service. Because managers use a voting algorithm, a majority of managers needs to be active to function.  This majority is called a Quorum. If quorum is lost, access to data is lost. Be aware that acces to data is lost, not the data itself.
An odd number of managers is recommended,  as an (low) even number of manager can get in a certain state where no majority  is determined.  The maximum number of managers is 5.

Failover manager
A failover manager is a special edition of a manager. Instead of running on a storage node, the failover manager runs as a virtual appliance. A failover manager only function is maintaining quorum. When designing a SAN spanning two sites, running a failover manager is recommended. The optimum placement of the failover manager is a third site. Place an even amount of managers in both sites and run the failover manager at an independent site.

If a third site is not available, run the failover manager local on a server, creating a logical seperated site

Volumenames
And the last design issue, volume names cannot bechanged.  The volumename is the only setting which can’t be edited after creation. Plan your naming convention carefully, otherwise you will end up recreating volumes and restoring data. If someone of HP is reading this, please change this behaviour!

Lefthand SAN – Lessons learned is a post from: frankdenneman.nl

I’ve tried to use the Centralized Management Console, but it’s tedious and error prone work. Mind numbing repetitive exercises makes me ask really dumb questions on twitter such as where did SCSI id 7 go? <homer>D’oh!</homer>. So to protect myself from further bashing and being ridiculed I started to search for the Lefthand CLI to be able to automate the creation and assignment of volumes on a Lefthand SAN.

CLIQ 
A CLI is available for the HP EVA series, but not much info is being published about the Lefthand Command Line Interface (CLIQ). But deep down in the bowels of the HP site a document about the CLIQ is published;

The “User Manuals API CLI User Manual” http://h20000.www2.hp.com/bc/docs/support/SupportManual/c01806770/c01806770.pdf?jumpid=reg_R1002_USEN

(But help in the SSH session will offer almost the same amount of info.)

Log in info
Storage nodes configured with SAN/IQ 8.0 are standard equipped with the CLIQ and can be accessed by SSH. Log in to the CLIQ via SSH using the node IP address, but use port 16022 instead of the default port 22. When nodes are a part of a management group any node can be used to access the CLIQ. Open a SSH session to a random node and use the management group user and password information.

Use of CLIQ
The CLIQ is not case sensitive and it the ordering of parameters is not specified. Any order will do.

Let create a volume
createVolume volumeName=VMFS001 clusterName=ESX-CLUSTER01 size=1TB Replication=2 thinProvision=1 description=”VI Datastore VMFS001″

Size:
The following sizes can be used: MB, GB, and TB

Replication:
The replication level for the volume 1=none,2-way,3-way or 4-way

ThinProvision:
• 0 – Full-provisioning
• 1 – Thin-provisioning

Assigning the volume
assignvolume volumeName=VMFS001 initiator=iqn.1998-01.com.vmware:esx001.acme.com;iqn.1998-01.com.vmware:esx002.acme.com

An IQN is used to assign the server to the volume, if a server item is preconfigured in the CMC, the CMC will list the server item at the Assigned Server tab of the volume.

Assigning multiple servers to a volume must be done with one command, if two separate commands are being used, the last command will overwrite the first command. Use a ; to delimit IQNs.

Those two commands saved me a lot of trouble and lots of unnecessary tedious work, hopefully you can benefit from these as well.

Deploying Volumes with Lefthand CLIQ is a post from: frankdenneman.nl

I’ve tried to use the Centralized Management Console, but it’s tedious and error prone work. Mind numbing repetitive exercises makes me ask really dumb questions on twitter such as where did SCSI id 7 go? <homer>D’oh!</homer>. So to protect myself from further bashing and being ridiculed I started to search for the Lefthand CLI to be able to automate the creation and assignment of volumes on a Lefthand SAN.

CLIQ 
A CLI is available for the HP EVA series, but not much info is being published about the Lefthand Command Line Interface (CLIQ). But deep down in the bowels of the HP site a document about the CLIQ is published;

The “User Manuals API CLI User Manual” http://h20000.www2.hp.com/bc/docs/support/SupportManual/c01806770/c01806770.pdf?jumpid=reg_R1002_USEN

(But help in the SSH session will offer almost the same amount of info.)

Log in info
Storage nodes configured with SAN/IQ 8.0 are standard equipped with the CLIQ and can be accessed by SSH. Log in to the CLIQ via SSH using the node IP address, but use port 16022 instead of the default port 22. When nodes are a part of a management group any node can be used to access the CLIQ. Open a SSH session to a random node and use the management group user and password information.

Use of CLIQ
The CLIQ is not case sensitive and it the ordering of parameters is not specified. Any order will do.

Let create a volume
createVolume volumeName=VMFS001 clusterName=ESX-CLUSTER01 size=1TB Replication=2 thinProvision=1 description=”VI Datastore VMFS001″

Size:
The following sizes can be used: MB, GB, and TB

Replication:
The replication level for the volume 1=none,2-way,3-way or 4-way

ThinProvision:
• 0 – Full-provisioning
• 1 – Thin-provisioning

Assigning the volume
assignvolume volumeName=VMFS001 initiator=iqn.1998-01.com.vmware:esx001.acme.com;iqn.1998-01.com.vmware:esx002.acme.com

An IQN is used to assign the server to the volume, if a server item is preconfigured in the CMC, the CMC will list the server item at the Assigned Server tab of the volume.

Assigning multiple servers to a volume must be done with one command, if two separate commands are being used, the last command will overwrite the first command. Use a ; to delimit IQNs.

Those two commands saved me a lot of trouble and lots of unnecessary tedious work, hopefully you can benefit from these as well.

Deploying Volumes with Lefthand (CLIQ) is a post from: frankdenneman.nl

But when a (lefthand) LUN is presented to the ESX server, it will use a unique target ID instead of a unique LUN ID. (vmhba1:2:0) I have done some testing and discovered that the assigned target ID can differ from ESX server to ESX server.

I’m curious if the target ID is used when creating the UUID of the VMFS datastore. And I’m especially interested in what will happen if multiple ESX hosts are going to communicate with the LUN when all the ESX hosts will use a different “path”

Maybe there isn’t a problem at all and different targets will work well, but is seems that I need to stop thinking in FC solutions and get used to iscsi Lefthand “quirks”.

I’ve read the field guide for VMware infrastructures, I googled on terms like “iscsi lun scheme’s” but I cannot seem to find any real-life scenario’s.
Maybe my Google skills are pitiful at the moment, and maybe someone can shed some lights on this and how they solved this “problem”.

My first lefthand ISCSI VI architecture is a post from: frankdenneman.nl

 
Qlogic Execution Throttle
Qlogic has a firmware setting called „Execution Throttle” which specifies the maximum number of simultaneous commands the adapter will send. The default value is 16, increasing the value above 64 has little to no effect, because the maximum parallel execution of SCSI operations is 64.
(Page 170 of ESX 3.5 VMware SAN System Design and Deployment Guide)
 
If the QD is increased, execution throttle and the DSNRO must be set with similar values, but to calculate the proper QD the fan-in ratio of the storage port needs to be calculated.
 
Target Port Queue Depth
A queue exist on the storage array controller port as well, this is called the “Target Port Queue Depth“. Modern midrange storage arrays, like most EMC- and HP arrays can handle around 2048 outstanding IO’s. 2048 IO’s sounds a lot, but most of the time multiple servers communicate with the storage controller at the same time. Because a port can only service one request at a time, additional requests are placed in queue and when the storage controller port receives more than 2048 IO requests, the queue gets flooded. When the queue depth is reached, this status is called (QFULL), the storage controller issues an IO throttling command to the host to suspend further requests until space in the queue becomes available. The ESX host accepts the IO throttling command and decreases the LUN queue depth to the minimum value, which is 1!
 
The VMkernel will check every 2 seconds to check if the QFULL condition is resolved. If it is resolved, the VMkernel will slowly increase the LUN queue depth to its normal value, usually this can take up to 60 seconds.
 
Calculating the queue depth\Execution Throttle value
To prevent flooding the target port queue depth, the result of the combination of number of host paths + execution throttle value + number of presented LUNs through the host port must be less than the target port queue depth. In short T => P * q * L

T = Target Port Queue Depth
P = Paths connected to the target port
Q = Queue depth
L = number of LUN presented to the host through this port

Despite having four paths to the LUN, ESX can only utilize one (active) path for sending IO. As a result, when calculating the appropriate queue depth, you use only the active path for “Paths connected to the target port (P)” in the calculation, i.e. P=1.

But in a virtual infrastructure environment, multiple ESX hosts communicate with the storage port, therefore the QD should be calculated by the following formula:
 
T => ESX Host 1 (P * Q * L) + ESX Host 2 (P * Q * L) ….. + ESX Host n (P * Q * L)
 
For example an 8 ESX host cluster connects to 15 LUNS (L) presented by an EVA8000 (4 target ports)* An ESX server issues IO through one active path (P), so P=1 and L=15.
 
The execution throttle\queue depth can be set to 136,5=> T=2048 (1 * Q * 15) = 136,5
But using this setting one ESX host can fill the entire target port queue depth by itself, but the environment exists of 8 ESX hosts. 136,5/ 8 = 17,06
 
In this situation all the ESX Host communicate to all the LUNs through one port. Which does not happen in many situations if a proper load-balancing design is applied. Most arrays have two controllers and every controller has at least two ports. In the case of a controller failure, at least two ports are available to accept IO requests.
 
It is possible to calculate the queue depth conservatively to ensure a minimum decrease of performance when losing a controller during a failure, but this will lead to underutilizing the storage array during normal operation, which will hopefully be 99,999% of the time. It is better to calculate a value which utlilize the array properly without flooding the target port queue.
 
If you assume that multiple ports are available and that all LUNs are balanced across the available ports on the controllers, it will effectively quadruple the target port queue depth and therefore increase the values of the execution throttle in the example above to 68. Besides the fact that you cannot increase this value above 64, it is wise to decrease the value to a number below max value, it will create a buffer for safety 

What’s the Best Setting for Queue Depth?
The examples mentioned are pure worst case scenario stuff, most of the time it is highly unlikely that all hosts perform at their maximum level at any one time. Changing the defaults can improve throughput, but most of the time it is just a shot in the dark. Although you are configuring your ESX hosts with the same values, not every load on the ESX server is the same. Every environment is different and so the optimal queue depths would differ. One needs to test and analyse its environment. Please do not increase the QD without analysing the environment; this can be more harmful than useful.

Increasing the queue depth? is a post from: frankdenneman.nl

AA type
The HP EVA array range (4x,6x,8x) are categorized by VMware as active-active arrays. Active/active storage arrays are further divided into two categories:

• Symmetrical Active-Active (SAA)
• Asymmetric Active-Active (AAA)

SAA arrays are considered by many as true active-active arrays. IO request can be issued over all paths and every controller in the array can accept and send IO to the LUN. EMC DMXs and HP XPs are SAA arrays.

Asymmetric Active-Active and Asymmetric Logical Unit Access (ALUA) compliance
The arrays from the EVA family are dual controller AAA arrays and are compliant with the SCSI Asymmetric Logical Unit Access (ALUA) standard for LUN access/failover and I/O processing.

In an Asymmetric Active-Active Array both controllers are online and both can accept IO, but only one controller is assigned as the owning controller of the LUN. The owning controller can issue IO commands directly to the LUN, this is called an optimized path. The non-owning controller, or proxy controller can accept IO commands, but cannot communicate with the LUN. This is called an non-optimized path. If a read request reaches the array through the proxy controller, it will be forwarded to the owning controller of the LUN. This behavior is called a proxy read.

It looks like I’m describing an active/passive array, but the main difference is that Active-Passive arrays transfer the ownership as soon as it receives IO on the non-owning controller. IO request are transferred between the controllers on the back-end of an AAA, making the process transparent to the ESX host. In a asymmetric Active-Active Array, storage processor ports have a certain port state with respect to a given LUN:

• Active\Optimized
• Active\Non-Optimized
• Standby
• Unavailable
• Transitioning

If the access characteristics of a port differs from another port asymmetric logical unit access occurs. ALUA provides a way to allow to report the states of the port to the host, the host can use the states of the port to prioritize paths. Unfortunately, ESX 3.x does not support ALUA, what that actually means is that ESX does not has the ability to identify, the LUN controller ownership, or to put it more precisely the the active\optimized or active non-optimized paths. ALUA support is implemented in the PSA (Pluggable Storage Architecture) of ESX4.

Proxy reads
Read IO requests received by the proxy controller (1) are sent to the owning controller (2), which retrieves the data from disk (3), caches the read data (4) and mirrors the data to the cache of the proxy controller (5). The proxy controller satisfies the host read request (6), making this process transparent to the ESX host. Proxy reads add unnecessary latency to the IO request. It also creates higher mirror port utilization. IO-Write commands to the proxy controller suffer less performance impact. Due to fault protection all writes are mirrored in both controllers’ caches, but the owning controller is still responsible for flushing the data to disk.

Mirror Port
The controller mirror ports are used for cache writes and proxy reads. If you setup your environment with correct multipathing, the mirror ports will only have to handle the write mirroring.

Implicit LUN transition
If the EVA array detects, in a period of 60 minutes, that at least 66% of the total read request to a LUN are proxy reads, ownership is transitioned to the non-owning proxy controller and making it the owning controller. Justin’s powershell script assigns the same path to every server the same way. This way the EVA should switch the managing controller within the hour. (If you have multiple ESX hosts run multiple VM’s on the LUN of course)

Continuous Access DR Groups
If HP Continuous Access (CA) is used to replicate LUNs between two arrays, extra care must be taken when planning to use a LUN balancing script. CA sets up replication relationships between LUNs on different arrays; this is called a Data Replication (DR) group. A DR group can be considered a consistency group, all LUNs in a DR group fail over together, share a log and preserve write order within the group. Because of this requirements, one controller is assigned as managing (owning) controller for all member LUNS.

Implicit LUN transition
Because one controller manages a group of LUNs, CA disables implicit LUN transitions for all DR group members. Mixing Implicit LUN transition (ILT) together with a large DR group can be a recipe for disaster. Imagine if ILT was switched on and the array detects too much proxy reads on a LUN in the group.

The array will evoke an ILT for that LUN. Because all the LUN in the DR group must use the same controller to meet the consistency requirement all members are transitioned together.

So far so good, but you can bet on it that a new proxy read situation appears, due to the multiple hosts communicating with the disks. This will evoke another Implicit LUN transition.

And now we are back where we started. Enabling Implicit LUN transition can create some sort of bouncing group of LUNs between controllers. This is a sure way of giving you SAN administrator a small stroke. The downside of disabling ILT on a DR group is the possibility that LUNs may experience excessive proxy reads.

Using auto balance script on CA managed LUNs
Because ESX does not inquire about the status of the port when testing the path, the auto balance scripts cannot discover the optimized path. When paths are being initialized during the boot of the ESX host, it will just enumerate the paths available to it. Scanning the first controller and the lowest device number, this path might not necessarily be the path to the managing controller. Because all paths are active, the script will select the next path for a LUN. This can lead to IO requests arriving at the proxy controller and because implicit LUN transition is disabled, proxy reads will keep occurring.

Custom load balancing
If you have high workload intensity you might experience negative impact on IO performance when running a generic LUN balancing script. If you implement CA in your virtual infrastructure, it is better to take some time to design a custom load balance script. Using a well designed load balancing script along with fixed multipathing policy does not eliminate proxy reads, but it should only occur as a temporary condition during failures.

In the example, each DR group contains 5 LUNS, because every LUN in the DR group share the same managing controller, the optimized path for LUN 1-5 is through Storage Processor A. Alternating HBA’s are used to load balance on the ESX side.

Determining the managing controller
To correctly load balance on EVA arrays you must know which controller owns the LUN. Command View EVA shows the managing controller of the LUN, but checking several LUNs via Command View EVA can be a lengthy process.
Using Command view EVA (CVE)

• On the presentation tab of the LUN (vdisk) properties
• On the general tab of the DR Group properties

HP Storage System Scripting Utility (SSSU) can help you to speed up discovering the management controller.Using SSSU:

LS VDISK (Name will suffice, no need for complete path)
The owning controller is listed as online controller (controller name)

I’m not aware of any powershell tools to manage an EVA storage array.
My contacts at HP cannot confirm if any new tooling except SSSU and CVE will appear soon

HP CA and the use of LUN balancing scripts is a post from: frankdenneman.nl