Using esxtop to identify storage performance issues

Details

This article provides information about esxtop and latency statistics that can be used when troubleshooting performance issues with SAN-connected storage (Fibre Channel or iSCSI).

Solution

The interactive esxtop utility can be used to provide I/O metrics over various devices attached to a VMware ESX host.

Configuring monitoring using esxtop

To monitor storage performance per HBA:

Start esxtop by typing esxtop at the command line.
Press d to switch to disk view (HBA mode).
Press f to modify the fields that are displayed.
To view the entire Device name, press SHIFT + L and enter 38 in Change the name field size.
Press b, c, d, e, h, and j to toggle the fields and press Enter.
Press s, then 2 to alter the update time to every 2 seconds and press Enter.
See Analyzing esxtop columns for a description of relevant columns.

Note: The following options are only available in VMware ESX 3.5 and later.

To monitor storage performance on a per-LUN basis:

Start esxtop by typing esxtop from the command line.
Press u to switch to disk view (LUN mode).
Press f to modify the fields that are displayed.
Press b, c, f, and h to toggle the fields and press Enter.
Press s, then 2 to alter the update time to every 2 seconds and press Enter.
See Analyzing esxtop columns for a description of relevant columns.

For more information, see How to increase the width of the device field in esxtop to show the complete naa id (1035989).

To monitor storage performance on a per-virtual machine basis:

Start esxtop by typing esxtop at the command line.
Type v to switch to disk view (virtual machine mode).
Press f to modify the fields that are displayed.
Press b, d, e, h, and j to toggle the fields and press Enter.
Press s, then 2 to alter the update time to every 2 seconds and press Enter.
See Analyzing esxtop columns for a description of relevant columns.

Analyzing esxtop columns

This table lists the relevant columns and a brief description of these values:

Column	Description
CMDS/s	This is the number of IOPS (Input/Output Operations Per Second) being sent to or coming from the device or virtual machine being monitored
DAVG/cmd	This is the average response time in milliseconds per command being sent to the device
KAVG/cmd	This is the amount of time the command spends in the VMkernel
GAVG/cmd	This is the response time as it is perceived by the guest operating system. This number is calculated with the formula: DAVG + KAVG = GAVG

These columns are for both reads and writes, whereas xAVG/rd is for reads and xAVG/wr is for writes. The combined value of these columns is the best way to monitor performance, but high read or write response time it may indicate that the read or write cache is disabled on the array.

All arrays perform differently, howeverDAVG/cmd, KAVG/cmd, and GAVG/cmd should not exceed more than 10 milliseconds (ms) for sustained periods of time.

Note: VMware ESX 3.0.x does not include direct functionality to monitor individual LUNs or virtual machines usingesxtop. Inactive LUNs lower the average for DAVG/cmd, KAVG/cmd, and GAVG/cmd.

These values are also visible from the vCenter Server performance charts. For more information, see Performance Charts in the Basic System Administration Guide.

If you experience high latency times, investigate current performance metrics and running configuration for the switches and the SAN targets. Check for errors or logging that may suggest a delay in operations being sent to, received, and acknowledged. This includes the array’s ability to process I/O from a spindle count aspect, or the array’s ability to handle the load presented to it.

If the response time increases to over 5000 ms (or 5 seconds), VMware ESX will time out the command and abort the operation. These events are logged; abort messages and other SCSI errors can be reviewed inthe following logs:

ESX – /var/log/vmkernel
ESXi – /var/log/messages

The type of storage logging you may see in these files depends on the configuration of the server. You can find the value of these options by navigating to Host > Configuration > Advanced Settings > SCSI > SCSI.Log* or SCSI.Print*.