Adding, accessing LUNs, creating VMs or scanning and refreshing HBAs (hw iscsi) on ESX result in errors on the storage system:
idm: [ID 465152 kern.warning] WARNING: idm_refcnt_rele: refobj ffffff228be9f000, depth: 8, stacks(14): 0xfffffffff832fa0b, 0xfffffffff8377bc4, 0xfffffffff83792b6, 0xfffffffff8377470, 0xfffffffff8330ab2, 0xfffffffff83305ea, 0xfffffffff8338f1b, 0xfffffffffb86ea08,
vmkernel.log during vmhba rescan
cpu4:33303)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x89 (0x4130804461c0, 35530) to dev "naa.6000402e50000000482a4098f8184efd" on path "vmhba2:C0:T0:L0" Failed: H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2015-01-21T14:07:43.405Z cpu4:33303)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6000402e50000000482a4098f8184efd" state in doubt; requested fast path state update...
vmkernel log during vmhba update
cpu5:32833)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6000402e50000000482a4098f8184efd" state in doubt; requested fast path state update...
cpu5:32833)ScsiDeviceIO: 2337: Cmd(0x413080441e40) 0x89, CmdSN 0x4d from world 1303441 to dev "naa.6000402e50000000482a4098f8184efd" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
cpu4:1303441)FSAts: 1184: vol 'datastore1', lock at 4331520: Unlocking lock failed: Timeout:
cpu5:32833)ScsiDeviceIO: 2337: Cmd(0x413080441e40) 0x89, CmdSN 0x4d from world 1303441 to dev "naa.6000402e50000000482a4098f8184efd" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
cpu4:1303441)FSAts: 1184: vol 'datastore1', lock at 4331520: Unlocking lock failed: Timeout:
~ # esxcfg-rescan -u vmhba2
Error: Not all VMFS volumes were updated; the error encountered was 'Timeout'.
Error: Not all VMFS volumes were updated; the error encountered was 'Timeout'.
Setting pulse [HB state abcdef02 offset 3939840 gen 1 stampUS 488532767167 uuid 54b84c8e-953eb25c-44aa-001e675af992 jrnl <FB 0> drv 14.60] on vol 'datastore1' failed: Timeout
This is due to a problem where ATS only is not properly supported by the storage system.
ATS-only is enabled on the LUN (vmfs5 only)
# vmkfstools -Ph -v1 /vmfs/volumes/VMFS-volume-name
You see output similar to:
VMFS-5.54 file system spanning 1 partitions.
File system label (if any): ats-test-1
Mode: public ATS-only
VMFS-5.54 file system spanning 1 partitions.
File system label (if any): ats-test-1
Mode: public ATS-only
# esxcli system settings advanced list -o /VMFS3/HardwareAcceleratedLockingYou see output similar to this:Path: /VMFS3/HardwareAcceleratedLocking
Type: integer
Int Value: 1
The Int Value of 1 is indicating that VAAI features are enabled.
Disable ATS locking:
# esxcli system settings advanced set -i 0 -o /VMFS3/HardwareAcceleratedLocking
The Int Value of 0 is indicates that VAAI features are disabled.
rescan all the adapters and the errors should no longer appear.
The LUNs can now also be used for adding VMs and creating files / directories.
There are 2 ways to disable ATS
There are 2 ways to disable ATS
- Per datastore
# vmkfstools --configATSOnly 0 /vmfs/devices/disks/naa.xxxxxxxxxxxxxxxxxxxxxxxxxx
- Per esxi host
# esxcli system settings advanced set -i 0 -o /VMFS3/HardwareAcceleratedLocking
To re-enable just set 1
To check if it’s enabled or disabled:
vmkfstools -Ph -v1 /vmfs/volumes/VMFS-volume-name
You see output similar to:
VMFS-5.54 file system spanning 1 partitions.
File system label (if any): ats-test-1
Mode: public ATS-only
VMFS-5.54 file system spanning 1 partitions.
File system label (if any): ats-test-1
Mode: public ATS-only
Enabling/disabling takes effect immediately without requiring a reboot.
If VMs are running on the datastore, the setting can’t be changed, until all VMs are stopped or moved.