As the title suggests, all hosts see the LUNs... can't figure out the issue. Below is a capture of the vmkernel.log captured from the server as it was doing a Rescan (which results in a loop of activity in the logfile and lasts 10+ minutes with no change to the storage).
The SAN is preesnting the LUNs and successfully... so it has to be host side. In an act of desperation, yesterday I reinstalled ESXi on one of the troubled hosts. It was able to see the datastores. Today, they were no longer visible.
Thanks in advance for any help.
vmkernel.log:
2013-06-26T13:07:56.778Z cpu1:15038)Partition: 558: Read of GPT entries (hdrlba=1) failed on "naa.600144f0f9e18d0000005160917e0001": I/O error
2013-06-26T13:07:56.778Z cpu1:15038)Partition: 661: Read from primary gpt table failed on "naa.600144f0f9e18d0000005160917e0001".
2013-06-26T13:07:56.778Z cpu1:15038)ScsiDeviceIO: 2303: Cmd(0x4124003a4680) 0x88, CmdSN 0xabd from world 0 to dev "naa.600144f0f9e18d0000005160917e0001" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0.
2013-06-26T13:07:56.863Z cpu1:8193)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:07:57.005Z cpu1:8193)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:07:57.063Z cpu8:8776)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x410022110b10 network resource pool netsched.pools.persist.iscsi associated
2013-06-26T13:07:57.063Z cpu8:8776)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x410022110b10 network tracker id 1 tracker.iSCSI.192.168.70.10 associated
2013-06-26T13:07:57.091Z cpu1:8193)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:07:57.234Z cpu1:8193)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:07:57.378Z cpu1:8193)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:07:57.520Z cpu1:8193)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:07:57.563Z cpu1:8193)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.600144f0f9e18d0000005160917e0001" state in doubt; requested fast path state update...
2013-06-26T13:07:57.568Z cpu8:8776)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba38:CH:0 T:0 CN:0: iSCSI connection is being marked "ONLINE"
2013-06-26T13:07:57.568Z cpu8:8776)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000002 TARGET: iqn.2012-11.com.rhineaux:nex70 TPGT: 2 TSIH: 0]
2013-06-26T13:07:57.568Z cpu8:8776)WARNING: iscsi_vmk: iscsivmk_StartConnection: Conn [CID: 0 L: 192.168.70.50:49286 R: 192.168.70.10:3260]
2013-06-26T13:07:57.578Z cpu8:8603)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0. Act:NONE
2013-06-26T13:07:57.578Z cpu8:8603)ScsiDeviceIO: 2303: Cmd(0x4124003a4680) 0x88, CmdSN 0xabd from world 0 to dev "naa.600144f0f9e18d0000005160917e0001" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
2013-06-26T13:07:57.578Z cpu8:8603)ScsiCore: 1483: Power-on Reset occurred on naa.600144f0f9e18d0000005160917e0001
2013-06-26T13:08:07.901Z cpu8:8776)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba38:CH:0 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2013-06-26T13:08:07.901Z cpu8:8776)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000002 TARGET: iqn.2012-11.com.rhineaux:nex70 TPGT: 2 TSIH: 0]
2013-06-26T13:08:07.901Z cpu8:8776)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.70.50:49286 R: 192.168.70.10:3260]
2013-06-26T13:08:07.901Z cpu10:8778)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.600144f0f9e18d0000005160917e0001" state in doubt; requested fast path state update...
2013-06-26T13:08:07.901Z cpu10:8778)ScsiDeviceIO: 2303: Cmd(0x4124003a4680) 0x88, CmdSN 0xabe from world 0 to dev "naa.600144f0f9e18d0000005160917e0001" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0.
2013-06-26T13:08:07.988Z cpu10:13652)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:08:08.130Z cpu10:13652)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:08:08.273Z cpu10:13652)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:08:08.415Z cpu10:8202)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:08:08.558Z cpu10:8805)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:08:08.558Z cpu10:8805)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.600144f0f9e18d0000005160917e0001" state in doubt; requested fast path state update...
2013-06-26T13:08:08.696Z cpu10:8202)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:08:08.837Z cpu10:8202)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:08:08.980Z cpu10:8202)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:08:09.123Z cpu10:8202)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:08:09.265Z cpu10:8202)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:08:09.409Z cpu10:8202)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:08:09.551Z cpu10:8671)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:08:09.565Z cpu10:8202)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.600144f0f9e18d0000005160917e0001" state in doubt; requested fast path state update...
2013-06-26T13:08:09.694Z cpu10:8202)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:08:09.838Z cpu10:8202)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:08:09.980Z cpu10:8202)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:08:10.123Z cpu10:8202)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:08:10.265Z cpu10:8778)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:08:10.409Z cpu10:8778)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:08:10.551Z cpu10:8202)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:08:10.551Z cpu10:8202)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.600144f0f9e18d0000005160917e0001" state in doubt; requested fast path state update...
2013-06-26T13:08:10.673Z cpu8:8776)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x410022110b10 network resource pool netsched.pools.persist.iscsi associated
2013-06-26T13:08:10.673Z cpu8:8776)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x410022110b10 network tracker id 1 tracker.iSCSI.192.168.70.10 associated
2013-06-26T13:08:10.694Z cpu10:8778)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:08:10.838Z cpu10:8202)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:08:10.979Z cpu10:8202)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:08:11.123Z cpu10:8202)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL
2013-06-26T13:08:11.178Z cpu8:8776)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba38:CH:0 T:0 CN:0: iSCSI connection is being marked "ONLINE"
2013-06-26T13:08:11.178Z cpu8:8776)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000002 TARGET: iqn.2012-11.com.rhineaux:nex70 TPGT: 2 TSIH: 0]
2013-06-26T13:08:11.178Z cpu8:8776)WARNING: iscsi_vmk: iscsivmk_StartConnection: Conn [CID: 0 L: 192.168.70.50:59905 R: 192.168.70.10:3260]
2013-06-26T13:08:11.191Z cpu12:8204)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x88 (0x4124003a4680, 0) to dev "naa.600144f0f9e18d0000005160917e0001" on path "vmhba38:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0. Act:NONE
2013-06-26T13:08:11.191Z cpu12:8204)ScsiDeviceIO: 2303: Cmd(0x4124003a4680) 0x88, CmdSN 0xabe from world 0 to dev "naa.600144f0f9e18d0000005160917e0001" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
2013-06-26T13:08:11.191Z cpu12:8204)ScsiCore: 1483: Power-on Reset occurred on naa.600144f0f9e18d0000005160917e0001
/vmfs/volumes #