Back to All Scenarios
PASSEDvendor / vmfs_datastore_inaccessible
VMFS Datastore Inaccessible — All VMs on Store Frozen
A VMFS 6 datastore becomes inaccessible due to a storage path failure (all-paths-down APD condition). All 22 VMs on the datastore freeze with I/O errors. vSphere triggers APD timeout handling after 140 seconds.
Pattern
VMWARE_EVENT
Severity
CRITICAL
Confidence
90%
Remediation
Remote Hands
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | VMWARE_EVENT | VMWARE_EVENT | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 50 linked | |
| Cascade Escalation | Yes | Yes | |
| Remediation | — | Remote Hands — Corax contacts on-site support via call, email, or API |
Scenario Conditions
VMware VMFS 6 datastore on FC SAN. 2 HBA paths (both failed). 22 VMs on affected datastore. APD handling set to 'Power off and restart VMs'. Timeout: 140 seconds.
Injected Error Messages (4)
VMware VMFS datastore DS-SAN-01 inaccessible — All-Paths-Down (APD) condition on esxi-02, NMP: nmp_ThrottleLogForDevice: Throttle VMW_SATP_DEFAULT_AA: 0 paths, SCSI sense code H:0x7, datastore heartbeat lost
Brocade FC switch port offline — port 12 (connected to ESXi HBA) showing link failure, CRC errors on fiber path, zone configuration intact but physical link down on both paths
FileServer-01 VM frozen — disk I/O error after VMFS datastore APD, guest OS hanging on I/O wait, SMB shares inaccessible, all file operations returning input/output error
Web server VM unresponsive after datastore failure — VMware APD timeout reached (140s), vSphere initiating emergency VM power-off and restart on alternate datastore
Neural Engine Root Cause Analysis
ESXi host esxi-02 has lost all storage paths to SAN datastore DS-SAN-01, creating an All-Paths-Down (APD) condition. This indicates a complete failure of connectivity between the ESXi host and the shared storage, likely due to SAN fabric issues, storage controller failure, or HBA/network adapter problems on the host. The presence of 16 correlated incidents suggests this is affecting multiple components and likely represents a broader infrastructure failure rather than an isolated host issue.
Remediation Plan
1. Immediately check SAN fabric health and connectivity paths between esxi-02 and DS-SAN-01. 2. Verify storage array controller status and multipathing configuration. 3. Examine HBA status and network adapters on esxi-02. 4. Review storage switch logs for port failures or fabric issues. 5. If storage paths can be restored, rescan HBAs and datastores. 6. Consider migrating VMs to other hosts if storage cannot be quickly restored. 7. Investigate root cause of the 16 correlated incidents to address the broader infrastructure issue.
Tested: 2026-03-30Monitors: 4 | Incidents: 4Test ID: cmncjhc5z01i2obqefnym5kko