Back to All Scenarios
PASSEDvendor / dell_vxrail_hci_node_failure
Dell VxRail HCI Node Failure
A Dell VxRail E660F node experiences a motherboard failure, removing compute and storage capacity from the vSAN cluster. The cluster enters a reduced state with lower fault tolerance.
Pattern
VMWARE_EVENT
Severity
CRITICAL
Confidence
92%
Remediation
Remote Hands
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | VMWARE_EVENT | VMWARE_EVENT | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 27 linked | |
| Cascade Escalation | Yes | Yes | |
| Remediation | — | Remote Hands — Corax contacts on-site support via call, email, or API |
Scenario Conditions
Dell VxRail cluster with 4 E660F nodes. Node 3 motherboard failure. vSAN FTT=1 policy. Cluster loses 25% compute and storage. vSAN rebuild started on remaining 3 nodes. Resource contention increasing.
Injected Error Messages (3)
dell VxRail E660F node 3 failure detected — VxRail Manager reporting node esxi-vxrail-03 as unreachable, dell VxRail cluster health: critical, node removed from vSAN and compute resource pool, cluster operating at 75% capacity (3 of 4 nodes), dell VxRail automatic remediation initiated: vSAN data rebuild in progress on remaining nodes
dell VxRail E660F node 3 iDRAC reporting hardware failure — motherboard component failure detected, dell iDRAC system event log: 'critical — system board failure', POST not completing, node powered off by BMC safety protocol, dell VxRail node warranty status: active, hardware replacement required
vSAN cluster in reduced redundancy after dell VxRail node loss — vSAN health alarm: 'reduced redundancy', FTT=1 objects now at FTT=0 on rebuilt components, vSAN rebuild consuming 40% of remaining disk I/O bandwidth, VM performance impacted during resync, dell VxRail cluster cannot tolerate another node failure until rebuild completes (ETA: 8 hours)
Neural Engine Root Cause Analysis
Dell VxRail node esxi-vxrail-03 has experienced a hardware or ESXi host failure, causing it to become unreachable and be automatically removed from the vSAN cluster and compute resource pool. The VxRail Manager at 10.50.3.100 is likely responding with errors or timeouts because it's reporting the critical cluster state and coordinating automatic remediation processes. The 12 correlated incidents suggest a cascade effect as workloads redistribute across the remaining 3 nodes, potentially causing performance degradation and additional monitoring alerts.
Remediation Plan
1. Verify physical status of esxi-vxrail-03 node (power, network, hardware LEDs). 2. Check ESXi host accessibility via SSH/console and review vmkernel logs for hardware errors. 3. Monitor vSAN data rebuild progress and ensure remaining nodes have sufficient storage capacity. 4. If hardware failure confirmed, engage Dell support for node replacement following VxRail procedures. 5. Validate cluster performance and resource utilization on remaining nodes. 6. Once node is restored, re-add to cluster and rebalance workloads.
Tested: 2026-03-30Monitors: 3 | Incidents: 3Test ID: cmnck6hy30746obqecu1ku7zn