Back to All Scenarios
PASSEDvendor / hpe_simplivity_node_rebuild
HPE SimpliVity HCI Node Rebuild Failure
An HPE SimpliVity 380 Gen10 node fails during a rebuild operation after a previous disk replacement, leaving the cluster in a reduced state with lower storage efficiency and no fault tolerance.
Pattern
VMWARE_EVENT
Severity
CRITICAL
Confidence
92%
Remediation
Remote Hands
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | VMWARE_EVENT | VMWARE_EVENT | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 21 linked | |
| Cascade Escalation | N/A | No | |
| Remediation | — | Remote Hands — Corax contacts on-site support via call, email, or API |
Scenario Conditions
HPE SimpliVity 380 cluster with 3 nodes. Node 2 disk replaced, rebuild started. Rebuild fails at 68% due to OmniStack controller crash. Cluster at 2 healthy nodes. Data resiliency: at risk.
Injected Error Messages (2)
hpe SimpliVity 380 node rebuild failure — OmniStack Virtual Controller on node 2 crashed at 68% rebuild completion, hpe SimpliVity cluster resiliency: at risk (2 of 3 nodes healthy), data resiliency policy requires 3 nodes for full redundancy, hpe SimpliVity federation reporting cluster alarm: 'resiliency reduced', operations suspended during recovery
hpe SimpliVity OmniStack controller on node 2 unresponsive — hpe OmniStack Virtual Controller process terminated during data rebuild operation, node 2 not participating in cluster data services, hpe SimpliVity accelerator card (OAC) reporting healthy but controller software failed, manual intervention required to restart OmniStack VC and resume rebuild from checkpoint
Neural Engine Root Cause Analysis
The HPE SimpliVity OmniStack Virtual Controller on node 2 crashed at 68% rebuild completion, indicating a critical hardware or software failure during the node reconstruction process. This has degraded the cluster from 3 healthy nodes to only 2, compromising data resiliency and triggering automatic suspension of operations to prevent data loss. The crash during rebuild suggests underlying infrastructure issues such as storage subsystem failure, memory exhaustion, or hypervisor instability that requires immediate attention.
Remediation Plan
1. Immediately assess cluster health status via SimpliVity vCenter plugin or CLI to confirm current node states. 2. Check hardware status of node 2 including storage controllers, memory modules, and network connectivity. 3. Review SimpliVity logs and VMware vCenter events for specific error codes leading to OVC crash. 4. If hardware is healthy, attempt OVC restart on node 2 following HPE procedures. 5. Once node 2 is stable, reinitiate controlled rebuild process with continuous monitoring. 6. Verify cluster resiliency is restored to 3/3 nodes before resuming normal operations. 7. Implement preventive measures based on crash analysis findings.
Tested: 2026-03-30Monitors: 2 | Incidents: 2Test ID: cmnck70pe0784obqe3ln9ijuv