Back to All Scenarios
PASSEDcascade / power_failure_cascade
Partial Power Failure — UPS Battery Exhaustion in Server Room
A utility power outage exceeds UPS battery runtime. The UPS runs out of battery and shuts down. Half the rack loses power. Servers, switches, and storage go offline simultaneously. Generator fails to start due to dead battery.
Pattern
CONNECTION_REFUSED
Severity
CRITICAL
Confidence
95%
Remediation
Remote Hands
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | CONNECTION_REFUSED | CONNECTION_REFUSED | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 75 linked | |
| Cascade Escalation | Yes | Yes | |
| Remediation | — | Remote Hands — Corax contacts on-site support via call, email, or API |
Scenario Conditions
Single UPS feeding Rack 1-3. 30-minute battery. Power outage at 2AM. Generator start failure. Rack 1: core switch + firewall. Rack 2: production servers. Rack 3: SAN storage.
Injected Error Messages (5)
Core switch unreachable — connection refused, device powered off after UPS battery exhaustion, all downstream networks dark
FortiGate firewall offline — connection refused, Fortinet device lost power, all VPN tunnels down, internet connectivity lost for campus
Application server down — connection refused, server powered off after UPS shutdown, crash loop expected on power restoration
Database server offline — connection refused, PostgreSQL unclean shutdown after power loss, potential data corruption on recovery
SAN storage controller offline — connection refused, storage array powered off, disk failed warning expected on restart, RAID degradation risk
Neural Engine Root Cause Analysis
The core network switch (Core-SW1) has lost power due to UPS battery exhaustion, causing a complete network outage. This is a hardware infrastructure failure where the uninterruptible power supply could not maintain power long enough during a power outage, resulting in the switch powering off and becoming unreachable. The 19 correlated incidents indicate this is causing a cascade failure affecting all downstream network segments and services dependent on this core switch.
Remediation Plan
1. Immediately dispatch on-site technician to data center Rack 1. 2. Verify and restore primary power source to the rack if needed. 3. Replace or recharge UPS battery system. 4. Power on Core-SW1 switch and verify POST completion. 5. Test network connectivity and routing tables. 6. Monitor for successful restoration of downstream services. 7. Investigate root cause of power outage and UPS failure for prevention.
Tested: 2026-03-30Monitors: 5 | Incidents: 5Test ID: cmncjfbaw00ylobqekqussrzr