Back to All Scenarios
PASSEDinfrastructure / rto_violation
RTO Violation — Recovery Taking Too Long
During an actual disaster recovery activation, the recovery process is taking far longer than the 4-hour RTO SLA. After 6 hours, only 3 of 12 critical services are operational. The recovery runbook is outdated, automation scripts are failing, and key personnel are unreachable.
Pattern
UNKNOWN
Severity
CRITICAL
Confidence
85%
Remediation
Remote Hands
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | UNKNOWN | UNKNOWN | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 20 linked | |
| Cascade Escalation | N/A | No | |
| Remediation | — | Remote Hands — Corax contacts on-site support via call, email, or API |
Scenario Conditions
Production data center power failure. DR activation initiated. RTO SLA: 4 hours. After 6 hours: 3/12 services up. Runbook last updated 8 months ago. 2 of 5 key personnel unreachable. Automation scripts reference deprecated APIs.
Injected Error Messages (2)
DR recovery orchestration failing — Zerto VPG recovery initiated 6 hours ago, 3 of 12 virtual protection groups recovered successfully, 5 VPGs stuck in 'recovery in progress' state, 4 VPGs failed with errors, failure reasons: storage mapping incorrect for 2 VPGs, network mapping references non-existent port group for 2 VPGs, RTO SLA breached by 2 hours and counting, estimated time to full recovery: 4+ additional hours
service recovery status CRITICAL — RTO SLA of 4 hours breached, current recovery duration: 6 hours 15 minutes, services operational: email (partial), DNS, monitoring (3 of 12), services still down: ERP, CRM, file shares, telephony, remote access, web applications, databases, payroll, billing, recovery runbook references infrastructure components that no longer exist, automation scripts failing on API version mismatches, incident commander escalating to executive management
Neural Engine Root Cause Analysis
The Zerto DR orchestrator has encountered multiple configuration failures during a disaster recovery operation, indicating stale or incorrect DR mappings rather than a simple service failure. Storage mappings are pointing to unavailable storage resources and network mappings reference non-existent port groups, suggesting the DR configuration was not properly maintained or validated against current infrastructure. The 11 correlated incidents in the same timeframe strongly suggest this is part of a broader infrastructure event that triggered the DR failover, but outdated configuration mappings are preventing successful recovery.
Remediation Plan
1. Immediately access Zerto Virtual Manager console to assess current VPG states and error details. 2. Validate and update storage mappings for the 2 failed VPGs to point to available datastores. 3. Correct network mappings to reference existing port groups in the recovery site. 4. For the 5 VPGs stuck in 'recovery in progress', check for resource contention and restart recovery if safe to do so. 5. Prioritize VPGs based on business criticality to meet RTO requirements. 6. Once immediate recovery is stabilized, conduct full DR configuration audit and implement automated validation testing.
Tested: 2026-03-30Monitors: 2 | Incidents: 2Test ID: cmnckfece093kobqekkl1s46g