Back to All Scenarios
PASSEDvendor / datto_bcdr_appliance_failure
Datto BCDR Backup Appliance Failure
A Datto SIRIS 5 BCDR appliance experiences a storage controller failure, halting all local backups and breaking the chain for 15 protected servers. Cloud offsite replication also stops.
Pattern
BACKUP_FAILURE
Severity
CRITICAL
Confidence
95%
Remediation
Remote Hands
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | BACKUP_FAILURE | BACKUP_FAILURE | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 22 linked | |
| Cascade Escalation | N/A | No | |
| Remediation | — | Remote Hands — Corax contacts on-site support via call, email, or API |
Scenario Conditions
Datto SIRIS 5 protecting 15 servers. Storage controller failure on Datto appliance. Local snapshots failing. Cloud offsite sync interrupted. 15 servers unprotected. Datto portal showing appliance offline.
Injected Error Messages (2)
datto SIRIS 5 BCDR appliance failure — local storage controller reporting critical hardware error, datto appliance unable to create new snapshots, all 15 protected server backup jobs failing with 'storage write error', datto ZFS pool status: DEGRADED, local retention: intact but no new backups possible, datto appliance health: critical
datto cloud portal reporting SIRIS 5 appliance offline — last successful backup: 14 hours ago, datto offsite cloud replication halted, 15 server agents reporting backup failures to datto portal, datto BCDR chain integrity: broken for incremental backups, datto support case auto-generated for hardware replacement, RPO violation: 14 hours and growing for all protected servers
Neural Engine Root Cause Analysis
The Datto SIRIS 5 BCDR appliance is experiencing a critical hardware failure in its local storage controller, causing the ZFS pool to enter a DEGRADED state. This hardware-level failure is preventing the appliance from performing write operations, resulting in all 15 protected server backup jobs failing with 'storage write error' and the inability to create new snapshots. While existing backups remain intact, no new backup operations can complete until the storage controller hardware issue is resolved.
Remediation Plan
1. Immediately contact Datto support for hardware replacement - storage controller failure requires physical component replacement. 2. Check ZFS pool status via 'zpool status' to identify specific failed drives/controllers. 3. If RAID configuration allows, attempt to identify and replace failed drive(s) following Datto's hardware maintenance procedures. 4. Once hardware is repaired, verify ZFS pool returns to ONLINE status. 5. Test backup job functionality by running a manual backup of one protected server. 6. Resume all backup schedules once storage subsystem is fully operational. 7. Review backup retention policies and consider temporary offsite backup strategy during repair window.
Tested: 2026-03-30Monitors: 2 | Incidents: 2Test ID: cmnck9j6u07rfobqet06xj9fq