Back to All Scenarios
PASSEDserver / veeam_backup_repo_full
Veeam Backup Job Failure — Repository Full
The Veeam backup repository runs out of disk space during the nightly backup window. All backup jobs fail with 'insufficient disk space' errors. No backups have completed for 3 nights. RPO violated for all protected VMs.
Pattern
BACKUP_FAILURE
Severity
CRITICAL
Confidence
95%
Remediation
Remote Hands
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | BACKUP_FAILURE | BACKUP_FAILURE | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 16 linked | |
| Cascade Escalation | N/A | No | |
| Remediation | — | Remote Hands — Corax contacts on-site support via call, email, or API |
Scenario Conditions
Veeam Backup & Replication 12. Repository on RAID 6 (12TB usable). 95 VMs backed up nightly. Repository usage: 100%. Retention: 14 restore points. No capacity warnings configured.
Injected Error Messages (2)
Veeam backup job 'Nightly-VM-Backup' failed — Error: insufficient disk space on backup repository 'BackupRepo-01', free space: 0 bytes (12TB/12TB used), 95 VMs unprotected for 3 consecutive nights, RPO violation: 72 hours
Backup repository NAS-01 disk full — volume /backups at 100% capacity, Veeam synthetic full merge failed, GFS retention cleanup unable to free space, oldest restore point locked by tape job, no free extents available
Neural Engine Root Cause Analysis
The Veeam backup server is experiencing a complete storage capacity exhaustion on backup repository 'BackupRepo-01' with 12TB/12TB used (0 bytes free). This has caused the nightly backup job to fail for 3 consecutive nights, leaving 95 VMs unprotected and creating a 72-hour RPO violation. The 8 correlated incidents within the same timeframe suggest this storage exhaustion is causing cascading failures across dependent backup operations or related infrastructure components.
Remediation Plan
1. Immediately check disk usage on backup repository 'BackupRepo-01' and identify largest backup files. 2. Delete or archive old backup chains that exceed retention policy to free immediate space. 3. Verify backup retention settings are configured correctly to prevent future accumulation. 4. Check for failed backup deletions or stuck processes that may be preventing cleanup. 5. If space cannot be freed, temporarily redirect backups to alternate repository or add storage capacity. 6. Re-run failed backup jobs once sufficient space is available. 7. Monitor storage usage and implement proactive alerting at 80% capacity threshold.
Tested: 2026-03-30Monitors: 2 | Incidents: 2Test ID: cmncjjayl01xfobqe7cxraoi4