Back to All Scenarios
PASSEDserver / backup_cloud_timeout
Backup to Cloud Timeout — Offsite Backup Stalled
The offsite backup copy job to Wasabi S3 stalls at 23% due to ISP bandwidth throttling during business hours. The 8TB backup set cannot complete within the backup window. Offsite RPO violated for disaster recovery compliance.
Pattern
BACKUP_FAILURE
Severity
CRITICAL
Confidence
85%
Remediation
Remote Hands
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | BACKUP_FAILURE | BACKUP_FAILURE | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 18 linked | |
| Cascade Escalation | N/A | No | |
| Remediation | — | Remote Hands — Corax contacts on-site support via call, email, or API |
Scenario Conditions
Veeam backup copy job to Wasabi S3. 8TB backup set. 100Mbps upload link (throttled to 20Mbps during business hours by ISP). Job running for 14 hours at 23%. Backup window: 12 hours. DR compliance requires daily offsite copy.
Injected Error Messages (2)
Veeam backup copy job 'Offsite-DR-Copy' stalled — S3 upload to Wasabi bucket 'acme-dr-backups' at 23% after 14 hours, transfer rate: 2.8 MB/s (expected 12.5 MB/s), socket timeout errors, backup window exceeded by 2 hours, offsite RPO violated
WAN uplink bandwidth throttled — ISP circuit showing 20Mbps throughput (contracted 100Mbps), QoS shaping detected during business hours, Veeam cloud backup unable to achieve required transfer rate, SNMP ifOutOctets confirming throttle
Neural Engine Root Cause Analysis
The Veeam backup copy job 'Offsite-DR-Copy' is experiencing a severe network performance degradation when uploading to the Wasabi S3 bucket, with transfer rates dropping to 2.8 MB/s versus the expected 12.5 MB/s (78% reduction). The combination of socket timeout errors, 10 correlated incidents in the same time window, and the job stalling at 23% completion after 14 hours strongly indicates either network congestion, ISP throttling, or degraded connectivity between the Veeam server and Wasabi cloud storage. This has resulted in backup window violations and offsite RPO breach, creating a critical business continuity risk.
Remediation Plan
1. Immediately check network connectivity and bandwidth between Veeam server (10.10.100.70) and Wasabi endpoints using network diagnostics 2. Restart the stalled backup copy job to clear any hung connections 3. Verify Wasabi service status and check for any provider-side issues 4. Review network utilization and identify potential bandwidth contention from the 10 correlated incidents 5. Consider temporarily reducing concurrent backup streams or implementing QoS policies 6. If network issues persist, contact ISP to investigate potential throttling or routing issues 7. Monitor subsequent backup jobs for performance recovery
Tested: 2026-03-30Monitors: 2 | Incidents: 2Test ID: cmncjjprm020hobqe8xdhc8vy