Back to All Scenarios
PASSEDcloud / route53_health_check_cascade
AWS Route 53 Health Check Cascade — Multi-Region Failover Storm
A misconfigured Route 53 health check threshold causes all three regional endpoints to be marked unhealthy simultaneously during a brief network blip. Route 53 removes all records from DNS, causing a complete global outage even though all regions are actually healthy.
Pattern
AWS_CLOUD
Severity
CRITICAL
Confidence
85%
Remediation
Auto-Heal
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | AWS_CLOUD | AWS_CLOUD | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 28 linked | |
| Cascade Escalation | Yes | Yes | |
| Remediation | — | Auto-Heal — Corax resolves autonomously |
Scenario Conditions
Route 53 latency-based routing with 3 regions (us-east-1, eu-west-1, ap-southeast-1). Health check: 1 failure threshold (too aggressive). 30-second network blip triggers all health checks to fail simultaneously. All DNS records removed.
Injected Error Messages (3)
aws route 53 health check failed for us-east-1 endpoint — health check ID: hc-abc123 marked UNHEALTHY after 1 consecutive failure, endpoint actually responding normally after 30-second network blip, but aggressive threshold triggered premature failover, DNS record removed from route 53 response
aws route 53 health check cascade — eu-west-1 also marked UNHEALTHY during same 30-second window, all 3 regional endpoints simultaneously failed health checks, route 53 returning NXDOMAIN for app.company.com, global outage despite all backends being healthy, health check threshold of 1 failure is too aggressive
aws route 53 complete DNS blackout — all 3 regions marked unhealthy, no healthy records available for latency-based routing, all client DNS queries returning empty response, global service outage caused by health check misconfiguration rather than actual backend failure
Neural Engine Root Cause Analysis
This is a false positive triggered by AWS Route 53's overly aggressive health check threshold configuration. The endpoint https://app-us.company.com/health experienced a brief 30-second network blip but is responding normally, yet Route 53 marked it UNHEALTHY after just 1 consecutive failure and removed the DNS record. The 12 correlated incidents suggest this premature failover may have caused a cascade effect, potentially routing traffic away from a healthy endpoint or triggering additional monitoring alerts across dependent services.
Remediation Plan
1. Immediately verify endpoint health by directly testing https://app-us.company.com/health 2. If endpoint is healthy, force Route 53 health check re-evaluation by updating the health check configuration temporarily 3. Restore DNS record to Route 53 response if it was incorrectly removed 4. Adjust Route 53 health check threshold from 1 to 3 consecutive failures to prevent future false positives 5. Monitor for 10-15 minutes to ensure stable resolution and that correlated incidents clear
Tested: 2026-03-30Monitors: 3 | Incidents: 3Test ID: cmnckccjh08foobqegl89xxbr