Back to All Scenarios
PASSEDcloud / gcp_spanner_regional_outage
GCP Cloud Spanner Regional Outage
A Google Cloud Spanner regional instance experiences a zone-level failure in us-central1-a. The multi-zone configuration should provide automatic failover, but a configuration error in the instance's node count causes the remaining zones to be overloaded. Read and write latencies spike above SLA thresholds.
Pattern
STORAGE_IO_LATENCY
Severity
CRITICAL
Confidence
95%
Remediation
Remote Hands
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | STORAGE_IO_LATENCY | STORAGE_IO_LATENCY | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 42 linked | |
| Cascade Escalation | Yes | Yes | |
| Remediation | — | Remote Hands — Corax contacts on-site support via call, email, or API |
Scenario Conditions
GCP Cloud Spanner instance 'prod-spanner' in us-central1 (3 zones). Zone us-central1-a failure. Node count: 3 (1 per zone, minimum for multi-zone). Remaining 2 nodes overloaded at 95% CPU. Processing units insufficient for failover load. 8 services dependent on Spanner.
Injected Error Messages (3)
GCP Cloud Spanner degraded — instance 'prod-spanner' zone us-central1-a unavailable, remaining nodes in zones b and c at 95% processing unit utilization, read latency p99: 4,200ms (SLA: 200ms), write latency p99: 8,700ms (SLA: 500ms), GCP incident INC-2026-0329 acknowledged, automatic zone failover succeeded but remaining capacity insufficient for full workload
Transaction service experiencing high latency — Spanner gRPC calls returning DeadlineExceeded after 30 seconds, transaction commit latency: 8.7 seconds (baseline: 50ms), 34% of transactions aborted due to lock contention on overloaded nodes, payment processing queue depth: 15,000, customer checkout failures increasing at 200/minute
Analytics queries failing — Spanner read-only transactions returning UNAVAILABLE status, batch analytics pipeline halted after consecutive failures, dashboard data stale by 2 hours, BigQuery export job blocked on Spanner reads, SLA breach: availability dropped to 94.2% (target: 99.999%)
Neural Engine Root Cause Analysis
The prod-spanner Cloud Spanner instance has lost an entire availability zone (us-central1-a) due to GCP infrastructure issues (incident INC-2026-0329). While automatic zone failover succeeded, the remaining two zones (b and c) are running at 95% processing unit utilization, creating a capacity bottleneck that causes severe performance degradation with read/write latencies exceeding SLA by 20x and 17x respectively. The 16 correlated incidents suggest this is causing cascading failures across dependent services that rely on the database.
Remediation Plan
1. Immediately scale up Cloud Spanner processing units in remaining zones (us-central1-b and us-central1-c) to handle the redistributed load from the failed zone. 2. Monitor GCP status page and incident INC-2026-0329 for zone restoration timeline. 3. Implement application-level circuit breakers and retry policies with exponential backoff to reduce load pressure. 4. Consider temporarily scaling down non-critical read operations or implementing read replicas if available. 5. Once us-central1-a is restored, gradually rebalance the load across all three zones and scale processing units back to normal levels.
Tested: 2026-03-30Monitors: 3 | Incidents: 3Test ID: cmncjutjr04wbobqesa5pfv0t