Back to All Scenarios
PASSEDinfrastructure / api_rate_limit_partner_blocked
API Rate Limit Exceeded — Partner Integration Blocked
A partner integration is being blocked by API rate limiting after a bug in the partner's code causes it to retry failed requests in a tight loop. The rate limiter is correctly blocking the abusive traffic but is also blocking legitimate requests from the same partner, disrupting a critical business integration.
Pattern
FIREWALL_RULE_BLOCK
Severity
CRITICAL
Confidence
85%
Remediation
Remote Hands
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | FIREWALL_RULE_BLOCK | FIREWALL_RULE_BLOCK | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 18 linked | |
| Cascade Escalation | N/A | No | |
| Remediation | — | Remote Hands — Corax contacts on-site support via call, email, or API |
Scenario Conditions
API rate limit: 1000 requests/minute per API key. Partner sending 50000 requests/minute due to retry bug. Rate limiter blocking 98% of partner traffic. Legitimate requests also blocked. Partner handles 30% of total order volume.
Injected Error Messages (2)
API rate limiter CRITICAL — partner API key 'pk_partner_acme_prod' exceeding rate limit by 50x, request rate: 50000/minute (limit: 1000/minute), 98% of requests rejected with HTTP 429, rate limiter correctly blocking abusive traffic pattern but all requests from this API key affected including legitimate ones, partner 'ACME Corp' handles 30% of total order volume, blocked by policy on all inbound from partner, firewall rule added to drop traffic from partner IP range as additional protection
partner integration disrupted — ACME Corp integration completely blocked by firewall rule, 30% of daily order volume not processing, partner reporting all API calls returning rejections, partner's client library has retry-on-error bug causing exponential request multiplication, legitimate order submissions mixed with retry storm, unable to differentiate good requests from bad, business impact: $45K/hour in unprocessed orders, deny rule applied broadly to partner IP range
Neural Engine Root Cause Analysis
The API Gateway is experiencing a cascading failure triggered by ACME Corp's API key exceeding rate limits by 50x (50,000/minute vs 1,000/minute limit). While the rate limiter is correctly rejecting 98% of requests with HTTP 429, a firewall rule was subsequently added to drop all traffic from the partner's IP range, effectively blocking all legitimate traffic from a partner that handles 30% of total order volume. The 10 correlated incidents and FIREWALL_RULE_BLOCK pattern suggest this defensive measure has caused broader service disruption beyond just rate limiting.
Remediation Plan
1. Immediately assess if ACME Corp traffic spike is legitimate or malicious by contacting partner. 2. If legitimate: temporarily increase rate limit for pk_partner_acme_prod to accommodate higher volume and remove firewall block on partner IP range. 3. If malicious: keep firewall block but create exception rules for known legitimate ACME Corp endpoints/IPs. 4. Monitor rate limiter metrics and gateway health. 5. Implement graduated rate limiting strategy to avoid future binary blocking scenarios. 6. Establish emergency contact protocol with high-volume partners for rapid incident response.
Tested: 2026-03-30Monitors: 2 | Incidents: 2Test ID: cmnckhjj009kmobqe1nao4iws