Back to All Scenarios
PASSEDinfrastructure / websocket_connection_storm
WebSocket Connection Storm — Server Connection Limit Reached
A mobile app update with a WebSocket reconnection bug is causing millions of simultaneous WebSocket connection attempts. Each failed connection immediately retries, creating a connection storm that exhausts the server's file descriptor limit and blocks all new connections including the web application.
Pattern
CONNECTION_REFUSED
Severity
CRITICAL
Confidence
95%
Remediation
Remote Hands
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | CONNECTION_REFUSED | CONNECTION_REFUSED | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 18 linked | |
| Cascade Escalation | N/A | No | |
| Remediation | — | Remote Hands — Corax contacts on-site support via call, email, or API |
Scenario Conditions
WebSocket server with 65535 fd limit. Mobile app v3.2.0 has reconnection bug. 2 million mobile clients attempting simultaneous reconnection. File descriptors exhausted. Web application also blocked. Connection queue full.
Injected Error Messages (2)
WebSocket server file descriptors exhausted — current connections: 65535 (max fd limit reached), connection accept rate: 0 (cannot open new file descriptors), 2 million WebSocket connection attempts queued at network level, mobile app v3.2.0 reconnection bug causing immediate retry on disconnect creating exponential connection growth, econnrefused returned to all new connection attempts, existing WebSocket connections stable but no new connections possible
web application unreachable — app.company.com returning econnrefused to all HTTP requests, shared infrastructure with WebSocket server, backend connection pool exhausted, all available file descriptors consumed by WebSocket connection storm from mobile clients, web application health checks failing, all user traffic (web and mobile) blocked, econnrefused on port 443 and 8080
Neural Engine Root Cause Analysis
The WebSocket server has reached its file descriptor limit of 65535, preventing it from accepting new connections. This is caused by a mobile app v3.2.0 bug that creates exponential connection growth through immediate reconnection attempts on disconnect, leading to 2 million queued connection attempts. The server needs immediate file descriptor limit increase and connection cleanup to restore service.
Remediation Plan
1. Immediately increase system file descriptor limits (ulimit -n) for the WebSocket service. 2. Restart the WebSocket service to clear existing connections and reset the file descriptor count. 3. Implement connection rate limiting and maximum connection caps to prevent future exhaustion. 4. Deploy a hotfix for mobile app v3.2.0 to fix the reconnection bug. 5. Monitor connection patterns and establish alerting before reaching 80% of fd limits.
Tested: 2026-03-30Monitors: 2 | Incidents: 2Test ID: cmnckhk9509knobqes44855s9