Back to All Scenarios
PASSEDcloud / k8s_ingress_controller_crash
Kubernetes Ingress Controller Crash — External Traffic Blocked
The NGINX Ingress Controller in a production Kubernetes cluster crashes after a malformed Ingress resource is applied. The controller enters a CrashLoopBackOff state. All external HTTP/HTTPS traffic to the cluster is blocked because no ingress controller pods are running to route traffic to backend services.
Pattern
CONTAINER_EVENT
Severity
CRITICAL
Confidence
95%
Remediation
Auto-Heal
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | CONTAINER_EVENT | CONTAINER_EVENT | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 28 linked | |
| Cascade Escalation | Yes | Yes | |
| Remediation | — | Auto-Heal — Corax resolves autonomously |
Scenario Conditions
Kubernetes 1.29. NGINX Ingress Controller v1.9. Single ingress controller deployment (no HA). 45 Ingress resources routing traffic to 30 services. Malformed annotation causes NGINX config validation failure. Controller pod in CrashLoopBackOff.
Injected Error Messages (3)
Kubernetes NGINX Ingress Controller CrashLoopBackOff — pod 'ingress-nginx-controller-7f9b4c8d-x2j4' container runtime error: nginx configuration validation failed, invalid annotation 'nginx.ingress.kubernetes.io/server-snippet' on Ingress 'api-v2-routes', container restart count: 23, pod evicted and rescheduled 4 times, all 45 Ingress resources unserved
Production website completely unreachable — NGINX ingress controller down, external LoadBalancer health checks failing, all HTTP/HTTPS traffic receiving connection refused, website returning ERR_CONNECTION_REFUSED in browsers, Docker container for ingress not running, customer-facing outage affecting all web properties
API endpoint unreachable — ingress controller pod not running, API gateway returning connection refused on ports 80 and 443, all external API consumers receiving connection errors, webhook deliveries failing, mobile app showing 'cannot connect to server', container runtime unable to start NGINX process due to config error
Neural Engine Root Cause Analysis
The NGINX Ingress Controller is in a CrashLoopBackOff state due to a configuration validation failure caused by an invalid 'nginx.ingress.kubernetes.io/server-snippet' annotation on the 'api-v2-routes' Ingress resource. This invalid annotation is preventing the NGINX configuration from loading properly, causing the controller to crash repeatedly (23 restarts, 4 pod evictions). With all 45 Ingress resources unserved, this represents a complete ingress layer failure affecting the entire application routing infrastructure.
Remediation Plan
1. Identify and examine the 'api-v2-routes' Ingress resource to locate the problematic 'nginx.ingress.kubernetes.io/server-snippet' annotation. 2. Validate the annotation syntax and content against NGINX configuration standards. 3. Either fix the annotation syntax or remove it temporarily if invalid. 4. Apply the corrected Ingress resource configuration. 5. Monitor the ingress controller pod for successful restart and stable running state. 6. Verify that all 45 Ingress resources are being served properly. 7. If the issue persists, restart the ingress controller deployment to ensure clean state.
Tested: 2026-03-30Monitors: 3 | Incidents: 3Test ID: cmncjvb6c0506obqesc4tyaq4