Back to All Scenarios
PASSEDinfrastructure / siem_correlation_engine_overloaded
SIEM Correlation Engine Overloaded — Security Events Unprocessed
The SIEM correlation engine is overwhelmed by a 10x log volume increase caused by a misconfigured firewall debug logging. The correlation engine cannot keep up, creating a 3-hour processing backlog. Real-time security alerting is non-functional during this period.
Pattern
HIGH_CPU
Severity
CRITICAL
Confidence
92%
Remediation
Remote Hands
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | HIGH_CPU | HIGH_CPU | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 18 linked | |
| Cascade Escalation | N/A | No | |
| Remediation | — | Remote Hands — Corax contacts on-site support via call, email, or API |
Scenario Conditions
SIEM processing 50000 EPS normally. Firewall debug logging enabled accidentally, producing 500000 EPS. Correlation engine capacity: 75000 EPS. Processing backlog: 3 hours. Real-time security alerting delayed by 3+ hours.
Injected Error Messages (2)
SIEM correlation engine CRITICAL — event processing rate dropped to 75000 EPS while ingestion rate is 500000 EPS, processing backlog: 3 hours 22 minutes and growing, correlation searches exceeding time limits, real-time security alerts delayed by minimum 3 hours, scheduled searches skipping execution windows, notable events queue depth: 2.4 million unprocessed, correlation engine cpu at 100% across all 8 search heads
SIEM indexer cluster under extreme load — disk full warning on 4 of 6 indexers, indexing rate: 500000 EPS (10x normal baseline of 50000 EPS), root cause: firewall 'fw-core-01' debug logging accidentally enabled generating 450000 additional EPS of firewall session logs, SIEM license usage: 340% of daily limit, indexer cluster bucket replication falling behind, no space left on hot bucket volumes, security monitoring effectiveness: severely compromised
Neural Engine Root Cause Analysis
The Splunk SIEM correlation engine is experiencing severe resource exhaustion with CPU at 100% across all 8 search heads, causing event processing to fall drastically behind ingestion rates (75K vs 500K EPS). This has created a cascading failure where correlation searches exceed time limits, scheduled searches skip execution windows, and a massive backlog of 2.4 million unprocessed events has accumulated. The 10 correlated incidents suggest this is impacting multiple dependent systems and creating a significant blast radius across the security monitoring infrastructure.
Remediation Plan
1. Immediately scale out processing capacity by adding more search heads or increasing resource allocation to existing ones. 2. Temporarily disable non-critical scheduled searches to reduce CPU load and allow backlog processing. 3. Implement search optimization by reviewing and tuning correlation searches that are exceeding time limits. 4. Enable search prioritization to ensure critical security alerts are processed first. 5. Monitor the notable events queue reduction and gradually re-enable disabled searches once backlog is cleared. 6. Implement longer-term capacity planning and alerting thresholds to prevent recurrence.
Tested: 2026-03-30Monitors: 2 | Incidents: 2Test ID: cmnckgq3409doobqeuw6ob4d3