Back to All Scenarios
PASSEDnetwork / syslog_server_overwhelmed
Syslog Server Overwhelmed — Log Ingestion Failure
The centralized syslog server is overwhelmed by a log storm from a network event, dropping 80% of incoming messages. Critical security and compliance logs are being lost during an active incident.
Pattern
UNKNOWN
Severity
CRITICAL
Confidence
95%
Remediation
Auto-Heal
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | UNKNOWN | UNKNOWN | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 21 linked | |
| Cascade Escalation | N/A | No | |
| Remediation | — | Auto-Heal — Corax resolves autonomously |
Scenario Conditions
Rsyslog server receiving from 200 devices. Normal rate: 5000 msg/sec. Current rate: 150000 msg/sec from network event generating massive switch logs. Rsyslog dropping 80% of messages. Log-based alerting: non-functional.
Injected Error Messages (2)
syslog server overwhelmed — rsyslog process at maximum capacity, UDP receive buffer overflows: 120000 per minute, syslog ingestion rate: 150000 msg/sec (capacity: 30000 msg/sec), rsyslog dropping 80% of incoming messages, syslog queue depth: 2 million messages backed up, 200 network devices sending logs but server unable to process at current volume, log data loss: critical
SIEM log ingestion pipeline broken — syslog forwarder to SIEM dropping messages due to upstream rsyslog overflow, SIEM (Elasticsearch) receiving only 20% of expected log volume, security correlation rules not triggering due to missing log data, compliance log retention: gap in coverage for 200 devices, SIEM dashboard showing incomplete data, security team operating blind during active network incident
Neural Engine Root Cause Analysis
The rsyslog service is experiencing severe resource exhaustion due to incoming log volume (150,000 msg/sec) exceeding its processing capacity (30,000 msg/sec) by 5x. This has caused UDP receive buffer overflows, a 2 million message queue backlog, and 80% message drop rate. The service is likely configured with insufficient buffer sizes, worker threads, or memory allocation to handle the current log ingestion rate from 200 network devices.
Remediation Plan
1. Immediately increase rsyslog UDP receive buffer sizes (net.core.rmem_max, net.core.rmem_default) 2. Scale up rsyslog worker threads and input threads in /etc/rsyslog.conf 3. Increase rsyslog queue sizes (main message queue, action queues) 4. Add memory allocation for high-volume processing 5. Consider implementing log rate limiting on network devices 6. Restart rsyslog service with new configuration 7. Monitor message processing rate and queue depths
Tested: 2026-03-30Monitors: 2 | Incidents: 2Test ID: cmnckbgl6087uobqeg3w52evt