Back to All Scenarios
PASSEDnetwork / syslog_udp_overflow
Syslog UDP Overflow — Log Data Loss
The centralized syslog server cannot keep up with the volume of incoming UDP syslog messages during a network event. UDP packets are dropped at the kernel level, causing critical security and audit log data to be permanently lost.
Pattern
UNKNOWN
Severity
CRITICAL
Confidence
85%
Remediation
Auto-Heal
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | UNKNOWN | UNKNOWN | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 6 linked | |
| Cascade Escalation | N/A | No | |
| Remediation | — | Auto-Heal — Corax resolves autonomously |
Scenario Conditions
Rsyslog server receiving 50,000 messages/second during network event. UDP receive buffer overflow. Kernel dropping 30% of incoming packets. Security audit logs missing entries. SIEM correlation rules failing due to missing data.
Injected Error Messages (1)
Syslog server UDP buffer overflow — kernel dropping 30% of incoming syslog packets, rsyslog processing at 35,000 msg/sec but receiving 50,000 msg/sec, UDP receive buffer full: 15,000 packets dropped in last minute, security audit log entries missing, SIEM correlation gaps detected, compliance-critical firewall logs being lost during active network incident
Neural Engine Root Cause Analysis
The syslog server (syslog01) is experiencing a UDP buffer overflow due to insufficient kernel buffer size and rsyslog processing capacity. The system is receiving 50,000 messages/second but can only process 35,000 msg/sec, causing a 15,000 packet backlog and 30% packet loss. This resource exhaustion is likely triggered by an active network incident generating excessive log volume, overwhelming the syslog infrastructure and creating compliance gaps in SIEM correlation.
Remediation Plan
1. Immediately increase UDP receive buffer size (net.core.rmem_max and net.core.rmem_default) 2. Scale rsyslog processing by increasing worker threads and queue sizes 3. Implement rate limiting on syslog clients to prevent overflow 4. Add additional syslog servers for load distribution 5. Verify log integrity and identify missing entries for compliance reporting 6. Monitor buffer utilization to prevent recurrence
Tested: 2026-03-30Monitors: 1 | Incidents: 1Test ID: cmnck275n068qobqe9x7l4sp3