Back to All Scenarios
PASSEDcache / log_rotation_failure
Memcached 1.6 Log File Growing Unbounded — Rotation Failed on Amazon Linux 2023
Memcached 1.6 log file has grown to 48GB because log rotation is misconfigured. Disk pressure increasing.
Pattern
DISK_FULL
Severity
CRITICAL
Confidence
72%
Remediation
Auto-Heal
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | DISK_FULL | DISK_FULL | |
| Severity Assessment | MEDIUM | CRITICAL | |
| Incident Correlation | N/A | None | |
| Cascade Escalation | N/A | No | |
| Remediation | — | Auto-Heal — Corax resolves autonomously |
Scenario Conditions
Amazon Linux 2023. Memcached 1.6 writing to /var/log/memcached/memcached.log. logrotate config missing or broken. Log file at 48GB. Disk partition at 98%.
Injected Error Messages (1)
Memcached 1.6 log file /var/log/memcached/memcached.log at 48GB on Amazon Linux 2023 — logrotate broken, disk full imminent, partition at 98%, no space left warning, growing 9GB/day
Neural Engine Root Cause Analysis
Disk space exhaustion detected — the filesystem has reached capacity, preventing applications from writing data, logs, or temporary files. This can cause databases to crash, services to fail, and logging to stop (masking further issues). Inode exhaustion can also present as 'no space left' even when disk space appears available.
Remediation Plan
1. Identify the fullest filesystems with 'df -h' and check inode usage with 'df -i'.
2. Find and remove large unnecessary files: 'find / -type f -size +100M -exec ls -lh {} \;'.
3. Clear old log files, compress rotated logs, and verify logrotate is configured correctly.
4. Check for runaway log growth from verbose application logging.
5. Expand the volume or add additional storage if the data growth is legitimate.
Improvements Applied
- Severity: CRITICAL (expected MEDIUM)
Tested: 2026-04-02Monitors: 1 | Incidents: 1Test ID: cmnhnq5v805fslig7k4rojujc