Back to All Scenarios
PASSEDserver / linux_inode_exhaustion
Inode Exhaustion — Disk Not Full but Cannot Create Files
A Linux server runs out of inodes on the root filesystem despite having 40% disk space free. Millions of small session files created by a PHP application consumed all available inodes. No new files can be created, causing application failures and log rotation to break.
Pattern
DISK_FULL
Severity
CRITICAL
Confidence
95%
Remediation
Auto-Heal
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | DISK_FULL | DISK_FULL | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 4 linked | |
| Cascade Escalation | N/A | No | |
| Remediation | — | Auto-Heal — Corax resolves autonomously |
Scenario Conditions
Ubuntu 20.04 web server. ext4 filesystem with default inode ratio. 3.2 million session files in /tmp/php-sessions/. Disk usage 60% but inode usage 100%. Application cannot create temp files. Logrotate failing.
Injected Error Messages (1)
Linux server web-05 inode exhaustion — inode table full on root filesystem, cannot create new files despite 40% disk space free, df -i shows 100% inode usage on /dev/sda1, 3.2 million orphaned PHP session files consuming all inodes, application returning errors on file creation, logrotate failing: cannot create temp file
Neural Engine Root Cause Analysis
The web server web-05 is experiencing inode exhaustion on the root filesystem, with 3.2 million orphaned PHP session files consuming all available inodes despite having 40% free disk space. This prevents the system from creating any new files, causing the web application to fail and blocking critical system operations like logrotate. The inode table is at 100% capacity, making the filesystem effectively unusable even though physical storage space remains available.
Remediation Plan
1. Immediately clean up orphaned PHP session files from the session directory (typically /var/lib/php/sessions or /tmp) 2. Verify inode usage has decreased using 'df -i' 3. Restart the web server service to restore functionality 4. Implement proper session cleanup configuration in PHP settings 5. Set up monitoring for inode usage to prevent recurrence 6. Consider moving session storage to a separate partition or using alternative session handlers
Tested: 2026-03-30Monitors: 1 | Incidents: 1Test ID: cmncjz0cr05t8obqejey2d5av