Back to All Scenarios
PASSEDdatabase / disk_io_contention
ClickHouse 24.3 High Disk IO Wait — Performance Degraded on Ubuntu 22.04 LTS
ClickHouse 24.3 performance is severely degraded due to disk I/O contention from another process on the same host.
Pattern
STORAGE_IO_LATENCY
Severity
HIGH
Confidence
64%
Remediation
Auto-Heal
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | STORAGE_IO_LATENCY | STORAGE_IO_LATENCY | |
| Severity Assessment | HIGH | HIGH | |
| Incident Correlation | N/A | None | |
| Cascade Escalation | N/A | No | |
| Remediation | — | Auto-Heal — Corax resolves autonomously |
Scenario Conditions
Ubuntu 22.04 LTS. ClickHouse 24.3 disk i/o wait at 91%. Another process running heavy sequential writes. Read latency 917ms (normal: 2ms).
Injected Error Messages (1)
ClickHouse 24.3 disk i/o contention on Ubuntu 22.04 LTS — storage latency critical, read latency 917ms (normal 2ms), disk i/o wait 91%, clickhouse-server performance severely impacted
Neural Engine Root Cause Analysis
Storage I/O latency detected — disk operations are taking significantly longer than normal, causing application slowdowns, database query timeouts, and degraded user experience. High iowait indicates the CPU is spending excessive time waiting for disk operations to complete, often due to disk contention, failing drives, or storage subsystem overload.
Remediation Plan
1. Check I/O statistics with 'iostat -x 1 5' to identify which disk is bottlenecked.
2. Review iowait percentage in 'top' or 'vmstat' — sustained values above 20% indicate a problem.
3. Look for processes generating excessive I/O with 'iotop' or 'pidstat -d'.
4. Check RAID array health and disk SMART data for signs of drive failure.
5. Consider migrating to faster storage (SSD/NVMe) or offloading heavy I/O workloads.
Tested: 2026-04-02Monitors: 1 | Incidents: 1Test ID: cmnhnwhto057olijguq0ku6qc