Back to All Scenarios
PASSEDdatabase / oracle_tablespace_exhaustion
Oracle Tablespace Exhaustion — ORA-01653
An Oracle 19c production database runs out of space in the USERS tablespace after an overnight ETL job loads 3x the expected data volume. All INSERT and UPDATE operations fail with ORA-01653. The application returns errors on any write operation while reads continue to function.
Pattern
DATABASE_EVENT
Severity
CRITICAL
Confidence
95%
Remediation
Remote Hands
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | DATABASE_EVENT | DATABASE_EVENT | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 21 linked | |
| Cascade Escalation | N/A | No | |
| Remediation | — | Remote Hands — Corax contacts on-site support via call, email, or API |
Scenario Conditions
Oracle 19c on Linux. USERS tablespace: 500GB, 99.8% utilized. Autoextend disabled by DBA policy. ETL job loaded 150GB overnight (expected 50GB). ASM disk group DATA at 97%. 200 concurrent OLTP sessions.
Injected Error Messages (2)
Oracle tablespace USERS exhausted — ORA-01653: unable to extend table HR.EMPLOYEE_TRANSACTIONS by 8192 in tablespace USERS, DBA_TABLESPACE_USAGE_METRICS: 99.8% full (499.1GB/500GB), autoextend: OFF, ASM diskgroup DATA at 97%, ETL session SID 487 loaded 150GB (3x expected), redo log archiving at risk due to disk pressure, database connection pool rejecting new write operations
Application write operations failing — ORA-01653 on all INSERT/UPDATE statements, user registration blocked, order placement returning 'unable to process request', read-only operations still functioning, 200 active sessions queued on write locks, application error rate: 45% and climbing
Neural Engine Root Cause Analysis
The Oracle database ora-prod-01 has failed due to tablespace USERS being 99.8% full (499.1GB/500GB) with autoextend disabled, preventing table HR.EMPLOYEE_TRANSACTIONS from extending. An ETL process (SID 487) has loaded 150GB of data (3x expected volume), causing rapid tablespace exhaustion. The underlying ASM diskgroup DATA is at 97% capacity, creating additional storage pressure and putting redo log archiving at risk. This is a cascading storage failure affecting database write operations and likely causing the 10 correlated incidents.
Remediation Plan
1. IMMEDIATE: Kill the runaway ETL session (SID 487) to stop further data loading. 2. Enable autoextend on USERS tablespace as emergency measure if underlying disk has minimal space. 3. Add storage to ASM diskgroup DATA or add new datafiles to USERS tablespace from different disk location. 4. Investigate ETL job that loaded 3x expected data volume - check for data duplication, missing filters, or upstream data issues. 5. Implement tablespace monitoring alerts at 85% threshold. 6. Review and optimize HR.EMPLOYEE_TRANSACTIONS table structure and archival policies. 7. Restart database services once storage issues resolved.
Tested: 2026-03-30Monitors: 2 | Incidents: 2Test ID: cmncjtsth04mhobqeg3otfvet