Back to All Scenarios
PASSEDcloud / terraform_state_lock_conflict
Terraform State Lock Conflict — Parallel Apply Blocking Deployments
Two CI/CD pipelines triggered simultaneously attempt to run terraform apply against the same state file. The DynamoDB state lock prevents both from proceeding, but a stale lock from a crashed previous run is never released, blocking all infrastructure deployments for 4 hours.
Pattern
UNKNOWN
Severity
CRITICAL
Confidence
95%
Remediation
Auto-Heal
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | UNKNOWN | UNKNOWN | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 21 linked | |
| Cascade Escalation | N/A | No | |
| Remediation | — | Auto-Heal — Corax resolves autonomously |
Scenario Conditions
Terraform state stored in S3 with DynamoDB lock table. Two concurrent pipeline runs. Stale lock from prior crashed run. No lock timeout configured. 4 hours of blocked deployments.
Injected Error Messages (2)
terraform apply blocked — state lock acquisition failed, lock ID: 3f8a9b2c held by pipeline run #4521 which terminated abnormally 4 hours ago, DynamoDB lock table 'terraform-locks' contains stale entry, all subsequent terraform plan and apply operations are rejected, infrastructure deployment pipeline completely stalled, 12 pending infrastructure changes cannot be applied
terraform state file locked — concurrent access denied, lock held since 2026-03-29T02:15:00Z by process that no longer exists, manual force-unlock required using 'terraform force-unlock LOCK_ID', risk of state corruption if underlying process is actually still running, deployment freeze in effect across all environments
Neural Engine Root Cause Analysis
The Terraform state lock in DynamoDB table 'terraform-locks' is stuck with lock ID 3f8a9b2c from pipeline run #452 that terminated abnormally 4 hours ago. This stale lock prevents all subsequent Terraform operations (plan/apply) from executing, causing a complete infrastructure deployment pipeline blockage. The root cause is the failure of the previous pipeline run to properly release its state lock during abnormal termination, leaving a zombie lock that blocks all infrastructure operations.
Remediation Plan
1. Connect to DynamoDB and locate the terraform-locks table. 2. Query for lock ID 3f8a9b2c to confirm it exists and verify the associated pipeline run #452 timestamp. 3. Validate that pipeline run #452 is indeed terminated and not running. 4. Delete the stale lock entry from the DynamoDB terraform-locks table. 5. Verify lock removal by attempting a terraform plan operation. 6. Monitor subsequent pipeline runs to ensure normal operation resumes.
Tested: 2026-03-30Monitors: 2 | Incidents: 2Test ID: cmnckbw4q08bqobqepkdnu6mo