Back to All Scenarios
PASSEDserver / ssh_key_rotation_breaking_automation
SSH Key Rotation Breaking Automation
An automated SSH key rotation process replaces host keys on 20 servers but fails to update the known_hosts files on the Ansible control node and monitoring servers. All SSH-based automation, configuration management, and monitoring breaks simultaneously.
Pattern
UNKNOWN
Severity
CRITICAL
Confidence
95%
Remediation
Auto-Heal
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | UNKNOWN | UNKNOWN | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 6 linked | |
| Cascade Escalation | N/A | No | |
| Remediation | — | Auto-Heal — Corax resolves autonomously |
Scenario Conditions
20 Linux servers with rotated SSH host keys. Ansible control node with stale known_hosts. Monitoring server SSH checks all failing. Backup scripts using SCP failing. No pre-rotation known_hosts update performed.
Injected Error Messages (2)
Ansible playbook execution failing on all 20 managed hosts — SSH host key verification failed after key rotation, REMOTE HOST IDENTIFICATION HAS CHANGED, all automation halted, config management drift occurring, known_hosts mismatch on control node for all rotated servers
SSH-based monitoring checks failing for 20 servers — host key verification failed after rotation, all SSH script checks returning critical status, SCP backup transfers failing with host key mismatch, monitoring showing false positives across entire Linux fleet
Neural Engine Root Cause Analysis
The Ansible control node is failing to connect to all 20 managed hosts due to SSH host key verification failures following a key rotation event. The known_hosts file on the control node (10.10.6.70) contains outdated host keys for the managed servers, causing SSH to reject connections with 'REMOTE HOST IDENTIFICATION HAS CHANGED' errors. This is preventing all Ansible automation from executing, leading to configuration management drift across the entire infrastructure.
Remediation Plan
1. SSH to the Ansible control node (10.10.6.70) 2. Backup the current known_hosts file 3. Remove stale host key entries for all 20 managed hosts from ~/.ssh/known_hosts 4. Use ssh-keyscan to collect new host keys from all managed hosts 5. Update known_hosts with the new keys 6. Test SSH connectivity to a sample of managed hosts 7. Execute a basic Ansible ping playbook to verify full connectivity 8. Resume normal automation operations
Tested: 2026-03-30Monitors: 2 | Incidents: 2Test ID: cmncjze3v05uoobqeu4k65kb1