Back to All Scenarios
PASSEDnetwork / ntp_stratum_drift
NTP Stratum Drift — Multiple Servers Affected
The primary NTP server loses its upstream time source and begins drifting. As a stratum 1 source for the internal network, all downstream servers inherit the drift. Kerberos authentication begins failing when clock skew exceeds 5 minutes.
Pattern
UNKNOWN
Severity
CRITICAL
Confidence
95%
Remediation
Remote Hands
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | UNKNOWN | UNKNOWN | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 6 linked | |
| Cascade Escalation | N/A | No | |
| Remediation | — | Remote Hands — Corax contacts on-site support via call, email, or API |
Scenario Conditions
Primary NTP server (GPS receiver failure). 200 servers syncing from internal NTP. Clock drift rate: 2 minutes/hour. Kerberos max clock skew: 5 minutes. AD authentication failing after 2.5 hours of drift. Database transaction timestamps inconsistent.
Injected Error Messages (2)
NTP server stratum drift — primary NTP server ntp01 lost upstream GPS time source, stratum changed from 1 to 16 (unsynchronized), clock drifting at 2 minutes/hour, 200 downstream servers inheriting incorrect time, current drift: 7 minutes from actual UTC
Kerberos authentication failing due to clock skew — time difference between client and server exceeds maximum allowed (5 minutes), AD authentication rejected with KRB5KRB_AP_ERR_SKEW, NTP drift propagated to all domain controllers, database transaction timestamps inconsistent across cluster
Neural Engine Root Cause Analysis
The primary NTP server ntp01 has lost its upstream GPS time source, causing it to become unsynchronized (stratum 16). This is a hardware or connectivity failure affecting the GPS receiver or antenna system. The server is now drifting at 2 minutes per hour and has already accumulated 7 minutes of drift from UTC, propagating incorrect time to 200 downstream servers.
Remediation Plan
1. Check GPS antenna cable connections and power supply to GPS receiver 2. Verify GPS receiver hardware status and satellite lock 3. If GPS hardware failure confirmed, temporarily configure secondary NTP servers (pool.ntp.org) as upstream sources 4. Restart ntpd service to reset stratum and sync with new upstream sources 5. Monitor time drift correction across all 200 downstream servers 6. Replace GPS hardware if faulty 7. Implement redundant GPS/NTP architecture to prevent future single points of failure
Tested: 2026-03-30Monitors: 2 | Incidents: 2Test ID: cmnck2jgm06a4obqeelhcuvke