Back to All Scenarios
PASSEDinfrastructure / leap_second_time_sync_drift
Leap Second Causing Time Synchronization Drift
A positive leap second insertion causes time synchronization issues across the infrastructure. Some servers handle it by stepping the clock, others by smearing, and a few that missed the NTP update have clocks 1 second ahead. This causes TLS handshake failures, database replication errors, and authentication failures due to clock skew.
Pattern
UNKNOWN
Severity
CRITICAL
Confidence
95%
Remediation
Remote Hands
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | UNKNOWN | UNKNOWN | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 18 linked | |
| Cascade Escalation | N/A | No | |
| Remediation | — | Remote Hands — Corax contacts on-site support via call, email, or API |
Scenario Conditions
Positive leap second at 23:59:60 UTC. Mixed NTP handling: 40% step, 35% smear, 25% unaware. Clock skew: up to 1.5 seconds between servers. TLS connections failing due to time-based validation. Distributed systems experiencing ordering errors.
Injected Error Messages (2)
NTP time synchronization anomaly — leap second insertion at 23:59:60 UTC causing clock divergence across infrastructure, 80 servers stepped clock (1-second jump), 70 servers using leap smear (smooth adjustment over 24 hours), 50 servers missed NTP update and are now 1 second ahead of correct time, maximum inter-server clock skew: 1.5 seconds, NTP stratum 1 servers showing offset warnings, authentication failing for servers with >1 second skew (5-minute tolerance exceeded when combined with existing drift)
distributed system consistency errors — clock skew between application nodes causing: event ordering violations in distributed log (events appearing out of sequence), distributed lock manager lease expiration race conditions, database replication conflict resolution choosing wrong 'latest' version, inter-service request tracing showing impossible negative-duration spans, 3 out of 8 application nodes have clocks drifting by >1 second, token validation failing on 50 servers with stale time
Neural Engine Root Cause Analysis
A leap second insertion occurred at 23:59:60 UTC, causing widespread NTP synchronization failure across the infrastructure. Different servers handled the leap second differently - some performed clock steps (1-second jumps), others used leap smear (24-hour gradual adjustment), and 50 servers missed NTP updates entirely, creating maximum clock skew of 1.5 seconds. This divergence has broken authentication mechanisms that have strict time tolerance requirements, causing cascading failures across 10 correlated systems.
Remediation Plan
1. Immediately identify and prioritize critical systems affected by authentication failures. 2. Force NTP synchronization on the 50 servers that missed updates using 'ntpdate -s' or 'chronyd makestep'. 3. Restart time-sensitive services (Kerberos, SSL/TLS services, databases) on servers with >1 second skew. 4. Configure all NTP clients to use consistent leap second handling (preferably leap smearing). 5. Implement monitoring for clock skew >500ms to catch future issues earlier. 6. Update NTP configuration to use multiple reliable stratum 1/2 sources.
Tested: 2026-03-30Monitors: 2 | Incidents: 2Test ID: cmnckif2v09skobqest32vhfr