Back to All Scenarios
PASSEDinfrastructure / dst_cron_double_execution
Daylight Saving Time Causing Cron Job Double-Execution
During the 'fall back' daylight saving time transition, cron jobs scheduled between 1:00 AM and 2:00 AM execute twice because the clock passes through that hour twice. A billing job runs twice, double-charging 15,000 customers. A cleanup job runs twice, deleting data that the first run already processed.
Pattern
UNKNOWN
Severity
CRITICAL
Confidence
95%
Remediation
Remote Hands
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | UNKNOWN | UNKNOWN | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 26 linked | |
| Cascade Escalation | N/A | No | |
| Remediation | — | Remote Hands — Corax contacts on-site support via call, email, or API |
Scenario Conditions
DST fall-back transition (2:00 AM -> 1:00 AM). Billing cron at 1:30 AM runs twice. 15,000 customers double-charged. Cleanup cron at 1:45 AM runs twice, second run deletes active data. No idempotency guards on either job.
Injected Error Messages (2)
billing cron job double-execution detected — job 'nightly-billing-run' scheduled at 01:30 local time executed twice during DST fall-back transition, first execution at 01:30 EDT processed 15,247 customer invoices, second execution at 01:30 EST (after clock rollback) re-processed same 15,247 invoices, total charges: 2x expected amount, 15,247 customers double-charged, no idempotency key on billing transactions, payment gateway processed all duplicate charges successfully
data cleanup cron double-execution causing data loss — job 'nightly-data-cleanup' scheduled at 01:45 local time ran twice during DST transition, first execution correctly purged 45,000 expired session records, second execution at 01:45 EST ran against already-cleaned dataset and deleted 12,000 ACTIVE session records matching broadened time window, users experiencing forced logouts, session table missing 12,000 valid entries, no idempotency guard on cleanup operation
Neural Engine Root Cause Analysis
The billing cron job executed twice during the DST fall-back transition when clocks rolled back from 01:30 EDT to 01:30 EST, causing the same scheduled time to occur twice. The job lacks proper idempotency controls and timezone-aware scheduling, resulting in duplicate processing of 15,247 customer invoices and double-charging through the payment gateway. This is a classic DST scheduling bug combined with insufficient duplicate prevention mechanisms in the billing system.
Remediation Plan
1. Immediately halt the billing cron job to prevent further executions. 2. Identify and reverse the duplicate charges for all 15,247 affected customers through payment gateway refund API. 3. Implement UTC-based scheduling for all cron jobs to avoid DST transitions. 4. Add idempotency keys to all billing transactions using invoice ID + date hash. 5. Implement job execution tracking with distributed locks to prevent concurrent runs. 6. Add pre-execution checks to detect and skip already-processed billing periods. 7. Test the fix in staging environment with DST transition simulation before production deployment.
Tested: 2026-03-30Monitors: 2 | Incidents: 2Test ID: cmncki1nb09pjobqeu2difwkt