Postmortem: the day our cron jobs ran twice

At 2:03 a.m. local time on the morning the clocks changed, every scheduled job on our billing host fired twice. Invoices went out twice. A reconciliation email went out twice. Thankfully the payment job itself was idempotent by design, so no customer was charged twice — but it took us four hours to be confident of that, and confirming it was not charged twice is a different kind of stressful than knowing it wasn't.

This is the postmortem we wrote internally, lightly edited to remove anything specific to our customers. I'm publishing it because the root cause is a category of bug that's extremely easy to reintroduce even after you've seen it once, and I'd rather more people recognize the shape of it before it happens to them.

The short version: our job scheduler computed "jobs due to run" by checking whether the current wall-clock time had crossed a threshold since the host's last check, using local time rather than UTC. When clocks fell back an hour, the scheduler's local-time window logic saw the same hour happen twice and dutifully ran everything scheduled in that hour a second time.

Timeline

2:03 a.m. — Clocks fall back from 2:00 a.m. to 1:00 a.m. local time. Scheduler re-enters the 1:00-2:00 window.
2:04 a.m. — Every job scheduled between 1:00 and 2:00 a.m. is re-evaluated as "due" and fires a second time.
2:11 a.m. — First automated alert fires: duplicate row detection on the invoicing table, which we'd added for an unrelated reason eight months earlier and which turned out to be the thing that saved us.
2:19 a.m. — On-call engineer acknowledges, begins triage, quickly identifies duplicate job executions rather than a data integrity issue.
2:40 a.m. — Scheduler paused fleet-wide as a precaution while we assessed blast radius.
4:52 a.m. — Confirmed the payment processor call was idempotent via an idempotency key already in place, meaning no customer was double-charged despite the job running twice.
6:15 a.m. — Scheduler re-enabled with a temporary patch forcing all time comparisons to UTC.

Root cause

The scheduler had been written years earlier by an engineer no longer at the company, and it stored a "last run" timestamp and compared it against the current time using the host's local timezone, which was set to the company's primary office timezone rather than UTC. This had worked without incident for years because most jobs ran hourly or daily and local-time drift during a DST transition either skips an hour (spring forward) or repeats one (fall back), and the scheduler had never been exercised during a fall-back transition since its original deployment — an infrastructure migration two years prior had, by coincidence, happened to avoid that exact week each year.

Here's a simplified version of the actual logic that caused it:

def jobs_due(last_run_local, now_local, interval_minutes):
    elapsed = (now_local - last_run_local).total_seconds() / 60
    return elapsed >= interval_minutes

# now_local used datetime.now() without tzinfo,
# which silently uses the host's local timezone.
# During fall-back, now_local can be LESS than
# last_run_local was a moment ago, and once the
# clock catches back up past last_run_local again,
# "elapsed" briefly satisfies every job's interval
# a second time within the same clock hour.

Spring-forward transitions had been safe by pure luck: skipping an hour means some jobs get delayed by up to an hour, which is a minor annoyance, not a duplicate-execution bug. Fall-back is the dangerous direction, and because it only happens once a year, a codebase can carry this kind of bug for a long time without anyone noticing.

Why idempotency saved us

The only reason this postmortem doesn't involve apologizing to customers for double charges is that the payment call included a request-scoped idempotency key, generated per invoicing period rather than per job execution, which the payment processor used to collapse the duplicate request into a no-op on the second attempt. That design decision predates this incident by a year and had nothing to do with DST — it was added because of a completely unrelated retry-on-timeout bug. It's a good reminder that idempotency isn't a nice-to-have for jobs that touch money or any external system you don't fully control; it's the thing that turns a scheduling bug into a non-event instead of an incident report with a different tone entirely.

What we changed

All scheduler time comparisons now run in UTC, full stop, with a lint rule that flags any new use of naive local-time datetime objects in the scheduler codebase.
Added a dedicated integration test that fast-forwards a simulated clock through both DST transitions and asserts job execution counts.
Audited every other system that read the host's local time for scheduling decisions — we found two more, neither as consequential, both fixed the same week.
Required every new scheduled job touching an external system to document its idempotency strategy explicitly in the job's code comments and in the on-call runbook.

The uncomfortable lesson

Nobody who built this system intended to write a DST bug. The code had worked, visibly and repeatedly, for years. The actual lesson isn't "always use UTC," even though that's true and we should have been doing it from day one — it's that a bug which only triggers once a year on a specific calendar event can survive multiple team handoffs, multiple infrastructure migrations, and multiple rounds of "this code is old and stable, nobody touches it" precisely because stability and correctness aren't the same thing, and the gap between them is invisible until the one day a year it isn't.