Reliable cron: The Simplest Scheduled Job Gets Hard the Moment It's Distributed

cron may be the simplest piece of infrastructure there is: time’s up, run a job. Anyone who has written a crontab on a single machine takes it for granted. But add one word — “reliable” (that machine dies and the job still has to run) — and it turns overnight from the simplest thing into a hard problem that drags in distributed consensus. This post covers why, and the choice with no perfect answer that it forces on you.

Single-machine cron is easy; reliable cron is hard

Single-machine cron’s fatal flaw is obvious: it’s a single point of failure. That machine dies, every schedule stops, and you may not notice for several cycles. The intuitive fix is to make it distributed — several replicas, elect one leader to run things, hand over when it dies. But the moment you do that, a new hard problem appears: at the instant the leader hands over, the state of “which jobs have already run” must not be lost — otherwise the successor has no idea whether a job ran, and the result is either a re-run or a skip.

In other words, the difficulty of reliable cron isn’t the scheduling itself; it’s the durability of state: the ledger of “which jobs, in which cycle, ran or not” has to survive any machine’s crash and any change of leader. And that is exactly an application of distributed consensus — use Paxos or the like to store that ledger as a log everyone agrees on and that doesn’t disappear on failure.

No free exactly-once: skip vs. duplicate, pick one

Even with consensus holding the state, a sneakier problem hides in the gap between two actions: “decide to run” and “record that it ran”. If the leader crashes inside that gap, you inevitably hit one of two disasters:

So the real question to ask first is: for this job, is skipping more painful, or duplicating? Sending a billing notification twice is embarrassing, so you’d rather have a mechanism that blocks duplicates; producing an overwritable report, missing one run is worse than producing it twice, so you’d rather re-run. And the most elegant solution removes the choice altogether — make the job idempotent, so duplicate execution is harmless, and you can always choose “at least once” and sleep at night. It’s exactly what I kept stressing in the Airflow scheduling post: idempotent and re-runnable is the foundation of a data job, not a bonus.

One more pit: the midnight thundering herd

A final practical trap: everyone loves scheduling on the hour, especially 0 0 * * * (midnight). The result is that at 00:00:00 every day, hundreds or thousands of jobs surge out at once, grab resources at once, hit the same downstream at once — that’s the thundering herd. The fix is simple but often forgotten: add jitter, randomly scattering trigger times within a small window so no two jobs pile onto the same second.

Reflections

cron is the best example of “the simple gets hard once distributed”

I love using cron as an example, because it perfectly demonstrates a cruel law of distributed systems: however simple something is on one machine, that’s how hard it gets when distributed. Single-machine cron is a crontab any beginner can write; reliable cron needs leader election, a consensus log, crash-window analysis — several orders of magnitude harder, and the requirement sounded like “just add one word: reliable”. It’s made me ever more wary of the line “this requirement is simple, right?” — very often what’s simple is the happy path, and the real cost hides entirely in the edges of “what if it dies, what if it crashes right in the middle”. When estimating, estimate those edges, not the happy path.

Skip or duplicate: work out first which one your job fears

“No free exactly-once” is a sentence I think everyone doing scheduling, messaging or data pipelines should carve into their bones. Too many people assume the system will “run exactly once”, then stare in disbelief at duplicated invoices or a missed settlement after some failure. The reality is that you must choose between skipping and duplicating, so better to choose early and clear-headed. And my default answer is almost always — make the job idempotent, then choose “rather duplicate”. Because idempotency turns a “pick-one dilemma” into a comfort zone where “either choice is fine”; it’s the highest-return defensive design I’ve seen, and it says the same thing as re-runnable data pipelines and deduplication in messaging systems.

In the end, reliable cron is an exercise in consensus

The most interesting realisation writing this post was that “reliable cron” isn’t an independent topic at all; it’s an application of distributed consensus. You think you’re solving scheduling; what you’re actually solving is “how does a group of machines that can die agree on ‘did this job run’” — which is the same problem as electing a leader or managing a distributed lock, in different clothes. It confirms a belief of mine once more: the hard problems of distributed systems keep coming back to a handful of cores (consensus, state, failure boundaries); chew through the cores, and most “new problems” you meet are old ones with a new face. Which is also why infrastructure this low-level should use a proven off-the-shelf solution, rather than every team rebuilding that crash window for itself.