Building a distributed job scheduler without a queue

I built at and cron as a service. You ask it to call one of your HTTP endpoints, once at a given time or on a cron expression, and it does that with a priority, automatic retries, a dead letter queue for the calls that keep failing and a history you can query. It is written in Go, split into six services, and it keeps its state in MySQL and Cassandra.

I designed it for big tech scale, around a billion executions a day with peaks of 500,000 per second, and I built and load-tested it at a much smaller scale on my own PC. This post is the tour: the problem, the architecture, the decisions that matter and what the benchmarks found, including a bug of mine they caught. Two more posts go deeper, one on the design decisions in depth and one on the load tests. The code is on GitHub.

Why scheduling is harder than it looks#

The features fit in one sentence: run a call now, at a date or on a cron expression, with a priority (critical, high, normal or low), retries and then a dead letter queue (DLQ), and a status and history you can query.

The hard part is the non-functional list. An accepted task is never lost. Every execution happens at least once and is never dropped silently. The p99 scheduling lag, how late an execution starts compared with when it was due, has a target per priority: under 1 s for critical, under 20 s for high, a few minutes for normal and under 20 minutes for low. Low priority never starves, tenants get a fair share, the API aims at 99.99 %, and secrets never reach a log.

Then there is the trait that defines this domain: people schedule on the hour. They write 0 9 * * *, not 17 9 * * *. The average load of a cron platform is low, but nearly all of it lands on second :00, and at 09:00 the every-minute, hourly and daily crons all meet. In the load model I tested with, the average is about 38 executions per second, and 09:00:00 brings 30,000 of them due in the same second. A scheduler is judged by its peaks, not by its average.

Tasks and jobs#

Two words carry the whole design. A task is the configuration: the schedule, the priority, the HTTP request, the retry policy and next_run_at, the next time it is due. It changes rarely. A job is one execution of a task, and it never stops changing: it is written, claimed, retried and finished, thousands of times a second at scale.

They behave differently, so they live in different tables. When a job is created it copies the task’s request, timeout, retry policy and priority into its own row. Editing a task never changes a job that already exists: the job that fires at 09:00 is the one that was planned, even if someone edited the URL at 08:59:59.

The architecture#

Six services in Go, each built as a hexagon (ports and adapters), and two databases. The task service is the API to create, edit, pause, resume and delete tasks. The scheduler turns due tasks into jobs. The executors claim jobs, call the endpoints and save the results. The reconciler rescues jobs whose executor died. The archiver moves finished jobs from MySQL to Cassandra. The job service is the read side: the status of a job, the history of a task, the DLQ and manual retries.

What is missing matters as much as what is there. There is no message broker: the jobs table in MySQL is the queue.

The life of a job#

Take a task with the cron expression 0 9 * * * and follow tomorrow’s run.

The scheduler creates it ahead of time. Every few seconds it takes the active tasks due in the next 30 seconds and, in one transaction, inserts their jobs as pending and moves each task’s next_run_at to the following run. By 08:59:35 tomorrow’s job exists.

An executor claims it. Each executor takes as many pending jobs as it has free workers, among those due before its next poll, with FOR UPDATE SKIP LOCKED: it locks the rows it reads and skips the ones another transaction already holds, so executors never wait for each other. It marks them running with its own id, adds one to attempt and gives each a lease, a deadline after which the job counts as abandoned. Then it commits, before any HTTP call.

The job waits for its second. Each claimed job sleeps on its own timer in memory until 09:00:00, and then the executor sends the request with three headers: Idempotency-Key with the job id, X-Scheduled-At and X-Attempt.

The result is saved only if the job is still ours. A 2xx makes it succeeded. Anything else sends it back to pending with exponential backoff, or to dead once its attempts are used up. If the executor dies instead, the lease expires and the reconciler returns the job to pending.

Finished jobs move to history. The archiver writes them to Cassandra and deletes them from MySQL, an hour after they succeed or 14 days after they die, the window in which a tenant can retry them from the DLQ.

The decisions that shape it#

Each of these gets its alternatives, queries and edge cases in the design post. Here is the short version, with the price of each.

The table is the queue#

With SQS, RabbitMQ or Kafka, every job has to be written twice, to the database and to the queue, and nothing makes those two writes atomic. A crash between them leaves the job in one and not in the other. Kafka has a second problem for this workload: it tracks progress with one offset per partition, so a slow job holds back every job behind it. A transactional outbox (write to the database, and let a relay publish to the queue) would remove the dual write, but it adds the relay and some latency. With one source of truth, everything else gets simpler.

I took the idea from Shopify moving inventory reservations from Redis to MySQL for the same reason, claiming rows with SKIP LOCKED. The cost is that all the claiming load lands on MySQL, so the jobs table has to be sharded sooner, by hash with something like Vitess.

MySQL for the hot table, not Postgres#

Every state change of a job is conditional, UPDATE ... WHERE status = 'running' AND ..., and in MySQL that is a cheap WHERE. Postgres would pay more for this table. Its indexes point to the physical row, so an update can rewrite every index. The escape is a HOT update (heap-only tuple, an update that skips the indexes), which is impossible when an indexed column changes, and the claim needs status indexed. Add vacuum and bloat on a table with this much churn. On AWS this is RDS for MySQL or Aurora MySQL, both on InnoDB.

Cassandra for history only#

Cassandra takes huge write volumes, but it resolves conflicts with last-write-wins (the write with the newest timestamp wins, nothing is compared), and a conditional update needs a lightweight transaction, a Paxos round on every write. That is far too expensive for every state change. Once a job is finished, though, it never changes again, so history is append-only and those problems go away. It is partitioned by tenant, task and month, with a 30-day TTL. The archiver writes to Cassandra first and deletes from MySQL second, so a crash in between leaves a duplicate, never a lost job.

Leases in the database, not locks in Redis#

The claim already writes the job’s row, so a lock in Redis would be a second source of truth that can disagree with the database, and that can vanish in a failover, without saving a single write. The claim stores the lease in the row instead, lease_expires_at, counted from the job’s due time and not from the claim. Otherwise the reconciler would take back jobs that were still waiting on their timer. The rule behind it: never run a job that is not persisted.

At-least-once, with a fencing token#

If a call times out, there is no way to know whether the endpoint did the work. Exactly-once delivery over a network does not exist, so the scheduler promises at-least-once and gives the receiver a way to deduplicate: the same Idempotency-Key on every attempt of a job.

What the database can guarantee is that a late result never corrupts the job. Every result write carries this guard:

WHERE id = ? AND status = 'running' AND worker_id = ? AND attempt = ?

attempt grows on every claim, so it works as a fencing token, a number that increases with each new owner so that writes carrying an old number are rejected. If executor A is slow, its lease expires and executor B claims the job, A’s late write affects 0 rows. What it cannot prevent is the endpoint receiving the call twice, and that is what the idempotency key is for.

On time without polling every second#

Precision does not come from polling the database often. The scheduler creates jobs up to 30 seconds ahead, the executors claim them shortly before their time, and each job fires from its own timer in memory at the exact second. The scheduler’s query has no lower bound (next_run_at <= horizon), so a task the scheduler gets to late is delayed but never skipped, and the next run is computed from the previous next_run_at, not from now, so the runs missed during an outage are caught up afterwards.

SSRF is checked when connecting, not when saving the URL#

The executor calls URLs chosen by tenants from inside my network, next to MySQL and the cloud metadata service. Server-side request forgery (SSRF) is a tenant using it to reach them. Validating the URL when the task is created is not enough, because of DNS rebinding: a name can resolve to a public IP at validation and to an internal one at call time. So the check runs in the Control hook of Go’s net.Dialer, after DNS resolution and right before connect(), on the exact IP being dialed. Only public unicast addresses on ports 80 and 443 get through, with no proxy, no redirects and a generic error. I wrote up the attack and the fix in my notes.

What the benchmarks said#

The load model is a cron platform with 1,000 tenants and 50,000 tasks, where the 10 largest tenants hold 40 % of them, and endpoints that range from fast to slow, flaky and permanently down. I ran ten tests, about 260,000 jobs in total.

The last row is the interesting one. One executor sustained 301 jobs per second, two sustained 336 and three 330. The executors were idle, waiting for work, and MySQL had room left. The scheduler was the limit, and it used about a fifth of its CPU.

That combination, a bottleneck that is not busy, means it is waiting. The scheduler inserted a batch of up to 1,000 jobs in one statement, and then advanced each task’s next_run_at with its own UPDATE: 1,000 round trips per tick, each paying the 1 ms network. A tick took about 3.3 seconds. It was an N+1, and it was my bug. Debugging the benchmark results is what found it. The fix is one statement for the whole batch:

UPDATE tasks
JOIN (VALUES ROW(?, ?), ROW(?, ?), ...) AS next_runs (id, next_run_at)
  ON tasks.id = next_runs.id
SET tasks.next_run_at = next_runs.next_run_at

Measured on the scheduler alone, a tick of 1,000 tasks went from 5.5 s to 54 ms with 1 ms of latency to MySQL, and from 380 ms to 40 ms without it. I have not run the full system tests again since the fix, so I don’t know the new ceiling yet. The next limit is probably the executors’ CPU during bursts, and that one scales out.

One more result changed how I think about the design. At the 09:00 peak, critical jobs were exactly as late as low-priority ones. The executor claims by priority, but only among jobs that already exist, and the scheduler picks tasks by time alone, so a critical job in the last batch waited for the 29 batches before it. Priority has to apply at the source too. The benchmarks post has the full results and the other lessons.

What’s next#

First, run the capacity test again to find the real next limit. Then the parts of the design that are not built: a fast path for critical jobs, inserted as running and fired at once; priority at the source, with the jobs of large crons created further ahead so a peak is ready before it starts; and concurrency limits per tenant and per endpoint, so the 09:00 burst does not flood the tenants’ own APIs.

Beyond one shard, the path is to shard tasks and jobs by hash with Vitess, give each scheduler a set of task partitions it owns through a lease, run executors per shard with claims weighted by priority and tenant, and go multi-region for the 99.99 %. Further out, cells (each holding a group of tenants) and regional crons.

The code, the migrations, every service’s design notes and every benchmark report are in the repo.