distributed-scheduler

What it is#

A distributed job scheduler: at and cron as a service. Tenants schedule HTTP calls to their own endpoints, once or on a cron expression, with priorities, automatic retries, a dead letter queue and a queryable history. It is designed for around a billion executions a day and built and load-tested at a much smaller scale.

Architecture#

Six Go services, each hexagonal: a task API, a scheduler that turns due tasks into jobs, executors that claim jobs and call the endpoints, a reconciler for jobs whose executor died, an archiver that moves finished jobs to Cassandra, and a job API for status, history, the DLQ and manual retries.

There is no message broker. The jobs table in MySQL is the queue: executors claim rows with FOR UPDATE SKIP LOCKED, every state change is a conditional update, and attempt works as a fencing token so a late result can never overwrite a newer claim. Finished jobs move to Cassandra, written there first and deleted from MySQL second, so a crash leaves a duplicate and never a loss.

What I learned#

• One source of truth beats a broker when the job has to live in the database anyway: no dual write, and SKIP LOCKED hands out work safely.
• Nothing was lost in ten tests and about 260,000 jobs, including an executor killed mid-run and MySQL cut for 30 seconds.
• The 300 jobs/s ceiling was an N+1 of mine in the scheduler, one UPDATE per task. One statement per batch took a tick of 1,000 tasks from 5.5 s to 54 ms.
• Priority only orders what already exists. At the 09:00 peak, critical jobs were as late as everyone else.

What’s next#

Running the capacity test again after the fix, then the parts of the design that are not built yet: a fast path for critical jobs, priority at the source, concurrency limits per tenant and scheduler partitions owned through a lease.