Load-testing a distributed job scheduler on an AWS-shaped bench

This is the third part of the distributed job scheduler. The first part says what I built and the second why it is built that way. This one is about whether it holds: how many jobs it runs, how late they start, and what breaks first. It summarises benchmarks.md, which has every test in full.

The platform#

I wanted the most realistic setup I could build quickly on one machine. A real Kubernetes cluster (k3d, which runs k3s nodes as containers) plays EKS, with one control plane and three 4 GB worker nodes. Every service runs as one pod sized like a Fargate task, 0.5 vCPU and 1 GiB, with requests equal to limits so no pod borrows from another. MySQL 8.4 is sized like an RDS db.t4g.medium (2 vCPU, 4 GB, 3 GB of buffer pool) and Cassandra 5 gets 2 vCPU, both outside the cluster as managed databases would be. The services run unchanged, with the same images and configuration as anywhere else.

The jobs call a fake endpoint written in Go that plays every customer. It sleeps or fails as each task’s URL tells it to, and records when each call arrived against when it was due.

The machine forced a few decisions. The WSL kernel has no netem, so the 1 ms of an availability zone comes from Toxiproxy, a TCP proxy that delays every reply from the databases (the calibration measured +1.17 ms per query). The WSL kernel cannot throttle disk I/O either, so MySQL writes to a local SSD, far faster than an RDS volume. And the calibration round found that k3s, misreading WSL’s cgroup v1 layout, had started without CPU limits at all: every pod could use all 12 CPUs until I forced CPU quotas on.

The first runs, before the test plan, showed one more thing: with the default settings one executor could not run more than 5 jobs per second, for reasons I come back to in the lessons. So the tests run 2 executors with 10,000 workers each, claiming every second, and the scheduler ticks every second with a 10-second lookahead instead of its defaults of 5 and 30.

A load that looks like a cron platform#

What defines a cron platform is not its average load but its aligned peaks. The model has 1,000 tenants and 50,000 active tasks, with the 10 largest tenants holding 40 % of them:

That is about 38 jobs per second on average, with 1,000 at the start of every minute, 5,000 every 5 minutes and 30,000 at 09:00:00, when everything meets.

The endpoints behave like real customers: 80 % answer in about 150 ms with a long tail, 12 % take 1 to 10 s, 5 % take 10 to 30 s and some hit the timeout, 2 % are flaky and return 503 on 30 % of calls, and 1 % are down and always end dead. Every task has a 30 s timeout, 3 attempts and a 5 s backoff, and priorities are 5 % critical, 15 % high, 60 % normal and 20 % low.

Targets and how I measured#

The start delay is when the call reached the endpoint minus when the job was due, measured by the fake endpoint. MySQL’s started_at would not do, because it records the claim, which happens before the job is due. Next to that, a collector read CPU, throttling and memory of every pod, plus MySQL’s commits and row lock waits, every 2 seconds straight from cgroups and /proc, and each test took 20-second CPU profiles of the scheduler and one executor at their busiest moment. The tests are short, from 3 to 30 minutes, which is a limit I come back to at the end.

Results, P0 to P9#

All ten ran on the same day, about 260,000 jobs in total.

Every job in every test ended succeeded or dead. The 16 duplicates of P8 are the at-least-once contract at work: those jobs had called their endpoint but could not save the result while MySQL was gone, so their lease expired and the reconciler ran them again. P7 had none because the kill landed between bursts, when the executor held few jobs; a kill in the middle of a burst would test the reconciler harder. P4 also counted one duplicate with no failure injected. I am fairly sure it was a manual call of mine that slipped into the run, not the system.

The 300 jobs/s ceiling was my N+1#

P2 ran one executor against fast endpoints, offering 50, 100, 200 and then 400 jobs per second in 60-second steps, and then repeated 400 per second on two and three executors.

A ceiling that does not move when you add workers means the workers are not the bottleneck. The telemetry agreed: the pending queue rose and fell in steps of exactly 1,000 and sat at zero most of the time, so the executors were waiting for jobs, not the other way round, and MySQL had room left. The jobs came from the scheduler, and the scheduler used about 100 of its 500 millicores. A bottleneck that is not busy is waiting on something, and its CPU profile was mostly the MySQL driver executing statements.

Each tick the scheduler took up to 1,000 due tasks, inserted their 1,000 jobs in one statement, and then advanced each task’s next_run_at with its own UPDATE. That is 1,000 round trips per tick, each paying the 1 ms to MySQL and back, plus the statement itself: a tick took about 3.3 s, so the scheduler produced about 300 jobs per second however often it ticked, since a tick already lasted longer than the interval. It was a classic N+1, and I wrote it. The benchmarks are what made me look.

The fix is one statement per batch, a join against a list of rows built with VALUES ROW(...):

UPDATE tasks
JOIN (VALUES ROW(CAST(? AS BINARY(16)), CAST(? AS DATETIME(3))),
             ROW(CAST(? AS BINARY(16)), CAST(? AS DATETIME(3))), ...)
  AS next_runs (id, next_run_at)
  ON tasks.id = next_runs.id
SET tasks.next_run_at = next_runs.next_run_at;

The casts are there because a placeholder inside VALUES has no column to take its type from, and MySQL would read it as text. I measured the scheduler alone with the same load as the probe, 10,000 one-off tasks due at 1,000 per second, on MySQL 8.4 in Docker with and without Toxiproxy’s 1 ms. It was not the k3d bench, so compare the two columns with each other, not with the 3.3 s above:

With latency, the tick is about 100 times faster, because it no longer pays a round trip per task. I have not run the full system tests again since the fix, so every result in the table above still comes from the old scheduler, and I don’t know the new ceiling yet. The most likely candidate is the executors’ 0.5 vCPU, which was already being throttled at 400 jobs per second.

The same N+1 explains most of the misses in P3 and P4. At the 5-minute mark, 5,005 jobs are due at once and the last batches of 1,000 arrived after their time; the rest of the 6.7 s was two executors at 0.5 vCPU firing about 3,000 calls at once, which takes them about 2 s and does not go away with the fix. At 09:00, creating 30 batches took about 100 s.

What I learned#

Priority only orders what already exists. At the 09:00 peak, critical jobs had a p99 of 91.6 s, the same as everyone. The executor claims by priority, but the scheduler picks tasks by time alone, so on a second with 30,000 tasks due the batches come out in no order of priority, and a critical job in the last batch waits for the 29 before it. A faster scheduler shortens that wait but does not order it. Priority has to apply at the source, and the jobs of large crons should be created further ahead so the peak is ready before it starts.

A timeout shorter than the work repeats the work. In P5, with a 20 s timeout and endpoints taking 10 to 30 s, 618 jobs ended dead, and the endpoints finished 4,326 calls that the executor had already given up on. Each of those was then sent again by a retry. Without an idempotency key on the receiving side, that is duplicated work at the customer.

Waiting has a price, and it is Little’s law. Little’s law says the number of items in a system equals their arrival rate times the time each one spends inside. Applied to my own design: an executor claims jobs up to one polling interval ahead, and each job holds its worker while it waits on its timer, so the workers in use are at least the jobs per second times the claim lead. With the defaults, 50 workers and a 10 s lead, one executor tops out at 50 / 10 = 5 jobs per second. That is why the tests ran with 10,000 workers and a 1 s lead. The limit was a setting, but I only saw it by measuring.

The hardware was not the problem. With 0.5 vCPU per service and 2 CPUs for MySQL, nothing ran out except the executor’s CPU, for a second or two at each burst. MySQL peaked at about 2,500 commits per second with at most 3 row lock waits per second. Isolation held too: in P6 a large tenant whose endpoint hung held a few hundred of the 20,000 workers, and the others barely noticed. The next limit is probably the executors at bursts, and unlike the scheduler’s, more replicas or more CPU fix it. Saving results in batches, instead of one UPDATE per job, would help as well.

What this bench can’t tell me#

Everything shares one host with 12 CPUs; limits reduce the interference but do not remove it. Every byte to the databases goes through a proxy in user space, so the baseline is “the proxy alone”, not “no network”. The latency is a fixed 1 ms with no jitter, loss or tail. The disk is not capped, so anything bound by MySQL’s fsync is an upper bound. The hot table fits in memory, which a production table would not. And there is one endpoint over plain HTTP, with no TLS and easy connection reuse, where production would have thousands of customer hosts.

The calibration also found something I have not explained yet: every 31 seconds or so, a whole second of jobs started up to 1 s late, which the later tests point to the host’s disk writeback.

What comes next is running P2 again to find the real ceiling after the fix, then a run of hours instead of 30 minutes, latency with jitter and a capped disk closer to RDS. The full benchmarks, with every chart, resource table and CPU profile, are in the repo.