runtime/rants

Project

Coalesce (go-key-affinity-lb)

shipped

Routes requests by key so each node collapses duplicate in-flight work.

go · singleflight

View repo

What it is

A load balancer that routes by key affinity so that in-flight work can be deduplicated, turning 1000 concurrent requests into 4 backend calls.

Round-robin spreads requests for the same resource across every backend, so each node computes it independently: the thundering herd. Two mechanisms fix it, and they only work together: key affinity (the same key always goes to the same node) and singleflight (concurrent requests for a key on one node wait on one in-flight computation).

Architecture

                                  ┌──────────────┐
                    ┌────────────►│ node :8081   │  singleflight + result cache
                    │             ├──────────────┤
   client ─────────►│  balancer   │ node :8082   │
   channel-id       │   :8080     ├──────────────┤
                    │  fnv32a %   │ node :8083   │
                    └────────────►├──────────────┤
                                  │ node :8084   │
                                  └──────────────┘

The balancer reads the channel-id header, hashes it with FNV-1a and takes the modulo over the node count. Each node keys a singleflight.Group by channel id; the first request starts the work (a simulated 2s computation) and concurrent requests for the same channel receive its result. Completed results go into an in-memory map.

What I learned

The measured run: on a cold burst, 1000 requests cost 4 units of work and the whole burst took 2.06s, the latency of a single computation. The warm burst finished in 38ms, served from the caches.

From the design notes:

  • Header forwarding is load-bearing. Without copying channel-id, every node sees an empty key and the workload collapses onto one cache entry.
  • One http.Transport, not one per node. It already pools per backend; the value worth tuning is MaxIdleConnsPerHost, whose default of 2 causes connection churn under bursts.
  • Response bodies are always drained and closed, or connections never return to the idle pool.
  • Locks never span I/O. Holding the balancer’s mutex across the backend call would leave nothing concurrent for singleflight to coalesce.

Known limitations: modulo hashing (one node took two channels while another took none), no health checking, and caches that never expire.

What’s next

Esc