Backend & platform engineering
Deep dives into database internals, Kubernetes, Linux networking and Go, with hand-drawn diagrams, real code and measurements. By Joel Teodoro, backend engineer working with Go in production.
Pinned
Deep dive
The image never reaches the API server. What reaches it is a few hundred bytes of YAML. Tracing the real path from a commit to a process inside a container — a chain of independent loops, none of which knows the others exist.
onKubernetes & Platform postsLinux & Networking posts
14 min read
Latest writing
Build log
Every minute, every open order against every rider on duty, solved as one vehicle routing problem with VROOM. How the round works, why the solver's answer is only a proposal, and why the legs are diffed instead of rebuilt.
17 min read
Build log
One search box, six indexes, one round trip. How I built a multi-vertical search in Go on Elasticsearch, why it rebuilds its indexes instead of syncing them, and how I would sync them at a million documents per vertical.
16 min read
Build log
at and cron as a service, in Go: six services, MySQL as the queue, Cassandra for history. The decisions behind it, their trade-offs, and what load-testing it on an AWS-shaped bench found, including a bug of mine.
onSystem Design postsDatabase Internals postsGo posts
11 min read · Part 1 of Building a distributed job scheduler
Build log
An EKS and RDS lookalike on one PC, a cron platform with 30,000 jobs due at 09:00:00, and ten tests. Nothing was lost; the 300 jobs/s ceiling turned out to be an N+1 of mine.
onSystem Design postsDatabase Internals postsGo posts
11 min read · Part 3 of Building a distributed job scheduler
Build log
Why the jobs table is the queue, why MySQL and not Postgres, why Cassandra only for history, and how leases, fencing tokens and idempotency keys keep at-least-once honest. With the alternatives I rejected and the SQL behind each decision.
onSystem Design postsDatabase Internals postsGo posts
12 min read · Part 2 of Building a distributed job scheduler
Deep dive
I thought I understood net/http's connection pool. Then I found out the pool key ignores the path entirely, and that my own Transport config had silently turned off HTTP/2. Three discoveries, measured with httptrace.
onGo postsLinux & Networking posts
8 min read · Part 2 of net/http under the hood
Topics
Storage engines, indexes, MVCC, WAL and replication. Postgres, InnoDB, Redis and friends.
5 posts
How real systems are designed, where they break and why. Case studies and patterns.
6 posts
Kubernetes internals, controllers, CI/CD and the platform underneath.
1 post
Sockets, the kernel network stack, namespaces, cgroups: what containers are made of.
3 posts
Go runtime and standard library internals: concurrency, context, net/http.
10 posts
Projects
at and cron as a service, in Go: MySQL as the queue, Cassandra for history, leases and fencing tokens for at-least-once. Load-tested on an AWS-shaped bench.
go · mysql · cassandra · kubernetes
Series
Series · complete
at and cron as a service in Go, from the design decisions to the load tests.
3 parts
Series · ongoing
How Go's HTTP client really sends a request, from RoundTripper to the connection pool.
2 parts
Follow along via RSS.