runtime/rants

Project

orchard

active

A toy Kubernetes written by hand in Go: control plane, node agents and an L4 load balancer, running on VMs. A pod is a network namespace with a process.

go · linux · netns · iptables · vms

View repo

What it is

A miniature orchestrator, built by hand to understand how the real ones work. It is a work in progress and a learning project. Nothing wraps an existing tool: the load balancer moves the bytes itself, and the agent creates the network namespaces and writes its own iptables rules.

It started as an L4 load balancer and grew. In Kubernetes terms, the agent today is kubelet plus kube-proxy: pod lifecycle, IP allocation, veth pairs, a node bridge and a set of KUBE-* iptables chains. There are no images and no container runtime: a pod is a network namespace with a process in it.

Architecture

Two components, kept in one repository:

  • orchestrator/: the API server and the L4 load balancer. It tracks nodes, dispatches events to agents and forwards client TCP connections to a node.
  • agent/: the node agent. It registers with the control plane, then creates services and pods locally: namespaces, veth pairs, IP allocation and iptables rules.

Balancing happens twice: the load balancer picks a node, and that node’s iptables rules pick a pod. The data plane (:8080) and the control plane (:9000) always listen on different ports. Each node takes its own /24 of 10.244.0.0/16, derived from its hostname.

client ──▶ lb data plane ──▶ nodeIP:NodePort
                                   │
                            PREROUTING
                                   │
                            KUBE-SERVICES          -d clusterIP --dport → KUBE-SVC (local + remote)
                                   │               -d nodeIP   --dport → KUBE-EXT (local only)
                                   ▼
                            KUBE-SVC-<hash>        -m statistic --probability 1/n
                                   │
                    ┌──────────────┼──────────────┐
                    ▼              ▼              ▼
            KUBE-SEP-<hash>  KUBE-SEP-<hash>  (remote node backend)
                    │
              DNAT podIP:podPort
                    │
                    ▼
              br0 ──▶ veth ──▶ pod netns eth0

What I learned

  • The packet-by-packet balancing inside a node is done by the kernel. The agent computes the -m statistic probabilities and keeps the rules correct as pods come and go: a control plane, not a data plane.
  • The draw happens once per connection. Only the first packet goes through the statistic rules; the rest follow the conntrack entry.
  • NodePort traffic needs its own chain (KUBE-EXT-*), otherwise a packet could be balanced again and bounce between nodes.
  • A PID alone cannot identify a pod after an agent restart, because PIDs get reused. PID plus process start time from /proc does.

The README is explicit about the trade-offs: Linux and root only, no filesystem isolation or cgroups, a hardcoded CRI and CNI, and iptables rules written by shelling out instead of netlink.

What’s next

Declarative state. Today the control plane dispatches imperative events, with no desired state to reconcile against. The README calls this the single biggest thing separating the project from an orchestrator, and it is on the roadmap.

Esc