From git push to a Running Pod: nobody is orchestrating anything

Ask an engineer what happens when they merge to main and you get some version of this: CI builds an image, and Kubernetes deploys it. It's not wrong so much as it's two claims, and both of them are false.

The first is that something gets sent to the cluster. It doesn't. Your image never goes anywhere near the API server — the thing that arrives there is a few hundred bytes of JSON saying that one field now has a different string in it. The second is that something is coordinating the work. Nothing is. There is no deployment orchestrator, no component that owns the process from commit to running process. There are six or seven independent programs, each watching for state that concerns it, each writing one narrow kind of change, none of which knows the others exist.

Here is the real path.

  git push
     │
     ▼
  ┌──────────┐  webhook   ┌──────────────┐  docker push   ┌────────────┐
  │  forge   │ ─────────► │  CI runner   │ ─────────────► │  REGISTRY  │
  └──────────┘   (POST)   └──────┬───────┘   (layers)     └─────┬──────┘
                                 │                              │
                        PATCH ~400 bytes                        │ image bytes
                        "image: app:a3f9c2"                     │ (never touch
                                 │                              │  the control
                                 ▼                              │  plane)
                        ┌─────────────────┐                     │
                        │   API SERVER    │ ──► etcd            │
                        └────────┬────────┘                     │
                    watch │ watch │ watch │ watch               │
              ┌───────────┘       │       │     └──────────┐    │
              ▼                   ▼       ▼                ▼    │
        deployment ctrl    replicaset ctrl  scheduler   KUBELET ◄┘
                                                           │  pulls it itself
                                                           ▼
                                                      containerd

Everything before the cluster is not Kubernetes at all

The pipeline starts with a webhook, not with polling. Your forge holds an outbound HTTP client; when the push lands it sends a POST to a URL you registered, with the commit SHA and ref in the body. Jenkins never "watched" your repository. It received a request, the same way any HTTP server receives one.

A runner then clones that exact commit, runs the tests, and builds. What docker build produces is worth being precise about, because the rest of the article depends on it: an image is not a tarball of a filesystem. It's a JSON manifest that lists a config blob and an ordered set of layers, each identified by its own content digest.

$ crane manifest registry.example.com/payments:a3f9c2 | jq '.layers[] | {size, digest}'
{ "size": 3372416,  "digest": "sha256:8a1e...b04c" }   # distroless base
{ "size": 14958592, "digest": "sha256:c2f0...9a11" }   # your binary

Each layer is a tarball of changes to the filesystem, and the union of them is what the container sees. This is why a multi-stage Dockerfile with a distroless final stage matters operationally rather than aesthetically: the base layer's digest doesn't change between builds, so the node already has it. On the next deploy the kubelet compares digests, finds the base cached, and pulls 15 MB instead of 400 MB.

a3f9c2 is a tag, and a tag is a mutable pointer. docker push can move it to a different digest an hour from now, which means the same tag on two nodes can be two different binaries. The digest is the immutable name:

$ crane digest registry.example.com/payments:a3f9c2
sha256:4d1e9b7c0f3a...

Serious production deploys pin image: registry.example.com/payments@sha256:4d1e9b7c... and treat the tag as a human-readable label only.

None of this involves Kubernetes. If you deleted your cluster right now, everything up to this point would still work exactly the same.

The handoff is one field

This is the hinge of the whole thing. The pipeline's final step runs something equivalent to:

kubectl set image deployment/payments app=registry.example.com/payments:a3f9c2

Turn on the verbose logging and you can see precisely what that is on the wire:

$ kubectl set image deployment/payments app=registry.example.com/payments:a3f9c2 -v=8
PATCH https://10.0.0.1:6443/apis/apps/v1/namespaces/default/deployments/payments
Request Body: {"spec":{"template":{"spec":{"containers":[{"name":"app",
                "image":"registry.example.com/payments:a3f9c2"}]}}}}
PATCH https://10.0.0.1:6443/.../deployments/payments 200 OK in 12 milliseconds

An HTTP PATCH. About 130 bytes of body. It changes one string in one field of one object in etcd, and returns.

At the moment that 200 comes back, nothing has been deployed. No pod has been created, no image has been pulled, nothing has been told to do anything. The pipeline is finished and it has no idea whether the rollout will succeed — that's why kubectl rollout status exists as a separate command that sits there polling.

And this makes the central point unavoidable: the image cannot pass through the API server, not by convention but by construction. etcd rejects values over 1.5 MiB by default. Your image is 400 MB. The registry is a completely separate service, usually on a different host, frequently run by a different company, and the only thing the cluster ever learns about your build is a string that names it. The kubelet on the node will later dial that registry itself, over its own TCP connection, with its own credentials.

There are two ways to make that PATCH happen, and the difference is only about who holds credentials:

• Push. The pipeline has a kubeconfig with write access to the cluster and issues the PATCH from a CI runner on the public internet.
• Pull (GitOps). The pipeline commits a one-line tag change to a manifests repository. A controller inside the cluster clones that repo every 30 seconds or so and reconciles the difference. Note that it polls — it isn't a webhook receiver — which is exactly why the cluster never needs to be reachable from outside, and why no CI system ever holds cluster credentials.

Either way, what lands is the same field change. Everything after this point is identical.

The API server has no opinions

The API server is not the brain of the cluster, and treating it as one is the reason the rest feels mysterious. Here is its complete job for that PATCH:

Authenticate the client (client certificate, bearer token, OIDC). Authorise the verb against the resource via RBAC. Decode the object and fill in defaults — this is where imagePullPolicy: IfNotPresent and terminationGracePeriodSeconds: 30 appear in a spec you never wrote them into. Run mutating admission webhooks, which is where service meshes inject sidecars. Validate against the schema. Run validating admission webhooks, which is where your policy engine rejects the deploy for running as root. Write to etcd. Stream the result to everyone watching.

That's it. There is no business logic in there. It never creates a ReplicaSet, never picks a node, never starts anything on its own initiative. It is a validated, versioned object store with a streaming endpoint bolted on.

That poverty is the design. Because the API server implements no domain behaviour, hundreds of controllers — the built-in ones, your service mesh, your cert manager, your operators, code you wrote last Tuesday — can hang off it without any of them needing to be registered with each other or even known to it. The extension model is the storage model.

Five processes that never talk to each other

The PATCH changed .spec.template. Now watch what unfolds, remembering that nobody sent a message.

The Deployment controller notices the pod template hash changed and creates a new ReplicaSet — a brand new object, with a pod-template-hash label derived from the template's contents. It does not create pods. It never creates pods.

The ReplicaSet controller notices a ReplicaSet whose desired replica count exceeds the number of pods carrying its label, and creates Pod objects to close the gap. These pods have no node. spec.nodeName is the empty string, and they are Pending.

The scheduler notices pods with an empty spec.nodeName, runs filtering and scoring across the nodes, and writes exactly one field. It POSTs to the pod's binding subresource, which sets spec.nodeName: node-1. Then it moves on. The scheduler never contacts a node. It has no connection to the kubelet, no idea whether the node accepted anything. It made a note in a database.

The kubelet on node-1 has a watch open with a field selector for its own name, so the moment that field is written it sees a pod it is responsible for.

Four processes, four narrow concerns, zero messages. What connects them is the watch: a long-lived HTTP request that stays open while the API server streams objects down it.

$ kubectl get pods -v=8
GET https://10.0.0.1:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dnode-1
    &resourceVersion=48219&allowWatchBookmarks=true&timeoutSeconds=499&watch=true

Note what comes down that stream: whole objects, not diffs. That is not laziness, it's the property the entire system rests on. A diff requires the receiver to already hold the correct previous state; miss one message and every subsequent diff applies to the wrong base, so your copy is silently and permanently wrong. A whole object is self-sufficient. Drop a hundred of them and the next one still leaves you correct. This is what "level-triggered" means, and it's why a controller can crash, restart, re-list everything and be perfectly consistent thirty seconds later — there was never a message whose loss mattered.

The consequence surprises people who write event handlers for a living: the controller doesn't act on the payload it just received.

   ┌──── API server ────┐
   │  watch (HTTP, open)│
   └─────────┬──────────┘
             │ whole objects
             ▼
     ┌───────────────┐      ┌──────────────────┐
     │  Reflector    │ ───► │  local cache     │ ◄──── worker reads
     └───────┬───────┘      │  (indexed store) │       CURRENT state
             │              └──────────────────┘             ▲
             │ just the key                                  │
             ▼                                               │
     ┌────────────────────┐        ┌───────────────┐         │
     │ workqueue          │ ─────► │   worker      │ ────────┘
     │ "default/payments" │  pop   │  reconcile()  │
     │ (dedup + ratelimit)│        └───────────────┘
     └────────────────────┘

The informer keeps a local cache of every object it watches. When an event arrives, the handler extracts the key — default/payments — and puts that string on a workqueue. The worker later pops the key and reads the current object out of the cache. The event payload is thrown away.

So five rapid changes to the same Deployment collapse into one reconciliation, because the workqueue deduplicates identical keys and the worker reads whatever the object looks like when it finally gets there. The controller never asks "what changed?" It asks "what does this look like now, and what would make the world match it?" — a question you can answer correctly having missed every single event that led here.

What happens on the node

The kubelet is the first thing in this chain that touches anything real. It sees a pod bound to itself and calls containerd over CRI, a gRPC socket on the local filesystem.

First RunPodSandbox. containerd creates the pause container: a process that does nothing but block on a signal, whose only purpose is to hold the network, IPC and UTS namespaces open so your containers can join them and so the pod IP survives your app crashing.

Then the network gets set up, and here is a detail worth correcting because half the diagrams on the internet still show it wrong: the kubelet does not call CNI. It did, once, back when dockershim existed. Today containerd's CRI plugin does it during RunPodSandbox. And what it does is not an API call and not a pod — it forks an executable off local disk:

$ ls /opt/cni/bin/
bridge  host-local  loopback  portmap  ptp
$ ls /etc/cni/net.d/
10-flannel.conflist

The runtime execs that binary with CNI_COMMAND=ADD in the environment, writes the network config as JSON to its stdin, and reads JSON back off stdout. The plugin creates the veth pair, moves one end into the sandbox's namespace, allocates an IP, writes the routes, and exits. A short-lived process, gone before your container starts.

Next the image. The kubelet checks whether the digest is already present on the node and, if not, calls the CRI image service to pull it — the node dialling the registry directly, resolving imagePullSecrets itself. This is the only time your image bytes move, and neither the API server nor the scheduler is involved or informed.

Then the step almost nobody knows about, and the one that most changes how you debug. Before creating the container, the kubelet resolves the environment. It walks every envFrom and every valueFrom in the container spec, issues GETs to the API server for each referenced ConfigMap and Secret, and builds a flat list of key/value pairs which it hands to containerd inside ContainerConfig:

ContainerConfig{
  image:   "registry.example.com/payments@sha256:4d1e...",
  command: ["/app/payments"],
  envs:    [ {key:"DATABASE_URL", value:"postgres://user:hunter2@db:5432/prod"},
             {key:"LOG_LEVEL",    value:"info"} ],
  linux:   { resources: { memory_limit_in_bytes: 536870912 } },
}

Your configuration never enters the image. It's a field in a struct, sitting next to the command and the memory limit, assembled on the node moments before the process exists.

Finally CreateContainer, then StartContainer. containerd hands off to runc, which sets up the cgroups and namespaces and execs your binary as PID 1 in its own PID namespace.

One more thing about the kubelet: it doesn't strictly need the API server at all.

$ ls /etc/kubernetes/manifests/
etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml

The kubelet reads pod manifests straight off local disk and runs them, which is how the control plane itself boots — the API server is a static pod started by a kubelet that has nothing to talk to yet. The kubelet is a self-contained agent that happens to also watch an API, not a remote executor being driven by the scheduler.

Running is not reachable

Your container is running. No traffic is going to it, and the machinery that fixes that is a completely separate chain from the one you just followed.

  BECOMING RUNNING                    BECOMING REACHABLE
  ─────────────────                   ──────────────────
  scheduler  → nodeName               kubelet     → readinessProbe → Ready
  kubelet    → sandbox                endpoints c → EndpointSlice += 10.244.1.7
  containerd → CNI, IP                kube-proxy  → rewrite netfilter rules
  containerd → pull                          │
  kubelet    → resolve env                   └── then goes back to sleep
  CRI        → Create/StartContainer          (the KERNEL forwards packets)

The kubelet runs the readiness probe — an ordinary HTTP GET to your port every few seconds, from the node itself — and when it passes, writes a Ready condition into the pod's status.

The endpoints controller is watching for exactly that. It sees a pod that is Ready and matches a Service's selector, and appends the pod IP to an EndpointSlice object:

$ kubectl get endpointslices -l kubernetes.io/service-name=payments -o wide
NAME             ADDRESSTYPE   PORTS   ENDPOINTS                READY
payments-x7k2p   IPv4          8080    10.244.1.7,10.244.2.3    true,true

That object is the actual source of truth for "who is behind this Service". If a pod is Running but getting no traffic, this is the first thing to look at — an empty list here means the readiness probe never passed, and the problem is two chains upstream of anything network-related.

kube-proxy on every node watches Services and EndpointSlices, and when that slice changes it rewrites netfilter rules:

$ iptables -t nat -L KUBE-SERVICES -n | grep payments
KUBE-SVC-4N57TFCL4MD7ZTDA  tcp -- 0.0.0.0/0  10.96.84.201  /* default/payments cluster IP */ tcp dpt:80
 
$ iptables -t nat -L KUBE-SVC-4N57TFCL4MD7ZTDA -n
KUBE-SEP-XLPNRJ7VMOP4B3BQ  all -- 0.0.0.0/0  0.0.0.0/0  statistic mode random probability 0.50000000000
KUBE-SEP-QQVQMR5FGKGH2WZ2  all -- 0.0.0.0/0  0.0.0.0/0
 
$ iptables -t nat -L KUBE-SEP-XLPNRJ7VMOP4B3BQ -n
DNAT  tcp -- 0.0.0.0/0  0.0.0.0/0  tcp to:10.244.1.7:8080

Read those rules closely, because they settle a very common misconception: kube-proxy is not in the packet path. Your pod IPs are written literally into the rules. There is no lookup, no table consulted at request time, no userspace process to ask. The load balancing is a statistic match with a probability. The address translation is a DNAT target evaluated by netfilter in the kernel, in softirq context, before the packet ever reaches a socket.

kube-proxy wrote those rules and went back to sleep. Kill it right now and traffic keeps flowing exactly as before — you've only stopped updates, so the rules will go stale as pods change. That is a very different failure from "the proxy is down", and it's why the symptom is traffic to dead pods rather than no traffic at all.

You can watch the kernel's side of it directly:

$ conntrack -L | grep 10.96.84.201
tcp 6 86397 ESTABLISHED src=10.244.0.5 dst=10.96.84.201 sport=51234 dport=80 \
                        src=10.244.1.7 dst=10.244.0.5 sport=8080 dport=51234 [ASSURED]

The client sent to the Service IP 10.96.84.201. The reply tuple shows the real pod, 10.244.1.7:8080. The kernel remembers that mapping for the life of the flow — which, incidentally, is why an existing connection keeps hitting the same pod after the rules change, and why UDP entries to a deleted pod are a classic source of "it works after 30 seconds".

The rolling update is the whole system in miniature

Watch the numbers during a deploy and you can see the design in one screen.

The Deployment controller does exactly one thing: it edits two integers. It scales the new ReplicaSet from 0 up and the old one down, subject to maxSurge and maxUnavailable, and between steps it waits — not for pods to exist, but for the new ReplicaSet's readyReplicas to catch up.

It never deletes a pod. Watch it happen:

$ kubectl get rs -w
NAME                  DESIRED   CURRENT   READY
payments-7d4b8c9f5f   3         3         3      # old
payments-6f9c5d8b7c   0         0         0      # new
payments-6f9c5d8b7c   1         1         0
payments-6f9c5d8b7c   1         1         1      # readiness gate passed
payments-7d4b8c9f5f   2         3         3      # desired dropped to 2 — that is all it did
payments-7d4b8c9f5f   2         2         2      # the ReplicaSet controller deleted one

Look at the last two lines. The Deployment controller changed desired from 3 to 2 and stopped. The pod died because the old ReplicaSet's own controller woke up, counted three pods matching its selector, read a spec asking for two, and deleted one to close the gap. It has no idea a deploy is happening. It has never heard of the Deployment controller. It compared two numbers.

That's the shape of every piece of this. The scheduler assigns nodes and knows nothing about containers. The kubelet runs containers and knows nothing about Services. kube-proxy writes rules and knows nothing about pods being created. Each one reconciles its own narrow concern against the current state of the world, and the deploy is what emerges when they all run at the same time.

This is why the system self-heals, and it's the same property, not a second feature: the loops never stop running. Delete a pod by hand and the ReplicaSet controller recreates it for the same reason it deleted one during your rollout — the count doesn't match. And it's why any component can crash mid-deploy without losing the rollout. There was no message in flight to lose. The state is in etcd, and whatever comes back up will read it and carry on from wherever things actually are.