What is the reconciliation/control loop pattern, and why is it central to Kubernetes?
Quick Answer
A control loop continuously watches for the current (actual) state of some resource, compares it against the declared desired state, and takes action to close any gap — repeating indefinitely. Kubernetes is built almost entirely out of many independent control loops (one per controller type: Deployments, ReplicaSets, Nodes, and so on), each responsible for reconciling one narrow slice of cluster state, which is what makes the whole system self-healing and eventually consistent rather than requiring a human to notice and fix every drift.
Detailed Answer
The control loop, abstractly
loop forever:
observed_state = get_current_state()
desired_state = get_desired_state() # from the object's spec
if observed_state != desired_state:
take_action_to_reconcile()
This is a much older pattern than Kubernetes (thermostats and cruise control are the classic non-software examples), but Kubernetes is unusual in building nearly its entire system out of many independent instances of this same loop, each watching a narrow slice of the cluster.
A concrete example: the ReplicaSet controller
spec:
replicas: 3
The ReplicaSet controller's loop: watch for ReplicaSet objects and their currently-running matching Pods; if fewer than 3 matching Pods exist, create more; if more than 3 exist, delete the excess. This loop runs continuously and reactively (triggered by watch events, not fixed polling — see the API server question) — if a Pod is deleted or a node crashes, the controller notices the actual count dropped below desired and creates a replacement, with no human needing to notice or intervene.
Why "eventually consistent," not "instantly consistent"
A control loop doesn't guarantee the desired and actual state match at every instant — only that the system keeps working toward convergence. Between a node crashing and a replacement Pod becoming Ready, there's a real window where actual state (2 running Pods) doesn't match desired state (3) — this is expected and acceptable; the guarantee is that the gap keeps shrinking and eventually closes, not that it never opens.
Why Kubernetes is composed of many independent, narrow loops
Rather than one monolithic "make everything correct" process, Kubernetes splits reconciliation across many separate controllers, each responsible for one object type: the Deployment controller manages ReplicaSets (creating a new one and scaling the old one down during a rolling update); the ReplicaSet controller manages Pods; the Node controller watches node health; the Endpoint controller keeps Service endpoint lists in sync with matching Pods. Each loop is simple and independently understandable, and this decomposition is exactly what makes the system extensible — a Custom Resource Definition plus an Operator (see that topic) is just adding a new controller that reconciles a new kind of object, using the exact same underlying pattern as every built-in controller.
Why this matters more than it might first appear
Understanding the control-loop pattern explains why Kubernetes behaves the way it does in situations that otherwise seem surprising: why manually editing a Pod created by a Deployment gets silently overwritten (the Deployment's controller reconciles it back to match the template), why deleting a Pod managed by a ReplicaSet just causes a replacement to appear (the controller notices the gap and closes it), and why kubectl apply is safe to re-run repeatedly (each run is just one more observation feeding the same convergence process). A candidate who can explain a surprising Kubernetes behavior by tracing it back to "some controller's reconciliation loop did that" demonstrates real conceptual understanding, not just command memorization.