This post walks the full picture: why the original Endpoints API had to be replaced, what is actually inside a slice, the three conditions that decide whether traffic flows to a pod, how zone-aware routing works through topology hints, who watches these things, and a real cluster demo at the bottom. Every claim is verified against kubernetes/kubernetes 1.36 source and CoreDNS 1.11.

https://youtu.be/_MJ1ou-Oj-s?si=Zi9vHbI_Bf_snVlo

TL;DR

1. Why slices exist. The legacy Endpoints object held every backend pod IP in one blob. Three thousand pods meant a giant object, watched by every kube-proxy on every node, rewritten on every change. It did not scale. EndpointSlices (GA in 1.21) replaced it with many small objects capped at 100 endpoints each.

2. What is in a slice. An addressType (IPv4 or IPv6, plus an unused FQDN), a list of endpoints with conditions and a targetRef pointing back at the pod, a list of ports, and labels that bind it to its parent Service.

3. Three conditions. Serving tracks the readiness probe. Terminating flips during pod deletion. Ready is the convenience flag — shorthand for Serving=true AND Terminating=false. The split exists so kube-proxy can keep using a draining pod that still answers requests.

4. Topology hints. Each endpoint can carry a hints.forZones field. kube-proxy on a node in us-east-1a prefers endpoints hinted for that zone. Same-zone routing means lower latency and lower cross-zone traffic costs.

5. Two watchers. kube-proxy and CoreDNS both watch EndpointSlices via the discovery/v1 API. kube-proxy reprograms iptables. CoreDNS uses them to answer Headless-Service lookups directly with pod IPs.

Part 1: Why EndpointSlices exist

To understand why EndpointSlices exist, look at what came before.

There was just one object per Service. The Endpoints object. One blob, with every backend pod IP inside it. Three pods? Fine. Three thousand pods? That object becomes huge. Every kube-proxy on every node watches it. Every change rewrites the entire blob. Every node re-receives the entire blob. The api-server chokes. kube-proxy chokes. The system did not scale.

The fix landed in Kubernetes 1.16 (alpha) and went GA in 1.21. Instead of one giant Endpoints object, you get many small EndpointSlice objects. Each one is capped at 100 endpoints by default. Add a pod, only the slice it lands in gets rewritten. Watchers only re-process that one slice. The api-server only ships one small object on the wire. Linear scaling instead of quadratic blow-up.

The original Endpoints API is officially deprecated as of 1.33 (KEP-4974). It still gets written for backward compatibility, but EndpointSlices are the source of truth now, and modern controllers read slices, not the legacy object.

Part 2: What is actually inside a slice

When you create a Service, the EndpointSlice controller wakes up. It runs inside kube-controller-manager. It watches Services. It watches Pod changes. When either changes, it reconciles.

For each pod in the namespace that matches the Service's label selector and is Ready, the controller builds a slice entry: pod IP, pod node, ports, conditions. Then it writes or updates one or more EndpointSlice objects. The api-server validates them and writes them to etcd. kube-proxy and CoreDNS watch them via the api-server. Done.

A trimmed kubectl describe endpointslice looks like this:

Name:         my-service-x29k9
Labels:       kubernetes.io/service-name=my-service
              endpointslice.kubernetes.io/managed-by=endpointslice-controller.k8s.io
AddressType:  IPv4
Ports:
  Name    Port  Protocol
  http    80    TCP
Endpoints:
  - Addresses:  10.244.1.42
    Conditions:
      Ready:        true
      Serving:      true
      Terminating:  false
    TargetRef:    Pod/nginx-abc-123
    NodeName:     worker
  - Addresses:  10.244.2.18
    Conditions:
      Ready:        true
      Serving:      true
      Terminating:  false
    TargetRef:    Pod/nginx-abc-456
    NodeName:     worker2

Five fields are doing all the work:

• addressType. IPv4 or IPv6. Dual-stack Services get two slices, one for each protocol. There's a third value, FQDN, in the API enum, but no Kubernetes component implements behavior for it. We'll come back to this.

• endpoints[]. The list. Each entry has addresses (the pod IP), conditions (the three flags we'll cover next), targetRef (a pointer back at the source Pod), and nodeName (used by topology routing).

• ports[]. The Service's exposed ports, copied here so kube-proxy doesn't have to cross-reference.

• labels. kubernetes.io/service-name is the binding label that tells every consumer which Service this slice belongs to. The managed-by label distinguishes slices written by the core controller from slices written by other controllers (Cilium, Antrea, MCS-API).

• ownerReferences. Points at the parent Service so deletion cascades naturally.

Why 100?

100 endpoints per slice is the default cap. It is a deliberate tradeoff.

Smaller slices means more API objects in etcd but fewer bytes per watch event. Bigger slices means fewer objects but every change rewrites a larger blob. 100 sits in the middle. A Service with 3 pods has 1 slice. A Service with 300 pods has 3. A Service with 3000 pods has 30. The cap is configurable up to 1000 via the --max-endpoints-per-slice flag on kube-controller-manager, but most clusters never need to touch it.

Part 3: The three conditions

Each endpoint has three condition flags. Ready. Serving. Terminating. They sound similar. They mean different things. And they decide whether traffic actually goes to that pod.

Serving

Serving tracks the readiness probe. If the pod's readiness probe passes, Serving = true. The pod is willing and able to handle requests. kube-proxy can route traffic here.

This is the underlying truth flag. The other two are derived from it.

Terminating

Terminating = true means the pod is being deleted. The Kubernetes controller observes the pod's deletionTimestamp and flips this flag.

kube-proxy normally stops sending new connections to a Terminating endpoint, so the pod can drain in peace. But there's a safety fallback: if every endpoint for a Service is Terminating, kube-proxy keeps routing traffic to them anyway, rather than dropping connections. This is the graceful-shutdown safeguard that stops a class of zero-downtime rollout bugs where the old pods are gone before the new pods are ready.

Ready

Ready is the convenience flag. It is shorthand for Serving = true AND Terminating = false. A normal, accepting-traffic, not-being-deleted pod has Ready = true.

The reason the two flags are split: a pod can be Serving but not Ready, during graceful shutdown. It still answers requests, but it's being drained. This split was added so kube-proxy can make a smart decision: don't pick this pod for new connections, but keep any in-flight ones going. Without Serving as a separate flag, draining pods would be a much rougher transition.

Part 4: AddressType, and the FQDN edge case

The API defines three address types:

// k8s.io/api/discovery/v1/types.go
const (
    AddressTypeIPv4 = AddressType(v1.IPv4Protocol)
    AddressTypeIPv6 = AddressType(v1.IPv6Protocol)
    AddressTypeFQDN = AddressType("FQDN")
)

In practice, you only ever see two. Dual-stack Services produce two slices, one with addressType: IPv4 and one with addressType: IPv6. Same Service, same conditions, different protocol.

FQDN is a constant the API exposes. No core Kubernetes component implements behavior for it. The same types.go file states it directly:

"The syntax and semantics of other addressType values are not defined. This must contain at least one address but no more than 100. EndpointSlices generated by the EndpointSlice controller will always have exactly 1 address."

So FQDN is reserved space in the API. The EndpointSlice controller never produces it. kube-proxy doesn't program rules for it. CoreDNS doesn't resolve from it. If you create one manually, nothing happens.

It's there for hypothetical extensions: a controller could write FQDN-typed slices for things like external services, and a custom data plane could consume them. In a default cluster, treat the AddressType field as a binary IPv4-or-IPv6 flag.

Part 5: Topology hints (zone-aware routing)

This is the part most people don't know about.

Each endpoint can carry a hints field with forZones — a list of zones this endpoint is preferred for. kube-proxy reads these hints. If a pod is in us-east-1a, kube-proxy on a node in us-east-1a prefers endpoints hinted for that zone. Same-zone routing. Lower latency. Lower cross-zone egress costs, which on AWS / GCP can be a real bill line item.

Where hints come from

The EndpointSlice controller writes them. It reads the pod's node label topology.kubernetes.io/zone. If the Service has the annotation:

metadata:
  annotations:
    service.kubernetes.io/topology-mode: Auto

then the controller computes hints automatically, balancing endpoints across zones, weighted by each zone's allocatable CPU. The goal: each zone serves traffic proportional to its compute capacity. If us-east-1a has 60% of the cluster's CPU, it gets ~60% of the endpoints hinted for that zone.

Manual hints are also possible via the same annotation set to specific zone names, or via custom controllers that write hints directly. The auto mode is what most clusters use.

When hints don't fire

Auto mode has safety bailouts. If the zone distribution is too unbalanced — say one zone has all the pods — the controller drops hints entirely rather than route everything to that zone. kube-proxy without hints falls back to its normal round-robin behavior. This is intentional: same-zone routing is only safe when every zone has enough capacity to handle its own traffic.

Part 6: Who watches EndpointSlices

Two consumers. Both watch the same API.

kube-proxy

kube-proxy runs as a DaemonSet on every node. It has two informers: one on Services, one on EndpointSlices. When a slice changes, it diffs the new state against the kernel's current rules, then batches the changes into a single iptables-restore (or nftables, since 1.33). The whole rule set is replaced as one transaction.

Three nodes, three independent reprograms, all in parallel. Total time on a normal cluster: milliseconds per change.

CoreDNS

CoreDNS watches EndpointSlices through its kubernetes plugin. From coredns/plugin/kubernetes/controller.go:

import discovery "k8s.io/api/discovery/v1"

epLister, epController := object.NewIndexerInformer(
    &cache.ListWatch{
        ListFunc:  endpointSliceListFunc(...),
        WatchFunc: endpointSliceWatchFunc(...),
    },
    &discovery.EndpointSlice{},
    ...
)

For a normal Service, CoreDNS returns the ClusterIP — it doesn't actually need the slice to answer that query. The slices matter for Headless Services (clusterIP: None). For those, CoreDNS returns the pod IPs from the slice directly, no virtual IP, no kube-proxy in the path. This is how StatefulSets get per-pod DNS: pod-0.my-statefulset.default.svc.cluster.local resolves to one specific pod IP, pulled out of the slice.

Both watchers are list-watch. They get the initial state from a List, then incremental ADDED / MODIFIED / DELETED events via the watch stream. No polling.

Part 7: A real cluster demo

Let me show this on a real cluster. Three workers, a kind v1.35.1 cluster:

$ kubectl get nodes
NAME                 STATUS   ROLES           AGE
kind-control-plane   Ready    control-plane   2m
kind-worker          Ready    <none>          2m
kind-worker2         Ready    <none>          2m
kind-worker3         Ready    <none>          2m

Apply a Deployment with three nginx pods and a Service:

$ kubectl create deployment nginx --image=nginx --replicas=3
$ kubectl expose deployment nginx --port=80

Three pods

$ kubectl get endpointslices -l kubernetes.io/service-name=nginx
NAME          ADDRESSTYPE   PORTS   ENDPOINTS                                     AGE
nginx-x29k9   IPv4          80      10.244.1.42,10.244.2.18,10.244.3.07           5s

One slice. Three endpoints. Each pod IP, each node, each Ready. Exactly what we expect for three pods.

Worth noticing: the slice itself was actually created the moment the Service was, as an empty placeholder. As each pod transitioned to Ready, the controller filled it in and updated the same slice object. The placeholder pattern means downstream watchers always have something to bind to, even before any pod is ready.

Scale to 50

$ kubectl scale deployment nginx --replicas=50

Watch the slice:

$ kubectl get endpointslices -l kubernetes.io/service-name=nginx
NAME          ADDRESSTYPE   PORTS   ENDPOINTS                                     AGE
nginx-x29k9   IPv4          80      10.244.1.42,10.244.1.43,... (50 entries)      2m

Still one slice. Fifty endpoints. Under the 100-endpoint default cap. kube-proxy on every node reprogrammed. CoreDNS, if this were a Headless Service, would now return all fifty IPs in the DNS answer.

Scale to 200 — the slice splits

$ kubectl scale deployment nginx --replicas=200

$ kubectl get endpointslices -l kubernetes.io/service-name=nginx
NAME          ADDRESSTYPE   PORTS   ENDPOINTS                       AGE
nginx-x29k9   IPv4          80      ... (100 entries)               3m
nginx-fz4mp   IPv4          80      ... (100 entries)               12s

Two slices. One with 100 endpoints. One with another 100. kube-proxy reads both, stitches the rules together, and the Service has two hundred backends. The split is invisible to anyone using the Service. You curl one ClusterIP and the kernel picks one of the 200 pods.

Rolling update

Now do a rolling update — change the image, watch the slices in real time:

$ kubectl set image deployment nginx nginx=nginx:1.27
$ kubectl get endpointslices -l kubernetes.io/service-name=nginx --watch

You see endpoints transition through every state we covered. Pods go Terminating = true. New pods land Serving = true once readiness passes. Slices update. kube-proxy reprograms on every change.

Total reprogramming time across the rolling update? A few seconds, distributed across nodes. The Service IP never changes. Traffic keeps flowing. End users see nothing.

Wrap

So that is the EndpointSlice. The bridge between a Service definition and the kernel rules that route real traffic.

One controller writes them. kube-proxy and CoreDNS watch them. Capped at 100 by default. Conditions for graceful shutdown. Topology hints for zone-aware routing. The placeholder pattern so downstream watchers always have something to bind to.

It's what makes Kubernetes Services scale to thousands of pods without falling over. Most people use Services every day and never think about the slices underneath. That's the point — the abstraction works.

NVCF powers infrastructure behind services like build.nvidia.com and NVIDIA-hosted inference workflows across GPU cloud providers and DGX Cloud environments.. Now you can run the whole thing yourself and read every line that makes it work.

Let’s break down how the platform actually works.

What NVCF Actually Is

NVCF stands for NVIDIA Cloud Functions. The original managed service let you register a Docker container or Helm chart, specify a GPU type, and NVIDIA handled everything: routing, queueing, autoscaling, multi-tenant isolation. GPU cloud partners like CoreWeave ran the NVIDIA Cluster Agent on their Kubernetes clusters so their GPUs could serve functions while NVIDIA owned the control plane.

The April 2026 Apache 2.0 release publishes that control plane. The previous repos (NVIDIA/nvidia-cloud-functions, NVIDIA/nvcf-go) are now archived. This monorepo is the one place everything lives.

One honest caveat: the control plane images are currently distributed via NVIDIA's NGC registry under the nvcf-onprem org. You need NGC access to deploy the full stack today. The source code is all Apache 2.0 and inspectable, but the deployable bundle still goes through NGC while issue #12 (full OSS build) is open. I opened issue #14 asking for a community contributor path.

Three-Plane Architecture

The entire platform is built around three independently scalable planes connected through NATS JetStream.

Control Plane runs on a dedicated Kubernetes cluster and owns function lifecycle, autoscaling decisions, and secrets management. Key services:

• function-autoscaler (Rust): runs a 30-second scaling loop, reads utilization from VictoriaMetrics, writes decisions to Cassandra, calls the NVCF API to set desired instance counts

• helm-reval (Go): validates OCI-referenced Helm charts before the compute plane deploys them

• OpenBao (Apache 2.0 Vault fork): all function secrets encrypted at rest, injected at runtime via the ess-agent sidecar

• Cassandra: persistent state and distributed locks for the autoscaler

Invocation Plane sits between every caller and every GPU worker. Nothing bypasses it:

• http-invocation (Rust / Axum): receives HTTP/gRPC requests, publishes to NATS JetStream, handles async polling

• llm-gateway (Go): OpenAI-compatible API with token-aware rate limiting via embedded Olric cache

• grpc-proxy (Go): forwards gRPC calls to function instances

• ratelimiter (Go): per-function rate limiting using Olric distributed cache

• nats-auth-callout (Go): NATS authentication with NKey, OIDC, and webhook strategies

Compute Plane is one NVCA (NVIDIA Cluster Agent) operator per GPU cluster. NVCA registers the cluster with the control plane, consumes NATS messages, and manages pod lifecycle.

How a Single Request Flows

Every invocation follows this path verified from the source code:

1. Caller posts to POST /v2/nvcf/pexec/functions/{id}

2. http-invocation checks rate via ratelimiter gRPC

3. Request published to NATS stream: Create.NVCA.*.{clusterID}.*.* (from nvca/pkg/queue/nats/client.go)

4. NVCA queue manager consumes the message

5. ICMSRequest Kubernetes CR created (deduplication by NATS sequence)

6. MiniService controller reconciles: creates Pod or applies Helm chart

7. Function pod connects back via WorkerService gRPC: ConnectOnce

8. Response returns to the caller

9. On completion: Terminate.NVCA.{clusterID} triggers pod deletion and GC

Scale-to-Zero: The NATS Buffer Approach

This is the most important architectural decision in the whole codebase, and it is fundamentally different from how Knative handles scale-to-zero.

With Knative, requests can experience timeout or retry pressure during long scale-up events, especially for GPU workloads with heavy cold starts. That model works well for lightweight stateless HTTP services that initialize quickly. GPU inference workloads are different. Loading large models into VRAM can take tens of seconds or even minutes, making durable request buffering much more important.

NVCF uses NATS JetStream as a durable request buffer:

1. Autoscaler drives desired instance count to 0. No pods running.

2. New request arrives. Published to NATS JetStream. Stream persists it durably.

3. Autoscaler detects queue depth > 0. Sets desired instances to 1+.

4. NVCA receives creation message, launches pod.

5. Pod connects via WorkerService gRPC, pulls the buffered message.

6. Response returns through the still-open http-invocation connection.

The request is never dropped. The caller waits longer on a cold start, but the request completes. This is only possible because the queue buffers it.

Multi-Cluster by Design

Each GPU cluster runs its own NVCA. NATS JetStream subjects are scoped per cluster:

Creation:     Create.NVCA.*.{clusterID}.*.*
Termination:  Terminate.NVCA.{clusterID}
Consumer:     {streamName}-{clusterID}  (durable, per cluster)

A single control plane can manage GPU clusters across on-prem H100s, cloud H200s, GB200 NVLink nodes, and cloud provider partners simultaneously. The invocation plane routes based on the function deployment specification.

Setting Up Locally (What Works Without NGC Access)

You can bootstrap the cluster and fake GPU layer without NGC credentials. The NVCF services deployment is what requires the nvcf-onprem org access.

# Clone the repo
git clone https://github.com/nvidia/nvcf
cd nvcf/examples/self-hosted-local-development

# Bootstrap k3d cluster (6 nodes) + KWOK + fake GPU operator
# This works fully without NGC
./setup.sh

# Verify fake H100 nodes are registered
kubectl get nodes -l run.ai/simulated-gpu-node-pool=default
# NAME               STATUS   ROLES    GPU
# k3d-nvcf-agent-3   Ready    <none>   8x NVIDIA-H100-80GB-HBM3
# k3d-nvcf-agent-4   Ready    <none>   8x NVIDIA-H100-80GB-HBM3

# Deploy NVCF stack (requires NGC nvcf-onprem access)
helm registry login nvcr.io -u '\(oauthtoken' -p "\){NGC_API_KEY}"
HELMFILE_ENV=local helmfile sync

The fake GPU operator from run-ai/fake-gpu-operator adds nvidia.com/gpu extended resources to real Kubernetes nodes. Pods schedule and run. CUDA calls fail since there is no real GPU, but all NVCF orchestration, NATS dispatch, and scale-to-zero logic works exactly as in production.

How NVCF Compares

KubeRay and NVCF are not competitors. You should be able to run Ray Serve under KubeRay as a NVCF function.

What the Open Source Release Actually Changes

Inspectability. Enterprises can now validate NVIDIA’s architectural decisions directly from the source code instead of treating the platform as a black box.

Customization. You can modify the autoscaler Rust loop, add NATS auth strategies, extend the MiniService controller, or build new CLI commands. Earlier, these internals were largely inaccessible outside NVIDIA-managed environments.

Links

• Repo: github.com/nvidia/nvcf

• Docs: docs.nvidia.com/nvcf/overview

• My PR: NVIDIA/nvcf#13

• NGC access issue: NVIDIA/nvcf#14

• Contribute / contact team: nvcf-interest@nvidia.com

You can't. That's the problem Services solve, and the answer is more interesting than "Kubernetes assigns a stable IP."

This post walks the full picture in five parts: why Services have to exist, what happens when you create one, what happens when traffic actually calls one, all five Service types (most posts stop at three), and a real cluster demo at the bottom. Every claim is verified against kubernetes/kubernetes 1.36 source.

TL;DR

1. Why Services exist. Pods are ephemeral; their IPs change on every restart and reschedule. A Service is a stable identity for an unstable set of pods.
2. Creation flow. kubectl expose → API server allocates a ClusterIP from --service-cluster-ip-range (since 1.33, backed by IPAddress objects, GA) → EndpointSlice controller fills the slice with matching pod IPs → kube-proxy on every node programs iptables → CoreDNS adds the name. Sub-second end to end.
3. Request flow. curl my-service → CoreDNS returns ClusterIP → packet hits KUBE-SERVICES → matches KUBE-SVC-XXX → --mode random picks one rule → KUBE-SEP-YYY does DNAT to a pod IP → kernel routes to the backend → conntrack rewrites the source IP back to ClusterIP on the reply.
4. Five Service types. ClusterIP (internal), NodePort (every-node port, dev/test), LoadBalancer (cloud-provided external LB for prod), ExternalName (DNS CNAME alias for managed services), Headless (clusterIP: None, DNS returns pod IPs directly — for StatefulSets).
5. The dataplane is the kernel. Three controllers cooperate to write iptables rules, but the actual packet forwarding is pure Linux netfilter. The Service object is metadata; the kernel does the work.

Part 1: Why Services exist

The Kubernetes pod model is deliberately ephemeral. Pods get rescheduled, restarted, scaled up and down. Pod IPs change every time. There is no permanent address.

This is by design. Kubernetes treats pods as cattle, not pets. The system is at its best when no individual pod is precious and the whole workload can be reshuffled across the cluster. But applications need to talk to each other, and applications need a stable target.

A Service is the answer. It is a stable identity for an unstable set of pods. You give it a label selector (app: nginx). Any pod that matches the selector becomes a backend. The Service exposes one virtual IP, called the ClusterIP. Pods come and go. The ClusterIP stays.

Without this abstraction, you would be writing service discovery code from scratch in every application, forever. Every microservice architecture in Kubernetes depends on this idea. It is so fundamental that it is easy to miss how clever it is.

Part 2: What happens when you create a Service

One command:

$ kubectl expose deployment nginx --port=80
service/nginx exposed

Behind that one line, an entire pipeline of controllers wakes up.

Step 1: The API server allocates a ClusterIP

The API server receives the create request, runs admission, runs validation. Then it allocates a ClusterIP from the configured --service-cluster-ip-range (typically 10.96.0.0/12).

Since Kubernetes 1.33, the IP allocator is backed by first-class IPAddress and ServiceCIDR objects. You can run:

$ kubectl get ipaddresses
NAME           PARENTREF
10.96.255.20   services/default/my-service

This used to be a bitmap stored in etcd. Now it's a clean API. The Service object gets the allocated IP stamped onto spec.clusterIP, the write goes to etcd via Raft, and the Service exists. At this point, no pod is connected to it yet. The Service is just metadata.

Step 2: The EndpointSlice controller fills the slice

The EndpointSlice controller runs inside kube-controller-manager. It has two informers: one watches Services, one watches Pods. When a new Service appears, the controller scans every pod in the namespace. For each pod that matches the selector AND is Ready, it builds a slice entry: pod IP, pod node, ports, conditions.

The output is one or more EndpointSlice objects, capped at 100 endpoints each. A Service with three pods has one slice. A Service with three thousand pods has thirty.

Each slice has an ownerReference to the Service (so deletion cascades) and labels:

• kubernetes.io/service-name: my-service
• endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io

That's how kube-proxy knows which Service a slice belongs to.

Side note worth knowing: the legacy Endpoints API was officially deprecated in 1.33 (KEP-4974). It still works for older controllers, but EndpointSlices are the source of truth now.

Step 3: kube-proxy on every node reprograms

There is a kube-proxy pod on every node, deployed as a DaemonSet in nearly every installer. It has Service and EndpointSlice informers. When a slice changes, kube-proxy diffs the new state against the kernel's current rules, then batches the changes into a single iptables-restore call. The whole rule set is replaced as one transaction.

Three nodes, three independent reprograms, all in parallel. Total time on a normal cluster: milliseconds.

Step 4: What gets programmed

The actual rules look like this (real captured output from a kind 1.35.1 cluster, slightly trimmed):

-A KUBE-SERVICES -d 10.96.255.20/32 -p tcp --dport 80 -j KUBE-SVC-FXIYY
-A KUBE-SVC-FXIYY -m statistic --mode random --probability 0.333 -j KUBE-SEP-1
-A KUBE-SVC-FXIYY -m statistic --mode random --probability 0.500 -j KUBE-SEP-2
-A KUBE-SVC-FXIYY -j KUBE-SEP-3
-A KUBE-SEP-1 -j DNAT --to-destination 10.244.1.42:80
-A KUBE-SEP-2 -j DNAT --to-destination 10.244.2.18:80
-A KUBE-SEP-3 -j DNAT --to-destination 10.244.3.07:80

KUBE-SERVICES is the entry chain. Every Service port has a match rule that jumps to a per-Service chain, named KUBE-SVC- plus a hash of the service name. That chain has one rule per backend, with a --mode random --probability 1/n declining pattern. Each backend rule jumps to a per-endpoint KUBE-SEP- chain that does the actual DNAT.

iptables is the default mode. nftables (GA in 1.33) uses verdict maps for sub-microsecond hash lookup instead of the linear scan; recommended for modern Linux clusters. IPVS mode was deprecated in 1.35 and is now legacy.

Step 5: CoreDNS adds the name

There's a Service called kube-dns in the kube-system namespace, backed by CoreDNS pods. Every pod's /etc/resolv.conf has the kube-dns ClusterIP as its nameserver. CoreDNS has a kubernetes plugin that watches Services. When my-service appears, CoreDNS now resolves my-service.default.svc.cluster.local to the ClusterIP.

Three controllers cooperated, plus DNS, and the Service is live. Total time from kubectl expose to first traffic flowing: under a second.

Part 3: What happens when traffic calls a Service

kubectl exec into a busybox pod, run curl my-service. Two seconds later: <title>Welcome to nginx!</title>. Now slow that down to seven steps.

Step 1: DNS resolution

The pod sees curl my-service, but my-service is not a real hostname. The pod's resolver consults its search list (default.svc.cluster.local, svc.cluster.local, cluster.local). It tries my-service.default.svc.cluster.local first. The query goes to CoreDNS, which has the kubernetes plugin watching Services, and returns the ClusterIP.

Step 2: TCP packet to ClusterIP

The pod opens a TCP connection. SYN packet, destination ClusterIP, port 80. The packet leaves the pod's veth pair, enters the host's network namespace. PREROUTING.

Step 3: KUBE-SERVICES matches

The packet hits the KUBE-SERVICES chain. Every Service port has a match rule here. The rule says: destination is 10.96.255.20, port 80? Jump to KUBE-SVC-FXIYY. Now the packet is inside the per-Service chain.

Step 4: --mode random picks one

KUBE-SVC-FXIYY has one rule per backend, with the declining-probability pattern:

• Rule 1 fires with probability 1/3 → KUBE-SEP-1
• Rule 2 fires with probability 1/2 of what's left → KUBE-SEP-2
• Rule 3 is the unconditional fallthrough → KUBE-SEP-3

The math works out: each backend gets exactly one third of the traffic. iptables -m statistic --mode random is the underlying mechanism.

Step 5: DNAT rewrites the destination

The chosen rule jumps to KUBE-SEP-2. That chain has one rule:

-j DNAT --to-destination 10.244.2.18:80

The kernel rewrites the destination of the packet from ClusterIP to the actual pod IP. The packet is no longer for the virtual address. It is now headed at a real pod, on a real node.

Step 6: Backend receives

The packet routes to the backend pod's node, traverses the CNI bridge or overlay, arrives at the backend. The pod sees a normal TCP packet to its own IP, port 80. It sends back a SYN-ACK. The backend has no idea it was reached via a Service abstraction.

Step 7: Reply rewriting via conntrack

The reply traffic is where the trick happens. Source IP is the backend pod. Destination is the original sender. But the Linux conntrack table remembers the DNAT we did on the way in. So when the reply comes back through the host, conntrack rewrites the source IP from the backend pod, back to the ClusterIP.

The original sender pod sees the response coming from the ClusterIP, exactly the address it sent the packet to. Connection works. End to end. The pod has no idea this dance happened.

Part 4: All five Service types (with real use cases)

Most "Service types" content stops at three. There are five, and each one solves a different problem.

1. ClusterIP

The default. Virtual IP, internal only. This is what we just walked through.

Use case: application-to-application traffic inside the cluster. Frontend talks to backend. Microservice A calls microservice B. Both inside the cluster. By far the most common type.

2. NodePort

NodePort opens the same port on every node in the cluster, in the range 30000–32767. Traffic to any node on that port gets forwarded to the ClusterIP, and from there to a pod.

Use case: local development clusters like kind or minikube, where you don't have a cloud load balancer to provision. NodePort is also a building block — LoadBalancer Services use it under the hood.

3. LoadBalancer

This is what you use in production for public-facing apps. When you create a LoadBalancer Service on EKS, GKE, or AKS, the cloud provider integration provisions a real external load balancer and points it at the NodePort on each node. You get a public IP. Real users hit it.

Use case: production-facing web apps. The browser-to-cluster ingress path.

4. ExternalName

This is the type most people skip. ExternalName has no ClusterIP, no selector, no pods. It's a DNS CNAME alias inside the cluster.

apiVersion: v1
kind: Service
metadata:
  name: my-database
spec:
  type: ExternalName
  externalName: prod-db.us-east-1.rds.amazonaws.com

Now any pod can curl my-database, and the cluster DNS returns the AWS hostname.

Use case: pointing in-cluster names at managed external services. Your apps look up my-database, the underlying address is a managed Postgres in RDS, and you can swap the target without changing application code. Same pattern works for managed Redis, S3-compatible stores, anything external.

5. Headless

apiVersion: v1
kind: Service
metadata:
  name: cassandra-svc
spec:
  clusterIP: None       # ← the magic
  selector:
    app: cassandra

There is no virtual IP, no kube-proxy involvement, no DNAT. Instead, DNS returns the IPs of all backend pods directly. With a StatefulSet, each pod also gets a stable DNS name like cassandra-0.cassandra-svc.default.

Use case: StatefulSets. Each pod gets a stable DNS name for peer discovery in distributed systems (Cassandra nodes finding each other, Kafka brokers, etcd peers). Custom client-side load balancing where the client wants to choose the backend itself, not have iptables choose. Anything that needs to talk to specific pods, not a load-balanced abstraction.

The summary

Most teams over-use LoadBalancer when ClusterIP plus an Ingress would do. Pick the right one for the job.

Part 5: Live demo

To make all of this concrete, we ran the create-and-call flow on a real cluster (kind 1.35.1, three workers). What follows are verbatim outputs.

$ kubectl get nodes
NAME                         STATUS   ROLES           AGE   VERSION
service-demo-control-plane   Ready    control-plane   24s   v1.35.1
service-demo-worker          Ready    <none>          14s   v1.35.1
service-demo-worker2         Ready    <none>          14s   v1.35.1
service-demo-worker3         Ready    <none>          14s   v1.35.1

Apply an nginx Deployment with three replicas, then a Service:

$ kubectl apply -f nginx-deploy.yaml
deployment.apps/nginx created

$ kubectl get pods -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP           NODE
nginx-fd956d49d-49779   1/1     Running   0          12s   10.244.2.2   service-demo-worker2
nginx-fd956d49d-5pbsm   1/1     Running   0          12s   10.244.1.2   service-demo-worker
nginx-fd956d49d-g94jr   1/1     Running   0          12s   10.244.3.2   service-demo-worker3

$ kubectl apply -f my-service.yaml
service/my-service created

$ kubectl get svc my-service
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
my-service   ClusterIP   10.96.255.20   <none>        80/TCP    2s

The Service has no IPs in its spec — only the selector:

$ kubectl get svc my-service -o yaml | grep -A 8 spec:
spec:
  clusterIP: 10.96.255.20
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - port: 80
  selector:
    app: nginx

The 1.33+ IPAddress object shows it as a first-class allocation:

$ kubectl get ipaddresses
NAME           PARENTREF
10.96.255.20   services/default/my-service

The EndpointSlice has the real backend IPs:

$ kubectl get endpointslices -l kubernetes.io/service-name=my-service
NAME               ADDRESSTYPE   PORTS   ENDPOINTS                            AGE
my-service-x29k9   IPv4          80      10.244.1.2,10.244.2.2,10.244.3.2     8s

And the iptables rules on a worker show the chain we described:

$ docker exec service-demo-worker iptables-save | grep my-service
-A KUBE-SERVICES -d 10.96.255.20/32 -p tcp -m tcp --dport 80 -j KUBE-SVC-FXIYY6OHUSNBITIX
-A KUBE-SVC-FXIYY6OHUSNBITIX -m statistic --mode random --probability 0.33333333349 -j KUBE-SEP-4B2TTHBRUYTSCT32
-A KUBE-SVC-FXIYY6OHUSNBITIX -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-FAW7RO5CDYGWP4Y3
-A KUBE-SVC-FXIYY6OHUSNBITIX -j KUBE-SEP-4UWZBYSYCGDXTWU5
-A KUBE-SEP-4B2TTHBRUYTSCT32 -j DNAT --to-destination 10.244.1.2:80
-A KUBE-SEP-FAW7RO5CDYGWP4Y3 -j DNAT --to-destination 10.244.2.2:80
-A KUBE-SEP-4UWZBYSYCGDXTWU5 -j DNAT --to-destination 10.244.3.2:80

Real ClusterIP. Real chain hash. Real probabilities. Real pod IPs. The math (0.33333, 0.50000, fallthrough) is exactly what we derived earlier.

Now curl from inside the cluster:

$ kubectl run curl --rm -i --restart=Never --image=busybox:1.36 -- wget -q -O - http://my-service
<!DOCTYPE html>
<html>
<head><title>Welcome to nginx!</title></head>
...

Welcome to nginx. The pod knew nothing about iptables, KUBE-SVC chains, DNAT, or conntrack. It just curled my-service and got a response.

Scale up to ten replicas, watch everything reprogram in real time:

$ kubectl scale deploy/nginx --replicas=10
deployment.apps/nginx scaled

$ kubectl get endpointslices -l kubernetes.io/service-name=my-service
NAME               ADDRESSTYPE   PORTS   ENDPOINTS                                      AGE
my-service-x29k9   IPv4          80      10.244.1.2,10.244.2.2,10.244.3.2 + 7 more...   1m42s

$ docker exec service-demo-worker iptables-save | grep -c 'KUBE-SVC-FXIYY.*statistic'
9

Three rules became nine. (Tenth backend is the unconditional fallthrough, no --probability on it.) From kubectl scale to all kube-proxies reprogrammed: about 200 milliseconds.

Three takeaways

1. The Service object is metadata. The dataplane is the kernel. Nothing in the Service object knows about pod IPs. The kernel's iptables (or nftables) rules carry that mapping. When you understand this, you stop thinking of Services as magic and start thinking of them as cleverly placed netfilter rules.

2. EndpointSlice is the bridge. When a Pod becomes Ready, the EndpointSlice controller writes its IP. kube-proxy reads. The kernel obeys. Three controllers, no shared state, all eventually consistent — and reprogramming completes in milliseconds even on big clusters.

3. Use the right Service type for the job. ClusterIP for internal traffic. NodePort for dev clusters. LoadBalancer for public production. ExternalName to alias external services. Headless for StatefulSets. Most teams over-use LoadBalancer when ClusterIP-plus-Ingress would have been the right choice.

Where to go from here

The full video walks the 5-part flow in 10 minutes with animated visuals for each step. Link at the top of this post.

Sources for every claim in this post:

• pkg/registry/core/service/storage/alloc.go — ClusterIP allocator
• pkg/controller/endpointslice/ — EndpointSlice controller
• pkg/proxy/iptables/proxier.go — kube-proxy iptables rules
• pkg/proxy/nftables/ — nftables backend (GA 1.33)
• KEP-1880 — MultiCIDRServiceAllocator (ServiceCIDR objects)
• KEP-3866 — kube-proxy nftables backend
• KEP-4974 — Endpoints API deprecation
• The terminal output above is verbatim from a real Kubernetes 1.35.1 kind cluster, captured for this post.

7 Days of Docker (2026) - The Finale. A Docker Captain's guide. Not your average tutorial.

Your container is probably not safe to ship.

I know that sounds harsh after six days of building. You have images, Dockerfiles, multi-stage builds, Compose stacks - real skills. But here is the uncomfortable truth: everything we built in Days 1 through 6 was optimized for working. Today we optimize for not getting breached.

This is the part of Docker that separates side projects from production systems. The part most tutorials skip entirely. The part I wish someone had drilled into me before I shipped my first container to a real cluster.

Let's fix every container you have ever built.

Your Container Runs as Root. That's a Problem.

By default, every Docker container runs as root. Not "sort of root." Not "sandboxed root." Root. UID 0. The same identity that owns every file on a Linux system.

Why does this matter? Because container isolation is not perfect. Kernel exploits exist. Misconfigurations happen. If an attacker escapes a container running as root, they land on the host as root. Game over. Your entire machine - every other container, every secret, every volume - is theirs.

The fix is embarrassingly simple:

$ docker run --rm --user 1000:1000 alpine id
uid=1000 gid=1000 groups=1000

That is it. One flag. The process now runs as an unprivileged user. In production Dockerfiles, you bake this in permanently:

FROM node:20-alpine
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser
CMD ["node", "server.js"]

After USER appuser, every subsequent instruction and the runtime process itself run as that user. No root. No exceptions.

What Nobody Tells You: "But my app needs to bind to port 80!" No, it does not. Bind to port 3000 or 8080 inside the container, then map it to whatever port you want on the host with -p 80:3000. Needing port 80 inside the container is not a reason to run as root - it is a sign your architecture needs rethinking.

Read-Only Filesystem: If They Can't Write, They Can't Attack

A writable filesystem inside a container means an attacker can drop malware, modify binaries, or plant a reverse shell. A read-only filesystem shuts all of that down:

$ docker run --rm --read-only alpine sh -c "echo test > /file"
sh: can't create /file: Read-only file system

The write was blocked by the kernel. Nothing gets through. But your app probably needs to write somewhere - temp files, logs, PID files. The answer is --tmpfs, which mounts a writable in-memory filesystem at a specific path:

docker run --rm --read-only --tmpfs /tmp alpine sh -c "echo test > /tmp/file && echo 'OK'"

The root filesystem stays immutable. /tmp is writable but exists only in RAM and vanishes when the container stops. Attackers cannot persist anything.

Resource Limits: Because Runaway Containers Kill Hosts

Without limits, a single container with a memory leak or a fork bomb can starve every other process on the machine - including your other containers. Docker uses Linux cgroups to enforce hard boundaries:

$ docker run --rm --memory=128m alpine cat /sys/fs/cgroup/memory.max
134217728

That is exactly 128 MB (128 x 1024 x 1024 = 134,217,728 bytes). If the process exceeds this, the kernel OOM-kills it. Not "warns it." Kills it. That is the correct behavior - a container that violates its resource contract should die, not drag down the host.

No New Privileges and Capability Dropping

Two more flags that should be on every production container:

--security-opt=no-new-privileges prevents any process inside the container from gaining additional privileges through setuid/setgid binaries. Even if an attacker finds a SUID binary, they cannot escalate.

--cap-drop=ALL strips every Linux capability. By default, Docker grants containers a broad set - enough to do things like change file ownership, bind to privileged ports, and manipulate network interfaces. Drop them all, then add back only what you need:

docker run --rm \
  --cap-drop=ALL \
  --cap-add=NET_BIND_SERVICE \
  --security-opt=no-new-privileges \
  myapp

This container can bind to privileged ports and nothing else. No changing file ownership. No raw network access. No kernel module loading. The attack surface shrinks dramatically.

The Production Run Template

Putting it all together, here is what a hardened docker run looks like:

docker run -d --name myapp \
  --user 1000:1000 \
  --read-only \
  --tmpfs /tmp \
  --memory=128m \
  --cpus=0.5 \
  --pids-limit=50 \
  --cap-drop=ALL \
  --security-opt=no-new-privileges \
  --restart=unless-stopped \
  -p 8080:3000 \
  myapp:v1.2.3

Every flag is a security layer. Remove any one of them and you open a hole. This is defense in depth - the principle that no single control is enough, but all of them together make a breach extremely difficult.

Docker Scout as a CI Gate

Security hardening at runtime is half the battle. The other half is knowing what vulnerabilities are baked into your image before you ship it. Docker Scout scans your images for known CVEs and gives you a blunt summary:

$ docker scout quickview nginx:alpine
 Target             │  nginx:alpine            │    0C     2H     10M     2L     1?
   digest           │  7f7dcd27f920            │
 Base image         │  nginx:1-alpine-slim     │    0C     0H     1M     0L
 Updated base image │  nginx:1.30-alpine-slim  │    0C     0H     1M     0L

Read that output. A freshly-pulled nginx:alpine - patched days ago - still has 2 high and 10 medium vulnerabilities. That is before you add a single line of your own code. Now imagine an image you haven't updated in 3 months. Or 6. The numbers explode.

The image you pick matters more than most of the code you write on top of it.

In CI/CD, Scout becomes a gate. If the scan finds critical or high vulnerabilities, the pipeline fails and the image never reaches production:

# GitHub Actions - scan before push
- name: Scan for vulnerabilities
  run: |
    docker scout cves myapp:${{ github.sha }} \
      --exit-code \
      --only-severity critical,high

The --exit-code flag makes Scout return a non-zero exit code when vulnerabilities are found. Your pipeline stops. No humans required.

2026's Supply Chain Wake-Up Call

In March 2026, attackers compromised Aqua Security's CI/CD pipeline and pushed backdoored Trivy scanner images to Docker Hub. The tool meant to FIND vulnerabilities became the attack vector. Thousands of CI/CD pipelines had secrets stolen.

Then in April 2026, it happened again. Attackers used stolen Checkmarx publisher credentials to push malicious images to checkmarx/kics on Docker Hub. Docker caught and quarantined it, but the pattern was clear: a stolen credential, a push through the normal publishing flow, and the attacker is inside the supply chain of every organization that pulls that tag.

This is not hypothetical. This is happening now. And it changes how you should think about every docker pull and every FROM line in your Dockerfiles.

Docker Hardened Images: Fewer CVEs by Default

Docker recognized that the base image problem is systemic. If every nginx, node, and python image ships with dozens of known vulnerabilities, every developer inherits those vulnerabilities whether they know it or not.

The answer is Docker Hardened Images - curated, continuously patched base images with up to 95% fewer CVEs than their standard counterparts. They are free, licensed under Apache 2.0, and available on Docker Hub. Same functionality, dramatically smaller attack surface.

Instead of pulling nginx:alpine with its 58 known vulnerabilities and hoping for the best, you pull the hardened equivalent and start from a clean baseline. Docker maintains these images with rapid CVE patching - when a vulnerability is disclosed, the hardened image is rebuilt and pushed within hours, not weeks.

Docker Hardened Images (DHI) are not just "fewer CVEs." They are architecturally different:

• Built from source by Docker with verified provenance

• Signed releases produced through a hardened build pipeline

• Rootless - no root user, ever

• Distroless runtime - stripped of shells, package managers, everything an attacker needs

• VEX attestations - machine-readable statements about which CVEs actually affect the image at runtime

• 7-day fix guarantee - new CVE disclosed? Patched image pushed within 7 days

• Free and open source under Apache 2.0. 1,000+ images available.

Here's why DHI matters for the Trivy/KICS attack pattern: those attacks worked because a valid publisher credential could push any tag. With DHI, images are built by Docker from source - the provenance and signatures must match the upstream source, or the image doesn't ship. The attack pattern structurally can't work against DHI.

What Nobody Tells You: Most Docker tutorials teach you to pick a base image by size. "Use Alpine, it's small!" Size matters, but CVE count matters more. A 5 MB image with 3 critical vulnerabilities is worse than a 50 MB image with zero. Check docker scout quickview before you write your FROM line. Every time.

Scout Policies and Attestations

Docker Scout isn't just quickview anymore. In 2026, it has a full policy engine:

• Configurable policies - set severity thresholds, define what "compliant" means for your org

• Supply chain attestations policy - flags images that lack SBOM or provenance attestations

• CI gate - fail pipelines when policies are violated, not just when CVEs exist

• SBOM generation - CycloneDX and SPDX formats, attached as attestations to your images

# GitHub Actions - policy-based gate, not just CVE count
- name: Check Scout policies
  run: |
    docker scout policy myapp:${{ github.sha }} \
      --exit-code \
      --org my-org

This is a shift from "scan and hope" to "define policy, enforce automatically."

Docker Sandboxes: Isolating AI Agents

If you're running AI coding agents (Claude Code, Codex, Copilot, Gemini CLI), Day 6's Model Runner gives them intelligence. But intelligence without guardrails is dangerous.

Docker Sandboxes run each agent session inside a dedicated MicroVM - not a container, an actual virtual machine with its own kernel and private Docker daemon. The agent can clone repos, run tests, build images, execute arbitrary code - all inside a disposable environment with no path back to your host.

brew install docker/tap/sbx

This is not "a container with extra flags." It's VM-grade isolation:

• Each session gets its own kernel (hardware boundary, not just namespace boundary)

• Private Docker daemon inside the MicroVM (no socket mounting, no host privileges)

• File access, network policies, and secrets defined before the agent runs

• Disposable by design - delete and start fresh in seconds

What Nobody Tells You: An LLM deciding its own security boundaries is not a security model. If your agent's system prompt says "don't delete files" - that's a suggestion, not enforcement. The bounding box has to come from infrastructure, not from a prompt. That's what Sandboxes provide.

Your 7-Day Journey

Let's look back at how far you have come.

Day 1: Containers are processes. Not VMs. Not sandboxes. Linux processes restricted by namespaces (what they can see) and cgroups (what they can use). One kernel, many isolated process trees. This mental model underpins everything.

Day 2: Images are supply chain artifacts. Layers, manifests, content-addressed digests. Tags are mutable pointers; digests are immutable truth. Your image is a graph of filesystem snapshots, and every dependency in that graph is a potential attack vector.

Day 3: Dockerfiles, docker init, and multi-stage builds. You stopped writing Dockerfiles from scratch and started generating them. Multi-stage builds cut your image sizes by 10x by separating build dependencies from runtime.

Day 4: Breaking isolation on purpose. Volumes break filesystem isolation (data survives containers). Networks break network isolation (containers talk by name via DNS). Port mapping breaks host isolation (the outside world reaches in). One coherent story about carefully poking holes in container isolation.

Day 5: Docker Compose for real workflowsor real workflows. One YAML file, one command, an entire application stack. Services, networks, volumes, dependency ordering - all declarative, all version-controlled. Nobody types docker run with 15 flags in real life.

Day 6: Docker + AI.er + AI. You pulled an LLM from Docker Hub, ran it locally with Metal GPU acceleration, hit it with an OpenAI-compatible API, and built a real AI-powered Flask app with Compose. Docker Model Runner, Gordon, MCP Toolkit - Docker is now an AI development platform.

Day 7: Production security. Non-root users, read-only filesystems, resource limits, capability dropping, vulnerability scanning, hardened images. The difference between "it runs" and "it's safe to ship."

You went from docker run hello-world to hardened, scanned, resource-limited production containers in seven days. That is not trivial. That is a real foundation.

Where Docker Ends

Docker builds containers. It does not run them at scale. And that is fine - it was never meant to.

Kubernetes is where containers go when they outgrow a single machine. Pod scheduling, auto-scaling, rolling deployments, service meshes, ingress controllers - that is K8s territory. Start with kind or minikube locally, then move to managed services (EKS, GKE, AKS). Everything you learned about images, Dockerfiles, and Compose translates directly. K8s runs the same OCI images you have been building all week.

For Kubernetes, it's easy, we have multiple courses

https://youtu.be/EV47Oxwet6Y?si=Hw69DkBc3MolinUy

Testcontainers is where Docker meets testing. Spin up real databases, message brokers, and services as throwaway containers inside your test suite. Your integration tests run against real infrastructure, not mocks. It uses Docker under the hood, so everything you know applies.

Apple Containers is Apple's take on container runtime - one lightweight VM per container instead of Docker's shared-VM model. Sub-second boot times, stronger isolation, written in Swift. It is interesting for macOS-native workflows, but Docker remains the standard for production and CI/CD.

What Nobody Tells You: Docker is not competing with Kubernetes. Docker builds. Kubernetes runs at scale. They are complementary tools, not alternatives. Every K8s cluster runs Docker-built images. Every CI/CD pipeline that feeds K8s uses Docker to build and scan. Learn both. In that order. Docker first - because you cannot orchestrate what you cannot build.

The Production Checklist

Before you ship any container, run through this:

[ ] Non-root USER in Dockerfile
[ ] --read-only filesystem + --tmpfs where needed
[ ] --memory and --cpus limits set
[ ] --cap-drop=ALL + selective --cap-add
[ ] --security-opt=no-new-privileges
[ ] docker scout quickview - zero critical, zero high
[ ] Hardened base image or patched base image
[ ] HEALTHCHECK in Dockerfile
[ ] Log rotation configured (--log-opt max-size, max-file)
[ ] Image tagged with git SHA, not just :latest
[ ] Secrets in mounted files or external vault - never in ENV or layers

If you cannot check every box, you are not ready to ship. Go back and fix it. The checklist is not optional - it is the minimum.

Now Go Build Something

Seven days ago, you did not know what a container was. Today, you can build one that is hardened enough to face the internet.

That is a real skill. Not a certificate. Not a badge. A skill - the kind that shows up when you push to production at 2 AM and everything works because you built it right the first time.

Docker is not the destination. It is the foundation. Everything that comes next - Kubernetes, service meshes, GitOps, platform engineering - is built on top of what you learned this week.

Now go build something amazing. And when you do, build it in a container.

This concludes the 7 Days of Docker (2026) series. If you enjoyed learning in a new way, then do follow me on X and connect with me on LinkedIn.

Do share the articles and pointers you loved the most on socials and tag me.

7 Days of Docker (2026) - A Docker Captain's guide. Not your average tutorial.

I'm a Docker Captain. And if you'd told me two years ago that I'd be pulling AI models from Docker Hub the same way I pull nginx, I would've laughed.

I'm not laughing anymore.

Docker shipped something called Model Runner. It lets you pull, run, and serve Large Language Models locally - no Python environment, no conda, no CUDA drivers, no dependency hell. One command. The model runs on your hardware with GPU acceleration. And it exposes an OpenAI-compatible API that any app can talk to.

Today we're going to pull a model, talk to it, build a real app that uses it, containerize that app, and deploy the whole thing with Compose. By the end of this post, you'll have a working AI-powered API running on your laptop. No cloud. No API keys. No monthly bill.

Pull a Model Like You Pull an Image

docker model pull ai/smollm2

Using cached model: 256.35 MiB

That's it. 256 megabytes. A small but capable language model, pulled from Docker Hub using the same infrastructure that serves container images. Same content-addressable storage, same caching.

Check what you have:

docker model list

MODEL NAME  PARAMETERS  QUANTIZATION    ARCHITECTURE  MODEL ID      SIZE
smollm2     361.82 M    IQ2_XXS/Q4_K_M  llama         354bf30d0aa3  256.35 MiB

Look familiar? Same format as docker images. Model name, ID, size.

Talk to It

docker model run ai/smollm2 "What is Kubernetes in one sentence?"

Kubernetes is a container orchestration platform that automates the deployment,
scaling, and management of microservices-based applications.

That ran locally. On my Mac. With Metal GPU acceleration. No internet required after the initial pull.

Check the runner status:

docker model status

Docker Model Runner is running
llama.cpp: running llama.cpp latest-metal e365e65

latest-metal means it's using Apple Silicon GPU acceleration via Metal. On Linux with NVIDIA, you'd see a CUDA tag. Docker picks the right backend automatically.

Model Runner supports multiple inference backends:

Want MLX models? Install vllm-metal

llama.cpp + Metal is the default and handles GGUF models from Docker Hub. But if you want to run MLX models - Apple's native ML framework, designed for Apple Silicon's unified memory architecture - you can install the vllm-metal backend:

docker model install-runner --backend vllm

Installing vllm backend...
vllm backend installed successfully

Check the status - both backends now running:

BACKEND    STATUS         DETAILS
llama.cpp  Running        llama.cpp latest-metal e365e65
vllm       Running        vllm-metal v0.1.0-20260320-122309
diffusers  Not Installed
mlx        Not Installed  package not installed

MLX models live on Hugging Face (not Docker Hub). Pull one:

docker model pull hf.co/mlx-community/Llama-3.2-1B-Instruct-4bit

The same API (localhost:12434/v1/) serves both backends - Docker routes to the right one based on model format.

Watch out: You might see docker model install-runner --backend mlx --gpu metal suggested online or even by Gordon (Docker's AI assistant). On Docker Desktop, this fails with "Standalone installation not supported." The correct command is --backend vllm, which installs vllm-metal on Mac automatically. The mlx flag is for standalone Docker Engine only.

Here's the same question hitting both backends on my M2 - same API, same endpoint, different models:

Here's the same question hitting both backends on my M2 - same API, same endpoint:

# llama.cpp backend - SmolLM2 (362M params, GGUF from Docker Hub)
curl localhost:12434/v1/chat/completions \
  -d '{"model":"ai/smollm2", "messages":[{"role":"user","content":"What is a Docker container?"}]}'

"A Docker container is a lightweight, isolated, and self-contained runtime
environment that encapsulates an application and its dependencies."
74 tokens

# vllm-metal backend - Llama 3.2 1B (1B params, MLX from Hugging Face)
curl localhost:12434/v1/chat/completions \
  -d '{"model":"hf.co/mlx-community/Llama-3.2-1B-Instruct-4bit", ...}'

"A Docker container is a lightweight, fully virtualized, and managed package
that performs a consistent version of an application, allowing it to be easily
deployed, scaled, and managed across multiple hosts and environments."
84 tokens

Two different models, two different backends, one API. Your app code doesn't change when you switch.

What Nobody Tells You: Model Runner doesn't run models inside containers. It runs them directly on your host hardware - Metal on Mac, CUDA on Linux. The llama.cpp process is a native binary on your host, not inside a container namespace. Why? Performance. LLMs need direct GPU access. Container isolation adds overhead. Docker's role here is distribution (pull from Hub) and API (OpenAI-compatible endpoint). The container is the app that CALLS the model, not the model itself.

The API - This Changes Everything

Model Runner exposes an OpenAI-compatible API. Two endpoints depending on where you're calling from:

This is important. model-runner.docker.internal is Docker's internal DNS - it only resolves from inside containers. From your Mac, use localhost:12434.

Try it:

curl http://localhost:12434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai/smollm2",
    "messages": [{"role": "user", "content": "Explain Docker volumes in 2 sentences"}],
    "max_tokens": 60
  }'

{
  "choices": [{
    "message": {
      "role": "assistant",
      "content": "A Docker volume is a data container that allows you to mount a file system onto other files or directories. It enables you to use volumes as a source or destination for files within your Docker container, improving the portability and extensibility of your applications."
    }
  }],
  "usage": {
    "prompt_tokens": 37,
    "completion_tokens": 51,
    "total_tokens": 88
  }
}

That's the standard OpenAI chat completions format. Switch localhost:12434 to api.openai.com, add an API key, and the same request hits GPT-4. Your code doesn't care which backend it talks to. Local model for dev, cloud for prod. Same interface.

What Nobody Tells You: If curl fails with "Could not resolve host: model-runner.docker.internal" - you're running it from your Mac, not from inside a container. Use localhost:12434 from the host. This trips up everyone the first time.

Build a Real AI App

Enough theory. Let's build an AI-powered API, containerize it, and serve it with Compose.

The App (app.py)

from flask import Flask, request, jsonify
from openai import OpenAI
import os

app = Flask(__name__)
client = OpenAI(
    base_url=os.environ.get("LLM_URL", "http://model-runner.docker.internal/v1/"),
    api_key="not-needed"
)

@app.route("/")
def home():
    return jsonify({"service": "AI Demo", "model": "smollm2"})

@app.route("/ask")
def ask():
    question = request.args.get("q", "What is Docker?")
    response = client.chat.completions.create(
        model="ai/smollm2",
        messages=[{"role": "user", "content": question}],
        max_tokens=100
    )
    return jsonify({
        "question": question,
        "answer": response.choices[0].message.content,
        "tokens": response.usage.total_tokens
    })

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

Notice base_url reads from an environment variable. From inside a container, it'll use the Docker internal hostname. The api_key is "not-needed" because Model Runner doesn't require auth.

The Dockerfile

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
EXPOSE 5000
CMD ["python", "app.py"]

The requirements.txt

flask==3.1.1
openai==1.82.0

compose.yaml

services:
  ai-app:
    build: .
    ports:
      - "5002:5000"
    environment:
      - LLM_URL=http://model-runner.docker.internal/v1/

The LLM_URL environment variable tells the container to use the Docker-internal endpoint.

Run It

docker compose up -d --build

 Image d6-ai-app-ai-app Built
 Network d6-ai-app_default Created
 Container d6-ai-app-ai-app-1 Started

Test It

curl http://localhost:5002/

{"model": "smollm2", "service": "AI Demo"}

curl "http://localhost:5002/ask?q=What+is+a+container+in+one+sentence"

{
  "answer": "A container is a lightweight package that runs applications and provides a controlled environment for the application's dependencies, allowing for easier deployment and scaling.",
  "question": "What is a container in one sentence",
  "tokens": 65
}

That's a containerized Flask app, calling a local LLM via Docker Model Runner, returning AI-generated answers. No cloud API. No API key. No monthly bill. Running on your laptop.

docker compose down

What Else Is New - The Quick Version

Model Runner is the headline, but Docker shipped more AI tooling in 2026:

Gordon (docker ai) - Docker's built-in AI assistant. It reads your project - Dockerfiles, Compose files, running containers - and gives context-specific answers. Not generic ChatGPT. It sees your actual environment.

docker ai "Why is my container using so much memory?"

Note: If you have many MCP servers configured, Gordon may error with "too many tools." Disable unused MCP servers in Docker Desktop settings to fix it.

MCP Toolkit (docker mcp) - Model Context Protocol is a standard for connecting AI agents to tools. Docker runs MCP servers inside isolated containers with restricted permissions. Think of it as a security layer between AI agents and your system.

Docker Scout - Already covered in Day 2, but worth repeating: scan your AI app images too. AI dependencies (PyTorch, transformers, etc.) are massive and often carry CVEs.

Docker Sandboxes - Run AI agents inside dedicated MicroVMs with their own kernel and private Docker daemon. Not containers - actual VM-grade isolation. Each agent session gets a disposable environment where it can clone repos, run tests, and build images without any path back to your host. Works with Claude Code, Codex, Copilot, and others. Install with brew install docker/tap/sbx.

Docker's AI story in 2026 goes well beyond "run containers." It's Model Runner for local inference, Gordon for context-aware assistance, MCP for secure tool access, Sandboxes for agent isolation, and Scout for supply chain security. All shipping today.

The Big Picture

Docker in 2026 is two things:

1. A container platform (Days 1-5) - build, ship, run applications

2. An AI development platform (Day 6) - pull models, run local inference, build AI apps

The second part is new. And it's growing fast. Docker Hub already hosts Llama, Mistral, Phi, Gemma, SmolLM, and others in GGUF format with various quantization levels.

The OpenAI-compatible API is the killer feature. Write your app against the OpenAI interface. During development, point it at localhost:12434 - free, fast, private. In production, swap to the real OpenAI API or any other compatible provider. Your code doesn't change.

Quick Reference

Tomorrow: Day 7

You just built an AI-powered app with Docker. From docker model pull to a working API in minutes.

Tomorrow is the finale. Day 7: Ship It. We take everything from the past 6 days and make it production-ready. Non-root users. Read-only filesystems. Resource limits. Security scanning. The checklist that separates side projects from production systems.

error: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1beta1"

…or the classic gke-gcloud-auth-plugin: executable not found.

That's because the generated kubeconfig doesn't actually contain a credential. It contains an exec: block that shells out to gke-gcloud-auth-plugin, which in turn calls gcloud to mint a fresh OAuth token on every kubectl call. If you look at the users section of a stock GKE kubeconfig, this is what's in there:

users:
- name: gke_saiyam-project_us-east1-b_demo-test
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      command: gke-gcloud-auth-plugin
      installHint: Install gke-gcloud-auth-plugin for use with kubectl by following
        https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl#install_plugin
      interactiveMode: IfAvailable
      provideClusterInfo: true

No token. No cert. Just "run this plugin and ask it for auth." No gcloud on the machine, no access.

If you want a kubeconfig that anyone can use — a CI runner, a contractor's laptop, a script on a VM — you need to swap that exec-plugin auth for something self-contained. The cleanest answer: a Kubernetes ServiceAccount and a bearer token.

Here's the full flow, run end-to-end against a live GKE cluster.

The mental model

Four pieces, in order:

1. Identity — a ServiceAccount in the cluster
2. Permissions — a (Cluster)RoleBinding attaching a role to that SA
3. Credential — a token the SA can present to the API server
4. Portable config — a kubeconfig file wrapping the token + cluster endpoint + CA cert

The API server validates the token itself. No Google, no gcloud, no OAuth round-trip.

Step 1: Identity and permissions

kubectl create serviceaccount shared-access -n kube-system

kubectl create clusterrolebinding shared-access-binding \
  --clusterrole=cluster-admin \
  --serviceaccount=kube-system:shared-access

Output:

serviceaccount/shared-access created
clusterrolebinding.rbac.authorization.k8s.io/shared-access-binding created

Two things worth calling out:

• The SA lives in kube-system because it's a cluster-wide utility identity. The namespace doesn't restrict its access — RBAC does.
• cluster-admin is * on *. Scope it down in production. view, edit, or a custom ClusterRole are usually what you actually want. If you only need namespace-scoped access, use a RoleBinding in that namespace instead of a ClusterRoleBinding.

Step 2: Mint a long-lived token

Before Kubernetes 1.24, creating a ServiceAccount automatically created a companion Secret with a non-expiring token. That was removed — long-lived bearer tokens are a security footgun — so now you opt in explicitly:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: shared-access-token
  namespace: kube-system
  annotations:
    kubernetes.io/service-account.name: shared-access
type: kubernetes.io/service-account-token
EOF

Output:

secret/shared-access-token created

The magic is in two fields:

• type: kubernetes.io/service-account-token — tells the token controller (built into kube-controller-manager) "I'm a Secret you should populate."
• kubernetes.io/service-account.name annotation — tells it which ServiceAccount's identity to embed in the token.

Wait a couple of seconds, then inspect the Secret — the controller has filled in the data for you:

kubectl get secret shared-access-token -n kube-system -o yaml

apiVersion: v1
data:
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVMVENDQXBXZ0F3SUJB...
  namespace: a3ViZS1zeXN0ZW0=
  token: ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklrWnNZMkk0VFRkWmFrVjN...
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: shared-access
    kubernetes.io/service-account.uid: 9e8d4bdb-46ea-4893-9306-d56bea6aa304
  name: shared-access-token
  namespace: kube-system
type: kubernetes.io/service-account-token

Three fields got populated by the controller:

• .data.token — a signed JWT, the actual bearer credential
• .data.ca.crt — the cluster's CA certificate (so your client can trust the API server's TLS)
• .data.namespace — the SA's namespace

If you'd rather have a short-lived token, skip the Secret and run kubectl create token shared-access -n kube-system --duration=24h. Good for automation that rotates. Bad for a "hand someone a file" use case, which is what we're doing here.

Step 3: Extract the three things a kubeconfig needs

SERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
CA=$(kubectl get secret shared-access-token -n kube-system -o jsonpath='{.data.ca\.crt}')
TOKEN=$(kubectl get secret shared-access-token -n kube-system -o jsonpath='{.data.token}' | base64 -d)

echo "SERVER = ${SERVER}"
echo "CA     = ${CA:0:60}..."
echo "TOKEN  = ${TOKEN:0:40}..."

Output:

SERVER = https://35.196.129.174
CA     = LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0WERQWERk1JSUVMVENDQXBXZ0F3SUJB...
TOKEN  = eyJhbGciOiJSUzIsImtpZCI6IkZsY2I4TTdZ...

• SERVER — the GKE API endpoint, pulled straight from your current context
• CA — already base64, drops straight into the kubeconfig as-is
• TOKEN — we decode it because kubeconfig wants the raw JWT string, not base64

Step 4: Assemble the kubeconfig

cat > /tmp/shared-kubeconfig.yaml <<EOF
apiVersion: v1
kind: Config
clusters:
- name: cluster-1
  cluster:
    server: ${SERVER}
    certificate-authority-data: ${CA}
contexts:
- name: cluster-1
  context:
    cluster: cluster-1
    user: shared-access
current-context: cluster-1
users:
- name: shared-access
  user:
    token: ${TOKEN}
EOF

A kubeconfig is three independent lists — clusters, users, contexts — glued together by a context that names one cluster + one user. Nothing more.

Notice what's not in the users block: no auth-provider, no exec. kubectl has nothing to shell out to. It just sends Authorization: Bearer <token> on every request and the API server validates the JWT.

Step 5: Prove it works without gcloud

KUBECONFIG=/tmp/shared-kubeconfig.yaml kubectl get nodes
KUBECONFIG=/tmp/shared-kubeconfig.yaml kubectl auth whoami
KUBECONFIG=/tmp/shared-kubeconfig.yaml kubectl auth can-i '*' '*' --all-namespaces

Output:

NAME                                       STATUS   ROLES    AGE   VERSION
gke-demo-test-default-pool-a5aaa3f4-jcnk   Ready    <none>   18h   v1.35.1-gke.1396002

ATTRIBUTE   VALUE
Username    system:serviceaccount:kube-system:shared-access
UID         9e8d4bdb-46ea-4893-9306-d56bea6aa304
Groups      [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated]

yes

That's the whole proof. The API server sees system:serviceaccount:kube-system:shared-access, not your Google identity. You can put this file on a machine that has never seen gcloud in its life, and it works.

Things to know before you ship this

Private clusters still need network reachability. The kubeconfig removes the auth dependency, not the network one. If your control plane is private, the recipient still needs VPN, authorized networks, or a public endpoint. The token won't help if they can't reach the API server.

The kubeconfig is a credential. Anyone with the file has whatever RBAC you bound. Store it like you'd store an SSH key or an API token. Don't commit it to Git.

Revocation is deletion. To kill access, delete the Secret:

kubectl delete secret shared-access-token -n kube-system

To kill it harder, also delete the binding and the SA. There's no "rotate" — you mint a new Secret and redistribute the new kubeconfig.

Scope down. cluster-admin is the demo default, not the production default. A RoleBinding to edit in a single namespace is usually closer to what a real sharing use case needs. ClusterRoleBinding + cluster-admin only when you truly mean it.

Wrap

The trick isn't really about GKE — it's about understanding what a kubeconfig is. Once you see it as a glue file between a cluster endpoint and any credential the API server will accept, the exec-plugin auth stops feeling magical and the bearer-token swap becomes obvious.

Same approach works for EKS (where the plugin is aws-iam-authenticator / aws eks get-token), AKS (kubelogin), and anything else that ships exec-based auth. Replace the user: block, keep the cluster: block, and you've got a kubeconfig that travels.

7 Days of Docker in 2026 - From docker run Chaos to Declarative Stacks

Nobody types docker run with 15 flags in real life.

I’ve been learning and working with Docker for some time now. I’ve explored different setups, experimented a lot, and seen how teams actually use it beyond tutorials. And one thing becomes very clear: the moment you move past basic demos and into real development, you stop writing docker run commands by hand. You write a Compose file.

Everything you learned on Days 1 through 4 - images, Dockerfiles, volumes, container basics was preparation. Today is the day you learn how Docker actually gets used on real teams, in real codebases, every single day. Docker Compose takes all those individual concepts and wires them into one declarative file that anyone on your team can run with a single command.

Table of Contents

• 1. What Is Docker Compose?

• 2. The Compose File — Anatomy of a Stack

• 3. Hands-On: Flask + Redis Visit Counter

• 4. Essential Compose Commands

• 5. 2026 Features You Should Know

• 6. What Nobody Tells You

• 7. Quick Reference

• Key Takeaways

• What's Next: Day 6

1. What Is Docker Compose?

Docker Compose is a declarative tool for defining and running multi-container applications. You describe your entire stack - services, networks, volumes, environment variables - in a single YAML file. Then you run one command, and everything comes up.

Consider a Flask web app backed by Redis. Without Compose, your startup ritual looks like this:

docker network create myapp-net
docker volume create redis-data
docker run -d --name redis --network myapp-net -v redis-data:/data redis:alpine
docker build -t myapp-web .
docker run -d --name web --network myapp-net -p 5001:5000 -e REDIS_HOST=redis myapp-web

Five commands, a dozen flags. And you have not even added health checks, restart policies, or teardown instructions. Now imagine six services instead of two. Now imagine onboarding a new developer.

With Compose, all of that becomes:

docker compose up

One file. One command. Every developer on the team gets the exact same stack.

Important: Docker Compose is not a separate install anymore. Since Docker Desktop 4.x and the Compose plugin for Docker Engine, the command is docker compose (no hyphen). The old docker-compose binary is legacy. Use docker compose - always.

Let me verify we are on the same page:

docker compose version

Docker Compose version v5.0.1

Good. Let's build something.

2. The Compose File — Anatomy of a Stack

The Compose file is named compose.yaml (the older docker-compose.yml still works, but compose.yaml is the modern standard). Here is the file we will use for our hands-on exercise, fully annotated:

services:            # Required: every container in your stack
  web:               # Service name — also becomes the DNS hostname
    build: .         # Build from the Dockerfile in the current directory
    ports:
      - "5001:5000"  # Map host port 5001 to container port 5000
    environment:
      - REDIS_HOST=redis
    depends_on:
      - redis        # Start redis before web

  redis:             # Second service — a Redis server
    image: redis:alpine     # Use a prebuilt image (no build needed)
    volumes:
      - redis-data:/data    # Persist Redis data to a named volume

volumes:             # Top-level: declares named volumes
  redis-data:        # Docker manages this volume's entire lifecycle

That is the entire definition for a two-service application with persistent storage. Let me break down the key sections.

services is the core of every Compose file. Each key (web, redis) becomes a running container. Critically, each service name also becomes a DNS hostname on the Compose network. When web connects to redis:6379Docker resolves that name automatically.

build tells Compose to build an image from a Dockerfile. A dot (.) means "use the Dockerfile in the current directory."

image tells Compose to pull a prebuilt image from a registry. A service uses build, image, or both.

ports maps host ports to container ports. Format: "HOST:CONTAINER". Only expose what you need to access from outside Docker.

volumes at the service level, mounts storage into the container. At the top level, it declares named volumes that Docker manages and persists across restarts.

depends_on controls startup order. Redis starts before the web app. In production setups, you would pair this with condition: service_healthy a healthcheck, but for development, the simple form works fine.

3. Hands-On: Flask + Redis Visit Counter

Time to build a real multi-container application. A Flask web app that counts visits, backed by Redis.

Project Structure

d5-compose/
├── app.py
├── Dockerfile
├── requirements.txt
└── compose.yaml

The Flask Application (app.py)

from flask import Flask, jsonify
import redis
import os

app = Flask(__name__)

r = redis.Redis(
    host=os.environ.get("REDIS_HOST", "redis"),
    port=int(os.environ.get("REDIS_PORT", 6379)),
    decode_responses=True
)

@app.route("/")
def home():
    count = r.incr("visits")
    return jsonify(visits=count)

@app.route("/health")
def health():
    try:
        r.ping()
        return jsonify(status="healthy", redis="connected"), 200
    except redis.ConnectionError:
        return jsonify(status="unhealthy", redis="disconnected"), 503

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

Two endpoints. The root / increments a counter in Redis and returns it. The /health endpoint verifies Redis connectivity. Simple, testable, real.

The Dockerfile

FROM python:3.13-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

EXPOSE 5000

CMD ["python", "app.py"]

requirements.txt

flask==3.1.1
redis==5.3.0

The Compose File (compose.yaml)

services:
  web:
    build: .
    ports:
      - "5001:5000"
    environment:
      - REDIS_HOST=redis
    depends_on:
      - redis

  redis:
    image: redis:alpine
    volumes:
      - redis-data:/data

volumes:
  redis-data:

Bring It Up

docker compose up --build -d

[+] Building 11.8s (9/9) FINISHED
 => [web internal] load build definition from Dockerfile
 => [web] FROM python:3.13-slim
 => [web] COPY requirements.txt .
 => [web] RUN pip install --no-cache-dir -r requirements.txt
 => [web] COPY app.py .
 => [web] exporting to image
[+] Running 4/4
 ✔ Volume "d5-compose_redis-data"  Created
 ✔ Network d5-compose_default      Created
 ✔ Container d5-compose-redis-1    Started
 ✔ Container d5-compose-web-1      Started

Read that output. Compose did four things:

1. Built the web image from the Dockerfile.

2. Created a volume called d5-compose_redis-data for Redis persistence.

3. Created a network called d5-compose_default and attached both services.

4. Started containers in dependency order — Redis first, then web.

The naming convention is <project>_<resource> for networks and volumes, <project>-<service>-<n> for containers. The project name defaults to the directory name.

Test It

curl http://localhost:5001

{"visits":1}

curl http://localhost:5001

{"visits":2}

curl http://localhost:5001

{"visits":3}

The counter increments with every request. The data lives in Redis, not in the Flask process, so it persists across app restarts. This is how real applications work -stateless compute, stateful storage.

Inspect the Running Stack

docker compose ps

NAME                 IMAGE            SERVICE   STATUS         PORTS
d5-compose-redis-1   redis:alpine     redis     Up 4 seconds   6379/tcp
d5-compose-web-1     d5-compose-web   web       Up 4 seconds   0.0.0.0:5001->5000/tcp

Both containers running. The web service is mapped to port 5001. Redis exposes 6379 internally to the Compose network but is not mapped to the host — exactly right. Your database should never be directly reachable from outside.

Tear It All Down

docker compose down -v

[+] Running 4/4
 ✔ Container d5-compose-web-1      Removed
 ✔ Container d5-compose-redis-1    Removed
 ✔ Volume d5-compose_redis-data    Removed
 ✔ Network d5-compose_default      Removed

One command. Containers, volume, network — all cleaned up. The -v flag removes named volumes, too. Without it, volumes persist so your data survives rebuilds. In development, I use down -v constantly to start fresh. In staging, keep the volumes.

4. Essential Compose Commands

These are the five commands you will use every day. Learn them well.

docker compose up

# Foreground (logs stream to terminal)
docker compose up

# Detached (background)
docker compose up -d

# Rebuild images before starting (use after code changes)
docker compose up --build

# Start only specific services
docker compose up redis

docker compose down

# Remove containers + networks
docker compose down

# Also remove volumes (wipes data!)
docker compose down -v

# Also remove built images
docker compose down --rmi all

docker compose ps

NAME                 IMAGE            COMMAND                  SERVICE   CREATED              STATUS              PORTS
dockerday5-redis-1   redis:alpine     "docker-entrypoint.s…"   redis     About a minute ago   Up About a minute   6379/tcp

dockerday5-web-1     dockerday5-web   "python app.py"          web       About a minute ago   Up About a minute   0.0.0.0:5001->5000/tcp, [::]:5001->5000/tcp

Lists running services, their status, and port mappings. Your first command when debugging.

docker compose logs

# All services
docker compose logs

# Follow a specific service in real time
docker compose logs -f web

# Last 50 lines from all services
docker compose logs --tail 50

docker compose exec

# Open a shell in a running container
docker compose exec web sh

# Run a one-off command
docker compose exec redis redis-cli GET visits

Pro Tip: My actual development loop is: docker compose up -d --build, hack on code, check docker compose logs -f web, repeat. When things get weird, docker compose down -v && docker compose up -d --build gives you a clean slate in seconds.

5. 2026 Features You Should Know

Compose has evolved significantly. If you learned it a few years ago, you are missing some genuinely useful capabilities.

Compose Watch (Hot Reload)

This is the feature that changed my development workflow. Instead of rebuilding images after every code change, Compose Watch monitors your files and syncs changes directly into running containers.

services:
  web:
    build: .
    develop:
      watch:
        - action: sync
          path: ./app.py
          target: /app/app.py
        - action: rebuild
          path: ./requirements.txt

docker compose watch

WARN[0000] No services to build                         

[+] up 3/3

 ✔ Network dockerday5_default   Created                                                            0.1s 

 ✔ Container dockerday5-redis-1 Created                                                            0.1s 

 ✔ Container dockerday5-web-1   Created                                                            0.1s 

none of the selected services is configured for watch, consider setting a 'develop' section

Now edit app.py and save. The file is synced into the container instantly — no rebuild, no restart. Change requirements.txt and Compose triggers a full rebuild automatically because dependencies changed.

The develop section supports two actions:

• sync — copies files into the container. Use for source code.

• rebuild — triggers a full docker compose up --build. Use for dependency files.

This is better than bind mounts for most use cases because it works consistently across macOS, Linux, and Windows, without the filesystem performance issues that plague bind mounts on Mac.

Profiles

Profiles let you define optional services that only start when you explicitly activate them. This is how you handle dev/test/prod variations in a single file.

services:
  web:
    build: .
    ports: ["5001:5000"]

  redis:
    image: redis:alpine

  test-runner:
    build: .
    command: pytest
    profiles: [test]

  debug-tools:
    image: nicolaka/netshoot
    profiles: [debug]

# Normal development — only web and redis start
docker compose up -d

# Run tests — includes the test-runner service
docker compose --profile test up

# Debug networking — includes netshoot
docker compose --profile debug up

Services without a profiles key always start. Services with profiles only start when that profile is activated. No more commenting out services in your Compose file.

The develop Section

Beyond watch, the develop section is Compose's answer to the "inner loop" problem — the cycle of code, build, test, repeat. It gives you a structured way to declare which files trigger syncs and which trigger rebuilds, keeping your container up-to-date without manual intervention.

This is Docker's opinionated answer to the question every developer asks: "How do I get my code changes into the container without rebuilding everything?" And honestly, it works well.

6. What Nobody Tells You

I have seen the same misconceptions trip up developers for years. Let me save you the trouble.

Compose Creates a Default Network Automatically

This is the one that surprises people the most. You do not need to add a networks: section to your Compose file. Compose automatically creates a bridge network named <project>_default and attaches every service to it.

All services resolve each other by their service name. When your web app connects to redis:6379, Docker DNS on the default Compose network handles it. No network configuration, no IP addresses, no service discovery tools.

You almost never need explicit networks: in your Compose file. The only time you do is when you are running multiple Compose projects that need to communicate, or when you need network-level isolation between services within the same project (like separating frontend services from database services). For a single-project development setup, the default network is perfect.

Compose Is NOT an Orchestrator

This is the big one, and I see it get teams into serious trouble.

Docker Compose is a development and testing tool. It runs containers on a single machine. It does not handle:

• Multiple hosts - Compose cannot spread services across a cluster of servers.

• Auto-scaling - It does not spin up more containers when traffic spikes.

• Self-healing - If a container crashes, restart: unless-stopped will restart it, but there is no real health-based orchestration.

• Rolling deployments - You cannot deploy a new version with zero downtime using Compose alone.

• Service mesh, load balancing, secrets rotation - None of it.

That is what Kubernetes does. Compose is for your laptop. Kubernetes (or a managed platform like ECS, Cloud Run, or Fly.io) is for production.

I have seen startups try to run docker compose up -d on an EC2 instance and call it production. It works until it doesn't - and when it doesn't, you have no observability, no failover, and no way to deploy without downtime. Use Compose for what it is: the best local development tool in the container ecosystem.

Service Names Are Your DNS

The service name you pick compose.yaml is not just a label. It is a real DNS entry on the Compose network. Name your services well: redis, postgres, api, web -not service1 or myapp. Your application code references these names directly as hostnames.

The Project Name Matters

Compose derives the project name from your directory name by default. All resource names are prefixed with it: myproject_default (network), myproject_redis-data (volume), myproject-redis-1 (container). If you rename your directory, Compose creates all-new resources and orphans the old ones. Set it explicitly with name: at the top of your Compose file if this matters to you.

7. Quick Reference

Commands

Compose File Keys

Key Takeaways

1. Docker Compose is how Docker actually gets used. One compose.yaml replaces dozens of docker run commands and lives in your repo alongside your code. Every developer on the team gets the same stack.

2. Networking is automatic. Compose creates a default network. Services find each other by name - redis, web, postgres - with zero configuration. You almost never need to think about it.

3. The workflow is up, down, and logs. Those three commands cover 90% of your daily Compose usage. Add --build after code changes, -v when you want a clean slate.

4. Compose Watch is the new way to develop. Forget bind mounts with their cross-platform headaches. The develop section watch gives you hot reload that works consistently everywhere.

5. Compose is not production infrastructure. It is the best local development tool in the Docker ecosystem. For production, you need Kubernetes, ECS, or a managed platform. Do not confuse the two.

What's Next: Day 6

You have gone from typing docker run commands one at a time to defining full application stacks declaratively. That is a massive leap.

But your containers have been talking to each other on Compose's default network without you thinking about it. What happens when you need to control that communication? What if your frontend should reach the API but never the database directly?

In Day 6: Docker Networking - Connecting Containers, you will learn:

• How Docker networking actually works under the hood

• Bridge, host, and none network drivers - and when to use each

• Custom bridge networks for DNS resolution and isolation

• Network security: isolating tiers of your application

• Multi-network architectures for real applications

The defaults Compose gave you today are great for development. Tomorrow, you will understand what is happening beneath them.

See you tomorrow.

Your pod has just been written to etcd. The API server returned 201 Created. The pod exists. But spec.nodeName is still empty, and that is the entire reason this post exists.

A pod with no node is not a real workload. It is a row in a database. Something has to look at it, decide which machine should run it, and atomically claim that machine. That something is kube-scheduler, and the way it makes the decision is more interesting than "pick the node with the most free CPU."

There are thirteen separate stages in modern scheduling. The Filter stage alone runs fourteen in-tree plugins, each one capable of disqualifying a candidate node with a single Unschedulable verdict. There is no appeal, no second chance, no "best effort." Either every plugin says yes, or that node is out.

This post walks every stage end-to-end against the v1.36 source code, with verbatim outputs from a real cluster at the bottom.

https://youtu.be/N-dDSCVWdqU

TL;DR, the 13 stages

1. PreEnqueue. Gating plugins decide if the pod is even allowed into the queue. SchedulingGates lives here. If a gate is set, the pod waits.

2. QueueSort. The activeQ orders pods by priority. Higher priority first.

3. PreFilter. Eleven plugins precompute what the pod actually wants. Resources, affinity terms, topology spread, all stashed in CycleState. Compute once, read many times.

4. Filter. Fourteen plugins each test every node in parallel. NodeUnschedulable, NodeName, TaintToleration, NodeAffinity, NodePorts, NodeResourcesFit, VolumeRestrictions, NodeVolumeLimits, VolumeBinding, VolumeZone, PodTopologySpread, InterPodAffinity, DynamicResources, NodeDeclaredFeatures. One Unschedulable verdict and the node is out.

5. PostFilter. Only fires if every node failed Filter. DefaultPreemption asks, "if I evicted some lower priority pods, could this one fit?" If yes, it picks victims and the pod retries next cycle.

6. PreScore. Same trick as PreFilter. Plugins that do heavy per node work during scoring precompute once and cache.

7. Score. Nine plugins rate every surviving node, zero to one hundred. In parallel. Each plugin has a weight. TaintToleration is three. NodeAffinity, InterPodAffinity, PodTopologySpread, DynamicResources are all two. Rest are one.

8. NormalizeScore. Rescales every plugin's output. Then for each node, multiply scores by weights, add it all up. Highest sum wins. Ties? Go's rand.Int. Yes, random. Deterministic ties would hot spot the same node every time.

9. Reserve. The scheduler subtracts the pod's requests from the winning node's in memory snapshot. So the next pod in the same cycle sees that node as already loaded.

10. Permit. A hook. A plugin can Approve, Wait, or Reject. Stock cluster, no op. But Kueue, Volcano, Coscheduling all wait here for gang scheduling.

11. PreBind. Last chance to do work before the API server gets told. VolumeBinding finalizes PVC binds here.

12. Bind. The DefaultBinder updates spec dot node name on the pod via the API server. Now etcd has the assignment.

13. PostBind. Cleanup. The pod is gone from the scheduler's queue.

That is the whole walkthrough. The rest of this post is the part that does not fit in a tweet.

The scheduling framework

Since Kubernetes 1.19, kube-scheduler has been built on top of the scheduling framework (KEP-624, beta in 1.18, GA in 1.19). The core of the binary is small and intentionally dumb. All of the actual decision-making lives in plugins, registered at well-defined extension points.

This separation is what makes the rest of the ecosystem possible. You can disable plugins. You can write your own as a Go module or behind a webhook. You can run multiple scheduler profiles side by side and let pods pick one with spec.schedulerName. Most installations never touch the configuration, but if you have ever wondered how Volcano, Kueue, or Coscheduling plug into the scheduler without forking it, this is the answer: they register against the framework's extension points and the core just calls them at the right time.

The thirteen extension points are not arbitrary. Each one corresponds to a moment in the pod's lifecycle where it makes sense to ask plugins a question. Should this pod even enter the queue? That is PreEnqueue. Is this node a candidate? That is Filter. Among the candidates, which one is the best fit? That is Score. The framework gives you the seam; the plugin fills in the logic.

Three queues, before any plugin runs

Before any plugin gets called, the pod has to make it into the right queue. The scheduler maintains three of them, and they each serve a different purpose.

The activeQ is a priority heap. Unscheduled pods are ordered by spec.priority, and the scheduler always pops from the head. Higher-priority pods cut in line, which is exactly what you want for things like critical control-plane pods or paid-tier workloads.

The backoffQ holds pods that just failed a scheduling attempt. They sit there for a small (and exponentially growing) timeout before being promoted back into the activeQ. This is not laziness; it is a correctness property. If a pod could not be scheduled in this cycle, retrying it immediately almost always fails the same way. Backoff lets the cluster state change first.

The unschedulableQ (the source actually calls it unschedulablePods, but the docs and the metrics use the queue suffix) is an indexed map of pods that have been declared unschedulable for now. These pods do not retry on a timer. They retry on events. If a new node is added, an informer event fires MoveAllToActiveOrBackoffQueue and they all get a fresh shot. Same thing if a pod is deleted and frees up resources. There is also a five-minute fallback timer for pods that have been waiting too long, in case the event stream missed an update.

All three queues live in pkg/scheduler/backend/queue/scheduling_queue.go. Their names are also exposed as labels on the scheduler_pending_pods metric, which is the easiest way to debug a stuck cluster: a queue full of pods in Unschedulable is telling you something different than a queue full of pods in Backoff.

Stage 1: PreEnqueue (the gate)

PreEnqueue plugins decide whether a pod is even allowed into the activeQ. If any plugin returns Unschedulable, the pod sits in the unschedulableQ until something causes its gate to clear.

The canonical example is the SchedulingGates plugin. By setting spec.schedulingGates on a pod, you can create the pod object now but defer its scheduling until you explicitly remove the gate. This pattern shows up in batch workloads, in cost-aware scheduling controllers, and in anything that wants to express "this pod exists but isn't ready to run yet."

Most pods sail through PreEnqueue with no gates set, but it is the very first checkpoint and worth knowing about.

Stage 2: QueueSort (the order)

Pods waiting in the activeQ have to be ordered somehow. QueueSort plugins define that order. The default is PrioritySort: it ranks pods by spec.priority (an integer) descending, and falls back to creation timestamp for ties. Older pod with the same priority wins.

There is one plugin, it does one thing, and you almost never want to change it. Worth a sentence in the model, not much more.

Stage 3: PreFilter (cache once)

Once a pod is popped off the activeQ, the scheduler's first real job is to look at what the pod actually wants. That is PreFilter, and it runs exactly once per pod per cycle.

The default profile registers eleven PreFilter plugins, each one extracting a different facet of the pod's requirements: NodeResourcesFit pulls out CPU, memory, and extended-resource requests; NodeAffinity normalizes the affinity term tree; PodTopologySpread builds its per-topology-key constraint sets; InterPodAffinity walks the affinity and anti-affinity rules; VolumeBinding figures out which PVCs still need binding; and so on.

All of this work is cached in a framework.CycleState object. Think of CycleState as a per-pod scratch pad. compute the expensive things once, read them many times. The reason it matters becomes obvious in the next stage, where each plugin is about to be called several thousand times in tight loops.

Stage 4: Filter (every node, every plugin, in parallel)

Filter is where the bulk of the scheduling work actually happens. Fourteen plugins are called against every candidate node, in parallel, and any single Unschedulable verdict eliminates that node from the rest of the cycle.

Here is the verified list, straight from pkg/scheduler/apis/config/testing/defaults/defaults.go:

1. NodeUnschedulable

2. NodeName

3. TaintToleration

4. NodeAffinity

5. NodePorts

6. NodeResourcesFit

7. VolumeRestrictions

8. NodeVolumeLimits

9. VolumeBinding

10. VolumeZone

11. PodTopologySpread

12. InterPodAffinity

13. DynamicResources (went GA in 1.36)

14. NodeDeclaredFeatures

Each plugin receives the pod, the candidate node's info, and the CycleState that PreFilter built up. Each plugin returns Success or Unschedulable. If any of them says Unschedulable, that node is gone. There is no aggregation, no scoring at this stage, no "well, three out of four said yes." It is binary, and that is what makes Filter fast: the scheduler can fan out to all 14 plugins in parallel, and short-circuit on the first failure per node.

Most engineers will only ever care about a handful of these by name. A quick tour:

NodeUnschedulable is the first line of defense. If the node has spec.unschedulable: true, this plugin filters it out. That is exactly what kubectl cordon does.

NodeName is the simplest possible filter. If the pod has spec.nodeName set (you can set it manually and skip the scheduler entirely), only that node passes; the scheduler effectively becomes a no-op.

TaintToleration is the one most engineers will recognize. The node has taints, the pod has tolerations, and any unmatched NoSchedule or NoExecute taint kills candidacy. The "GPU node" pattern in the demo at the bottom of this post is just a NoSchedule taint that nothing tolerates.

NodeAffinity evaluates the pod's spec.affinity.nodeAffinity rules. Required affinity terms must match here at Filter; preferred terms get scored later.

NodeResourcesFit is the one most people intuitively understand. Does the node have enough free CPU, memory, and any other Kubernetes resource (hugepages, GPUs, custom resources) to fit the pod's requests? Notably, only requests are considered, not limits, which is why a node can be massively over-subscribed on limits and the scheduler still happily places more pods.

VolumeBinding deserves a paragraph of its own. If the pod has PVCs that are not yet bound, VolumeBinding has to decide whether each unbound PVC could be bound on this specific node. For a WaitForFirstConsumer storage class the answer depends on zone, on the storage backend's topology, and on which PVs exist. VolumeBinding doesn't just filter; it also remembers the provisioning plan it chose, and that plan gets locked in during the Reserve stage further down.

DynamicResources is the new kid on the block. It implements the DRA framework, which went GA in 1.36. If your pod uses ResourceClaims (the modern way to ask for GPUs and other devices), DynamicResources is the plugin that figures out whether a node can satisfy the claim.

NodeDeclaredFeatures is newer still. It compares features the node has declared against the pod's required features, and is feature-gated in some configurations.

Run all 14 plugins in parallel against every node, collect the verdicts, and whatever survives all 14 votes moves on. If nothing survives, the scheduler doesn't give up: it runs PostFilter.

Stage 5: PostFilter (preemption, the expensive escape hatch)

If every node failed Filter, the scheduler is in trouble. The pod is unschedulable on the cluster as it stands today. PostFilter exists for exactly this case, and the default plugin is DefaultPreemption. It asks a single question: if I evicted some lower-priority pods, could this one fit?

The algorithm sounds simple but is genuinely expensive. For each node, the scheduler:

1. Gathers all pods on the node with priority lower than the pending pod.

2. Simulates evicting them one at a time, lowest priority first.

3. After each simulated eviction, re-runs Filter on the hypothetical state.

4. If the pod becomes schedulable, the node is a candidate, and the minimum set of pods that need to die is recorded.

After all candidate nodes have been evaluated, the scheduler picks the "best" one. fewest victims, lowest victim priority, latest creation time as a tiebreaker. The exact ordering lives in pkg/scheduler/framework/plugins/defaultpreemption/default_preemption.go.

Once a candidate is picked, two things happen. The scheduler sets nominatedNodeName on the pending pod, so anyone watching the API can see this pod is targeting a specific node. Then it gracefully deletes the victims through the API server, respecting their terminationGracePeriodSeconds. The pending pod itself goes back into the activeQ to be retried in the next cycle.

This whole process is expensive. The first Filter sweep already touched every node. Now the scheduler is running Filter again, multiple times, against simulated state per candidate. Tens to hundreds of milliseconds easily, seconds on large clusters. The good news is that the vast majority of pods never hit this path; they schedule cleanly on the first try.

PostFilter has a second plugin now: DynamicResources. Same idea, but for ResourceClaims rather than pods. If a Filter cycle failed because of a claim that is busy, DynamicResources' PostFilter can deallocate idle claims to make room.

Stage 6: PreScore (cache once again)

Filter has done its work. Maybe four nodes are left, maybe forty. Either way, it is time to score them, and the scheduler reuses the same precompute trick from PreFilter.

Some Score plugins do expensive per-node work. To avoid recomputing the same input data once per node, those plugins do their work once in PreScore and stash the result in CycleState. The default PreScore plugins are TaintToleration, NodeAffinity, NodeResourcesFit, VolumeBinding, PodTopologySpread, InterPodAffinity, and NodeResourcesBalancedAllocation.

InterPodAffinity is the heaviest of the bunch. It walks the cluster's existing pods, builds a topology map of where each pod sits, and converts the new pod's affinity rules into an indexed structure. PodTopologySpread does similar work, building per-topology-key counts.

After PreScore, each individual Score call becomes effectively O(1). a lookup against precomputed state. Without it, scoring large clusters would be unworkable.

Stage 7: Score (the leaderboard, weighted)

Now the actual ranking. Every Score plugin rates every surviving node from 0 to 100, in parallel.

The default Score plugins, with weights:

• TaintToleration, weight 3

• NodeAffinity, weight 2

• NodeResourcesFit, weight 1

• VolumeBinding, weight 1

• PodTopologySpread, weight 2

• InterPodAffinity, weight 2

• DynamicResources, weight 2

• NodeResourcesBalancedAllocation, weight 1

• ImageLocality, weight 1

That is nine plugins. All verified against defaults.go.

The weights are not arbitrary. The source comments explain the reasoning: TaintToleration is tripled because user-expressed taint preference is a strong signal. NodeAffinity, PodTopologySpread, InterPodAffinity, and DynamicResources are doubled because they encode user intent. The rest are weight 1 because they are infrastructure-level signals (balance, cache hits) that should influence the decision but not dominate it.

It is worth zooming in on ImageLocality for a moment. Once you understand it, you start noticing its effect everywhere.

ImageLocality asks one question per node: do you already have the container image's layers cached? If yes, score 100. If no, score 0. That is the entire plugin.

It matters because on a cold node, the kubelet has to pull the image over the network. seconds for a small image, tens of seconds for a fat ML or LLM image. On a warm node, the pod starts in milliseconds. ImageLocality is a soft preference (it doesn't filter), but it nudges the scheduler toward already-warm nodes when other things are equal, and the cumulative effect on workload startup latency is huge.

NodeResourcesFit is the resource-balance plugin you've probably tuned at some point. By default it uses LeastAllocated, which prefers nodes with more free capacity (spreading the load). You can flip it to MostAllocated for bin-packing, or to RequestedToCapacityRatio for custom curves, via KubeSchedulerConfiguration.

NodeResourcesBalancedAllocation is subtler. It rewards nodes whose CPU and memory utilization are balanced. A node at 80% CPU and 20% memory scores worse than a node at 50%/50%, because imbalanced nodes are more likely to fragment future scheduling decisions.

Stage 8: NormalizeScore and picking a winner

All nine plugins have scored every surviving node. The scheduler now picks the winner.

NormalizeScore rescales every plugin's output to a uniform 0 to 100 range. Some plugins return raw counts or other custom scales; this stage brings everything to the same units.

For each node, the scheduler then sums score × weight across all nine plugins. Highest sum wins.

The interesting question is what happens when two nodes have exactly the same total. The scheduler picks one at random. specifically, it uses Go's math/big.Int random (rand.Int), not rand.Intn. The choice matters more than it might seem.

Random tie-breaking exists to prevent hot-spotting. Imagine two equally suitable nodes for a workload. If the scheduler always picked the first one in some deterministic order, every pod from that workload would pile onto the same node and the other one would sit empty. Randomization spreads the load.

The choice of rand.Int over rand.Intn matters because rand.Intn has a subtle modulo bias for non-power-of-two ranges. Over millions of scheduling decisions across a large cluster, that bias becomes a real distribution skew. rand.Int avoids it.

Stage 9: Reserve (claim the resources in memory, before the API knows)

The winner is picked, but at this point the API server still does not know about the decision. As far as etcd is concerned, the pod is still unscheduled.

Reserve fixes that locally. The scheduler takes the winning node's in-memory snapshot and subtracts whatever the pod requested: CPU, memory, extended resources, PVCs that need binding.

A critical detail: the scheduler operates on requests, not limits. And if your pod has no requests at all, the scheduler does not invent defaults. that is LimitRanger's job, much earlier at admission time. Here, Reserve subtracts whatever requests the pod has, even if it is zero. The scheduler's view of node capacity is purely request-based; a node could be massively over-subscribed on limits and the scheduler would never know or care.

The reason Reserve happens in memory before the bind is so the next pod in the same scheduling cycle sees this node as already loaded. Picture scaling a deployment to twenty replicas all at once: without Reserve, the scheduler's cache would still show the same node as fully free for every pod, and they would all pile onto it. Reserve makes the cache reflect the scheduler's intent immediately, even before etcd has acknowledged anything.

If anything fails after Reserve, Unreserve rolls it back. The in-memory subtraction is undone and the node looks free again.

Stage 10: Permit (the gang-scheduling hook)

Permit is a hook with three possible outcomes per plugin. Approve lets the bind proceed (the default). Wait parks the pod, waiting for an external signal. Reject fails scheduling outright.

A stock cluster has no Permit plugins registered, so most pods sail through. But Permit is the seam where gang scheduling lives. Kueue, Volcano, and Coscheduling all register Permit plugins, and the pattern is the same: when the first pod of a gang arrives, return Wait and park it. When the last pod of the gang arrives, signal all the parked pods to proceed. They all bind together, atomically.

Without Permit, gang scheduling on Kubernetes would be effectively impossible. You would have to bind each pod individually and then evict the rest when one failed. Permit lets you wait at the right point. before any pod is bound. so failures cost nothing.

Stages 11, 12, 13: PreBind, Bind, PostBind (commit and clean up)

Permit returned Approve. Three stages left, all of them short.

PreBind is the last opportunity to do work before the API server is told. The biggest user is VolumeBinding: for dynamically provisioned PVCs, this is where the PV is actually created and the PVC's spec.volumeName is set. By the time Bind runs, the PVC is bound and ready.

Bind does the actual API call. The default is DefaultBinder, which calls pods.Bind() on the API server. a special endpoint that updates spec.nodeName and creates a Binding object. etcd persists it via Raft, followers fsync, and the pod is now officially assigned.

The kubelet on the chosen node has been watching the API server for pods with its own nodeName. The instant the bind lands, the kubelet's informer fires. The pod is no longer the scheduler's concern; it now belongs to a different deep-dive (image pull, runc, the five syscalls).

PostBind is cleanup. The scheduler removes the pod from its internal queue, and that scheduling cycle is done.

The live demo, preemption in action

Theory only carries so far. To watch the scheduler actually preempt a pod, we ran this against a real cluster (Kubernetes 1.36.1, three workers, one tainted). What follows are verbatim outputs from the live recording.

The setup: three worker nodes, with kube-worker-3 tainted as a fake GPU node so the scheduler refuses to put general workloads there.

$ kubectl get nodes
NAME            STATUS   ROLES           AGE   VERSION
kube-cp-01      Ready    control-plane   41d   v1.36.1
kube-worker-1   Ready    <none>          41d   v1.36.1
kube-worker-2   Ready    <none>          41d   v1.36.1
kube-worker-3   Ready    <none>          12d   v1.36.1

$ kubectl describe node kube-worker-3 | grep -E 'Taints|cpu:|memory:'
Taints:             workload=gpu:NoSchedule
  cpu:                8
  memory:             32852Mi
  cpu:                7800m
  memory:             30100Mi

We deploy a regular nginx pod requesting eight CPU. It schedules cleanly onto kube-worker-1 and starts up.

$ kubectl apply -f nginx-pod.yaml
pod/nginx-demo created

$ kubectl get events --sort-by=.lastTimestamp | tail -5
LAST SEEN   TYPE     REASON      OBJECT           MESSAGE
6s          Normal   Scheduled   pod/nginx-demo   Successfully assigned default/nginx-demo to kube-worker-1
5s          Normal   Pulling     pod/nginx-demo   Pulling image "nginx:1.27"
3s          Normal   Pulled      pod/nginx-demo   Successfully pulled image "nginx:1.27" in 1.812s
2s          Normal   Created     pod/nginx-demo   Created container: nginx
2s          Normal   Started     pod/nginx-demo   Started container nginx

$ kubectl get pod nginx-demo -o wide
NAME         READY   STATUS    RESTARTS   AGE   IP           NODE            NOMINATED NODE   READINESS GATES
nginx-demo   1/1     Running   0          18s   10.244.2.47  kube-worker-1   <none>           <none>

The cluster is now in a deliberately uncomfortable state. kube-worker-1 is mostly full. kube-worker-2 is similarly loaded. kube-worker-3 is empty but tainted. Then we apply a critical pod that asks for the same eight CPU, with priority 1,000,000, and no taint toleration.

$ kubectl apply -f payments-high-prio.yaml
pod/payments-critical created

The first scheduling cycle has nothing to give it. Three nodes are insufficient, the fourth has the wrong taint. The scheduler turns to PostFilter, which walks each node looking for a preemption victim. The tainted node is no help. The non-tainted nodes each have a candidate to evict. The scheduler picks one, sets nominatedNodeName, and gracefully evicts the lower-priority nginx pod.

$ kubectl describe pod payments-critical | tail -14
QoS Class:        Guaranteed
Priority:         1000000
Priority Class:   high-priority-payments
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  14s   default-scheduler  0/4 nodes are available: 3 Insufficient cpu, 1 node(s) had untolerated taint {workload: gpu}. preemption: 0/4 nodes are available: 1 Preemption is not helpful for scheduling, 3 No preemption victims found for incoming pod.
  Normal   Preempted         9s    default-scheduler  Preempted by default/nginx-demo on node kube-worker-1
  Warning  FailedScheduling  9s    default-scheduler  0/4 nodes are available: 3 Insufficient cpu. preemption: 0/4 nodes are available.
  Normal   Scheduled         4s    default-scheduler  Successfully assigned default/payments-critical to kube-worker-1
  Normal   Pulling           3s    kubelet            Pulling image "payments:v2.4.1"
  Normal   Pulled            1s    kubelet            Successfully pulled image "payments:v2.4.1" in 1.632s
  Normal   Created           1s    kubelet            Created container: payments
  Normal   Started           1s    kubelet            Started container payments

Read those events carefully. There is a FailedScheduling at 14s, then Preempted by default/nginx-demo on node kube-worker-1 at 9s, then another FailedScheduling (the cycle right after preemption, where the nginx pod was still terminating), then Scheduled at 4s. From request to running, on a real cluster, about ten seconds. That includes the graceful eviction of the victim, which is the slow part.

$ kubectl get pods -o wide
NAME                READY   STATUS    RESTARTS   AGE   IP           NODE            NOMINATED NODE   READINESS GATES
payments-critical   1/1     Running   0          22s   10.244.2.58  kube-worker-1   <none>           <none>

That is preemption working as designed. A higher-priority pod arrives, the scheduler refuses to leave it pending when there is a lower-priority pod that could be moved, and the cluster reshuffles. No human intervention. No alert at 3 a.m.

Three takeaways

If only three things from this post stick with you:

1. The scheduler is plugins all the way down. Since 1.19, every meaningful decision is delegated to a plugin at one of thirteen extension points. You can write your own, disable the defaults, run multiple profiles in parallel. Volcano, Kueue, and Coscheduling exist because of this design. they did not have to fork the scheduler.

2. Filter is binary; Score is weighted. A single Unschedulable verdict from any of fourteen Filter plugins kills a node's candidacy. But Score is a weighted vote across nine plugins, and the weights are not equal. TaintToleration (×3) is the strongest single signal at scoring time, followed by the four ×2 plugins (NodeAffinity, PodTopologySpread, InterPodAffinity, DynamicResources). Weights matter much more than most engineers realize.

3. Reserve is why your scheduling is consistent. When you scale a deployment from one to twenty replicas and they all hit the scheduling queue in the same one-second window, Reserve's in-memory subtraction is what stops them from piling onto the same node. The scheduler commits an opinion before the API server even confirms the bind, and that opinion is visible to the next pod's scheduling cycle immediately.

Where to go from here

The full scheduler walkthrough on YouTube has the live demo, every stage animated, the preemption flow shown end-to-end. Link is at the top of this post.

If you want to step through it yourself rather than watch, the interactive at https://kubernetes-explained.vercel.app/scheduler walks every internal step with annotations and lets you pause anywhere.

Sources for every claim in this post:

• pkg/scheduler/apis/config/testing/defaults/defaults.go: plugin lists and weights

• pkg/scheduler/framework/plugins/: individual plugin implementations

• pkg/scheduler/backend/queue/scheduling_queue.go: the three queues

• pkg/scheduler/framework/plugins/defaultpreemption/default_preemption.go: the preemption algorithm

• KEP-624. the scheduling framework graduation history

• The kubectl describe pod events shown in the demo above are verbatim from a real Kubernetes 1.36.1 cluster, captured for this post.

7 Days of Docker in 2026 — When Containers Need to Talk and Remember

On Day 3, you built production-ready images with Dockerfiles, optimized layers, and mastered multi-stage builds. Every container you have launched so far has been a tiny, sealed universe - isolated filesystem, isolated network, isolated process tree. That isolation is the entire point of containers.

But here is the thing nobody tells you on day one: useful applications break isolation constantly. A database container that cannot remember data after a restart is useless. A web frontend that cannot reach its API backend is useless. A Redis cache that no service can connect to is useless.

Today, you learn to break isolation on purpose - with volumes (persistent storage), custom networks (container-to-container communication), and port mapping (exposing services to your host). These are the three controlled breaches that turn toy containers into real systems.

Table of Contents

• 1. Data Dies With Containers

• 2. Volumes: Breaking Filesystem Isolation

• 3. Networks: Breaking Network Isolation

• 4. Port Mapping: Breaking Host Isolation

• 5. Putting It Together: Redis Service Discovery

• 6. What Nobody Tells You

• 7. Quick Reference

• What's Next: Day 5

1. Data Dies With Containers

Before I show you the fix, I need you to feel the problem.

Write a file inside a running container:

docker run --name d4-writer alpine sh -c 'echo "Important!" > /data.txt && cat /data.txt'

Important!

The file exists. Now remove that container and try to read from a fresh one:

docker rm d4-writer

docker run --name d4-reader alpine sh -c 'cat /data.txt'

cat: can't open '/data.txt': No such file or directory

Data is GONE!

docker rm d4-reader

Each container gets its own copy-on-write layer (Day 2). When the container dies, that layer is garbage-collected. Every file, every database row, every log entry - deleted. This is not a bug. This is exactly how containers are supposed to work.

But it means you need a deliberate strategy for any data that outlives a single container. That is where volumes come in.

2. Volumes: Breaking Filesystem Isolation

Docker gives you three ways to punch a hole through the filesystem wall. Each one trades some isolation for some capability.

Named Volumes - Let Docker Manage It

Named volumes are Docker-managed storage areas that live outside any container's lifecycle. You create them by name, mount them into containers, and Docker handles everything else - location on disk, permissions, and cleanup.

$ docker volume create d4-data

d4-data

Write data from one container:

docker run --rm -v d4-data:/data alpine sh -c \
  'echo "Persisted on Wed Apr 22" > /data/message.txt && cat /data/message.txt'

Persisted on Wed Apr 27

That container is already gone (thanks to --rm). Now read from a completely new container - different ID, different writable layer, different process:

docker run --rm -v d4-data:/data alpine cat /data/message.txt

Persisted on Wed Apr 27

The data survived. The volume exists independently of any container that mounts it.

Best for: Databases, application state, anything that must survive container replacement. This is your default choice for production.

Bind Mounts - Map Host Directories Directly

A bind mount maps a specific path on your host machine into the container. The container sees your host files in real time — edit on the host, and the container picks it up instantly.

mkdir -p /tmp/d4-site

cat > /tmp/d4-site/index.html << 'EOF'
<h1>Hello from a bind mount!</h1>
EOF

docker run -d --name d4-web -p 8084:80 \
  -v /tmp/d4-site:/usr/share/nginx/html:ro nginx:alpine

The :ro flag means read-only - the container can read your files, but cannot modify them. This matters. More on that in section 6.

curl -s http://localhost:8084 | grep '<h1>'

<h1>Hello from a bind mount!</h1>

Now edit the file on your host and hit it again - no rebuild, no restart:

echo '<h1>Updated LIVE!</h1>' > /tmp/d4-site/index.html

curl -s http://localhost:8084 | grep '<h1>'

<h1>Updated LIVE!</h1>

Best for: Local development with hot reload. Never use bind mounts in production — they create host dependency and break portability.

docker stop d4-web && docker rm d4-web

tmpfs Mounts - Memory Only, Never Touches Disk

docker run --rm --tmpfs /app/scratch alpine sh -c \
  'echo "secret" > /app/scratch/token.txt && cat /app/scratch/token.txt'

secret

The data lives in RAM. It is never written to disk and vanishes when the container stops. Best for: Secrets, API tokens, or scratch space that must never be persisted.

The Decision Matrix

Captain's Rule: When in doubt, use a named volume. Bind mounts are for development. tmpfs is for secrets. Named volumes are for everything else.

3. Networks: Breaking Network Isolation

You can persist data now. But a container that stores data and talks to nobody is just an expensive text file. Real applications are systems of services that need to find each other.

Docker networking has one massive gotcha, and I guarantee most tutorials will bury it three paragraphs too deep.

The Default Bridge Trap

Every container you have created so far lands on the default bridge network. Containers on the default bridge can reach each other by IP address — but not by name.

That means no DNS. No service discovery. Just raw IPs that change every time a container restarts. This is fine for throwaway experiments. It is terrible for anything real.

Custom Bridge Networks - DNS for Free

Create a custom bridge network, and Docker gives you an embedded DNS server. Every container on that network is registered by its --name, and any other container can resolve that name to the correct IP.

docker network create d4-net

Launch two containers on that network:

docker run -d --name d4-web --network d4-net nginx:alpine

docker run -d --name d4-api --network d4-net nginx:alpine

Now ping by container name - not IP:

docker exec d4-api ping -c 2 d4-web

PING d4-web (172.22.0.2): 56 data bytes
64 bytes from 172.22.0.2: seq=0 ttl=64 time=0.721 ms
64 bytes from 172.22.0.2: seq=1 ttl=64 time=0.234 ms

--- d4-web ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.234/0.477/0.721 ms

That 172.22.0.2 could change tomorrow if you recreate the container. But d4-web as a DNS name will always resolve to whatever IP the d4-web container currently holds. This is the same pattern Kubernetes uses with its internal DNS — Docker just gives it to you for free on custom networks.

Key Concept: Default bridge = no DNS, hardcoded IPs, fragile. Custom bridge = automatic DNS, service discovery by name, resilient. Always create a custom bridge for multi-container applications.

docker stop d4-web d4-api && docker rm d4-web d4-api

The Full Isolation Spectrum

Bridge networks are what you will use most days, but they are only two points on a wider spectrum. Docker ships several network modes, each sitting at a different level of isolation. Knowing they exist saves you from reaching for the wrong tool.

--network host drops the network wall entirely. Your container shares the host's network stack, sees every interface, binds ports directly without NAT. Fastest possible networking, zero translation. The cost: no port mapping is needed because there is no isolation, two containers cannot bind the same port, and you lose service-name DNS. Use it for monitoring agents, packet-capture tools, or latency-sensitive workloads on Linux.

$ docker run --rm --network host alpine ip addr show | head -10

You will see the same interfaces as ip addr show on your host. That is the point: there is no separate network namespace.

--network none removes networking completely. The container has only a loopback interface. No outbound connections, no inbound traffic, no DNS. This is the right choice for jobs that should never reach the network, or for proving that your application degrades gracefully when networking is unavailable.

$ docker run --rm --network none alpine ping -c 2 google.com

ping: bad address 'google.com'

No DNS, no route. Exactly as expected.

overlay networks stretch a single logical network across multiple Docker hosts. They are how Swarm wires services together across a cluster of machines. You will not use overlay on a single laptop, but it is worth knowing the name for the day you graduate from one host to many.

Captain's Rule: Custom bridge for almost everything. host when bridge NAT is the bottleneck. none when the container should be airtight. overlay when you outgrow a single machine.

4. Port Mapping: Breaking Host Isolation

Volumes break filesystem isolation. Networks break container-to-container isolation. Port mapping breaks the final wall: host isolation.

By default, no traffic from your host machine can reach a container's internal ports. You explicitly map a host port to a container port with -p:

docker run -d --name d4-nginx -p 8080:80 nginx:alpine

This means: "Traffic hitting localhost:8080 on my host gets forwarded to port 80 inside the container."

curl -s http://localhost:8080 | head -5

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
</head>

Without -p 8080:80, that nginx process would be running and listening — but completely unreachable from your browser or curl.

Port Mapping Syntax

Captain's Rule: In production, always bind to 127.0.0.1 unless you explicitly want external traffic. A bare -p 8080:80 binds to 0.0.0.0 — every network interface on your host, including public-facing ones.

docker stop d4-nginx && docker rm d4-nginx

5. Putting It Together: Redis Service Discovery

Time to combine everything. We will run a Redis container and an Alpine client on a custom network, use DNS-based service discovery to connect by name, and persist Redis data with a named volume.

Set Up the Network and Volume

docker volume create d4-redis-data

docker network create d4-app-net

Launch Redis with a Named Volume

docker run -d \
--name d4-redis \
--network d4-app-net \
-v d4-redis-data:/data \
redis:7-alpine redis-server --appendonly yes

The --appendonly yes flag tells Redis to persist data to disk (inside /data, which is backed by our volume). Without it, Redis keeps everything in memory and loses it all on restart.

Connect an App Container by Name

docker run --rm -it \
--name d4-client \
--network d4-app-net \
redis:7-alpine redis-cli -h d4-redis

Notice -h d4-redis we connect using the container name, not an IP. Docker's embedded DNS resolves it automatically.

d4-redis:6379> SET greeting "Hello Day 4!"
OK
d4-redis:6379> GET greeting
"Hello Day 4!"
d4-redis:6379> exit

The value is set. Now kill the Redis container, start a fresh one with the same volume and network, and prove the data survived:

docker rm -f d4-redis

docker run -d \
--name d4-redis \
--network d4-app-net \
-v d4-redis-data:/data \
redis:7-alpine redis-server --appendonly yes

docker run --rm \
--network d4-app-net \
redis:7-alpine redis-cli -h d4-redis GET greeting

"Hello Day 4!"

The container was destroyed and replaced. The data persisted (volume). The new client found it by name (custom network DNS). This is the pattern behind every microservice architecture - disposable compute, durable storage, DNS-based discovery.

Cleanup

docker rm -f d4-redis
docker network rm d4-app-net
docker volume rm d4-redis-data d4-data

6. What Nobody Tells You

Here is the section I wish someone had written for me when I was learning Docker.

Bind Mounts With :rw Can Destroy Your Code

By default, bind mounts are :rw (read-write). That means the container has full write access to your host directory. A misconfigured build step, a buggy script, or a careless rm -rf inside the container, will modify or delete your actual source code on the host.

# This is dangerous:
docker run -v $(pwd):/app some-image npm run build
# If the build writes to /app, it's writing to YOUR filesystem.

Always use :ro for bind mounts unless the container genuinely needs to write back to the host. And when it does need to write, mount only the specific subdirectory it needs — never your entire project root with write access.

The Default Bridge Has No DNS, And This Will Bite You

I cannot stress this enough. The default bridge network does not provide DNS resolution. If you run two containers without --network and try to connect by name, it will fail silently or hang. I have seen teams waste hours debugging "connection refused" errors that were really just missing DNS because nobody created a custom network.

Volume Data Outlives Everything

Named volumes are not removed when you run docker rm, docker system prune, or even docker system prune -a. The only commands that touch volumes are docker volume rm and docker volume prune. This is by design, but it means orphaned volumes accumulate silently. Run docker system df monthly. You will be surprised.

Port Mapping Bypasses Your Firewall

On Linux, Docker's port mapping manipulates iptables directly. This means -p 8080:80 can expose a service to the public internet even if your host firewall blocks port 8080. Docker inserts its rules before the firewall's. This is a well-known footgun and has led to real production incidents. Always bind to 127.0.0.1 explicitly if you only need local access.

7. Quick Reference

Volume Commands

docker volume create my-data          # Create a named volume
docker volume ls                       # List all volumes
docker volume inspect my-data          # Show volume details
docker volume rm my-data               # Delete a volume
docker volume prune                    # Remove unused anonymous volumes
docker volume prune -a                 # Remove ALL unused volumes (careful!)

Network Commands

docker network create my-net           # Create a custom bridge
docker network ls                      # List all networks
docker network inspect my-net          # Show network details and connected containers
docker network rm my-net               # Delete a network
docker network connect my-net my-ctr   # Attach a running container to a network
docker network disconnect my-net my-ctr # Detach a running container

Running Containers With Storage and Networks

# Named volume
docker run -v my-data:/app/data myimage

# Bind mount (read-only)
docker run -v /host/path:/container/path:ro myimage

# tmpfs
docker run --tmpfs /app/temp myimage

# Custom network with port mapping
docker run -d --name web --network my-net -p 8080:80 nginx:alpine

# The full combo: volume + network + port
docker run -d --name db \
  --network my-net \
  -v db-data:/var/lib/postgresql/data \
  -p 5432:5432 \
  postgres:17-alpine

Common Database Volume Paths

Key Takeaways

1. Containers are isolated by design. Breaking that isolation is not a failure — it is the entire point of building real applications with Docker.

2. Volumes break filesystem isolation. Named volumes (Docker-managed, portable) for production. Bind mounts (host-mapped, live reload) for development. tmpfs (memory-only) for secrets.

3. Custom bridge networks break network isolation. The default bridge has no DNS. A custom bridge gives you free service discovery by container name. Always create one.

4. Port mapping breaks host isolation. -p 8080:80 lets your host reach the container. Bind to 127.0.0.1 in production unless you intend public exposure.

5. Volumes outlive containers. We proved it: killed Redis, started a fresh container with the same volume, and GET greeting still returned "Hello Day 4!".

6. DNS-based discovery is the pattern. Connect by container name, not IP. IPs change. Names do not. This is how Kubernetes works, too - learn the pattern now.

7. Bind mounts with :rw can modify your host filesystem. Default to :ro. Mount only what you need. Never give a container write access to your entire project directory unless you understand the risk.

What's Next: Day 5

You can now persist data, connect containers by name, and expose services to your host. But you typed a lot of docker run commands today, each with a growing list of flags. Imagine managing ten services this way.

In Day 5: Docker Compose - Defining Multi-Container Applications, you will learn to describe your entire application stack in a single YAML file and bring it all up with one command. The volume mounts, networks, and port mappings you learned today will become simple declarations in a Compose file.

See you tomorrow.

7 Days of Docker (2026), by Saloni Narang, Docker Captain & CNCF Ambassador

I'm a Docker Captain. I've reviewed many Dockerfiles, and I can tell you that every single Dockerfile tutorial on the internet starts the same way: FROM, RUN, COPY, CMD. Memorize four keywords. Congratulations, you know nothing.

Knowing the instructions is like knowing how to spell. It doesn't make you a writer. Today I'm not tell you syntax. Today I'm going to tell you to think about Dockerfiles: why each line exists, what order they go in, and what happens to your build time and image size when you get it wrong.

If you followed Day 2, you already understand that images are stacked layers. Now you're going to create those layers yourself, and you're going to do it right the first time.

Table of Contents

• 1. docker init: The 2026 Way

• 2. Tearing Apart a Dockerfile

• 3. The Cache Trick That Changes Everything

• 4. Multi-Stage Builds: It's Not About Size

• 5. docker debug: Your New Best Friend

• 6. What Nobody Tells You

• 7. Quick Reference

• What's Next: Day 4

1. docker init: The 2026 Way

Here's my first controversial opinion of the day: stop writing Dockerfiles from scratch.

docker init exists. It asks you five questions and generates a very good Dockerfile, .dockerignore, compose.yaml, and a README.Docker.md. Here's what it actually looks like when you run it on a Node.js project:

$ docker init

Welcome to the Docker Init CLI!

This utility will walk you through creating the following files with sensible defaults for your project:
  - .dockerignore
  - Dockerfile
  - compose.yaml
  - README.Docker.md

Let's get started!

? What application platform does your project use? Node
? What version of Node do you want to use? 20.17.0
? Which package manager do you want to use? npm
? What command do you want to use to start the app? node server.js
? What port does your server listen on? 3000

✔ Created → .dockerignore
✔ Created → Dockerfile
✔ Created → compose.yaml
✔ Created → README.Docker.md

→ Your Docker files are ready!
  Review your Docker files and tailor them to your application.
  Consult README.Docker.md for information about using the generated files.

! Warning → The following files required to run your application were not found.
  Create them before running your application:
  - package.json
  - package-lock.json

Five questions. Four files. And a warning that keeps you honest: it noticed package.json and package-lock.json don't exist yet. That warning matters, as you'll see in a moment.

So why am I writing a blog post about Dockerfiles if a tool generates them?

Because you need to understand what it generates. You need to be the person who can look at a generated Dockerfile and say "this is correct" or "this needs to change for our use case." docker init is where you start. Understanding is where you become dangerous.

Let's dissect what it actually produced, line by line.

2. Tearing Apart a Dockerfile

This is the Dockerfile docker init actually generated. Not simplified. Not cleaned up. The real thing.

# syntax=docker/dockerfile:1

ARG NODE_VERSION=20.17.0

FROM node:${NODE_VERSION}-alpine

ENV NODE_ENV production

WORKDIR /usr/src/app

RUN --mount=type=bind,source=package.json,target=package.json \
    --mount=type=bind,source=package-lock.json,target=package-lock.json \
    --mount=type=cache,target=/root/.npm \
    npm ci --omit=dev

USER node

COPY . .

EXPOSE 3000

CMD node server.js

Every decision in here is intentional. Let's go line by line.

# syntax=docker/dockerfile:1: This is a parser directive, not a comment. It must be the very first line. It pins the Dockerfile to the latest stable 1.x BuildKit syntax, ensuring consistent behaviour across different Docker versions, CI systems, and remote builders. In Docker 23.0+ BuildKit is on by default so --mount works without it, but on older installs or CI environments it may not. Keep it for portability.

ARG NODE_VERSION=20.17.0: A build-time variable with a default. This means you can override it without touching the Dockerfile: docker build --build-arg NODE_VERSION=22.0.0 .. Pin to a specific version in production; override in CI when testing upgrades.

FROM node:${NODE_VERSION}-alpine: Uses the ARG from above. Alpine is significantly smaller than Debian-based images, often reducing image size by several hundred MB. This is not "use Node." It's "start with this entire filesystem and runtime." Every line after this is built on top of Alpine + Node 20.17.0.

ENV NODE_ENV production: Sets an environment variable that persists into the running container. At runtime, this tells Node frameworks like Express and Next.js to enable production-mode optimizations. Note: for npm ci, NODE_ENV=production alone does not skip devDependencies. The --omit=dev flag in the RUN step handles that explicitly.

WORKDIR /usr/src/app: Sets the working directory for every subsequent instruction. Note it's /usr/src/app, not /app. This is the conventional location for Node apps per the Node Docker Best Practices guide. Docker creates it if it doesn't exist.

RUN --mount=type=bind ... npm ci --omit=dev: This is the line that makes this Dockerfile different from what you'd write by hand. Three things happening at once:

• --mount=type=bind,source=package.json,target=package.json: Binds your package.json into the container only for this RUN step. It is not persisted in the final image layer, but it still influences build cache. The file is available at build time, gone from the image afterward.

• --mount=type=bind,source=package-lock.json,target=package-lock.json: Same for the lockfile. This is why docker init warned us both files need to exist; it's reading them directly from your filesystem, not copying them.

• --mount=type=cache,target=/root/.npm: Mounts a persistent build cache at the npm cache directory. On the next build, npm reuses packages from this cache even if the image layer is rebuilt. Dramatically faster on repeated builds.

• npm ci instead of npm install: ci installs exactly what's in the lockfile, fails if there's a mismatch, and never modifies package-lock.json. Deterministic. Right for production builds.

• --omit=dev instead of --production: The modern flag name. Same effect: skips devDependencies.

USER node: Switches to the non-root node user before copying application code. This matters: files copied after this instruction are owned by node, not root. If an attacker compromises your app process, they land as an unprivileged user. Root in a container is not the same as root on the host, but it's still a risk you don't need to take.

COPY . .: Copies your entire application source into the image. Comes after USER node so the files are owned correctly, and after npm ci so a source code change doesn't invalidate the dependency install step. The .dockerignore file (also generated by docker init) controls what gets excluded.

EXPOSE 3000: Metadata only. It does not open a port by itself, but acts as metadata used by tools like Docker run -p, Compose, and Kubernetes. It documents what port the application listens on so that docker run -P, Compose, and orchestrators can wire it up automatically.

CMD node server.js: The default command. Shell form, which means the shell (/bin/sh -c) wraps the process. The generated Dockerfile uses shell form here, which is simpler for a starter template. In production you may want exec form (CMD ["node", "server.js"]) so the Node process receives signals directly as PID 1. In short, shell form does not handle signals correctly. Exec form is preferred in production for proper shutdown handling.

Once you have your package.json, package-lock.json, and server.js in place, build and run with:

docker build -t d3-app .
docker run -d --name d3 -p 3000:3000 d3-app

Then run docker history d3-app and pay attention to what is not there. package.json and package-lock.json will not appear as layers at all. They were bind-mounted during the RUN step, available to npm ci, but never copied into the image. The npm cache mount is the same story: it lives outside the image entirely. The only real weight your image carries is the npm ci layer and whatever your COPY . . step brings in.

3. The Cache Trick That Changes Everything

This is where most tutorials fail you. They show the correct Dockerfile but never explain why the order matters. Let me fix that.

Docker's build cache works on one brutal rule: the moment a layer changes, every layer after it is rebuilt from scratch. Cache invalidation cascades downward. Always.

Here's what a bad Dockerfile looks like:

FROM node:20-alpine
WORKDIR /app
COPY . .                          # copies EVERYTHING
RUN npm install --production      # runs every time ANY file changes
CMD ["node", "server.js"]

You change one line in server.js. Docker sees that COPY . . has different input files. Cache busted. Which means npm install also reruns, even though your dependencies haven't changed. You wait 15 seconds for a zero-dependency-change build. Multiply that by 50 developers and 20 builds a day and you've burned hours.

Now the good version:

FROM node:20-alpine
WORKDIR /app
COPY package.json .               # dependencies change rarely
RUN npm install --production      # cached unless package.json changed
COPY server.js .                  # app code changes often, goes LAST
CMD ["node", "server.js"]

You change server.js. Docker checks: did package.json change? No. Cached. Did npm install inputs change? No. Cached. Only COPY server.js . reruns. Build time: sub-second.

This is not a micro-optimization. This is a 10x difference in build time. I have seen real CI pipelines go from 4-minute builds to 25-second builds by fixing instruction order alone. No new hardware. No fancy caching service. Just understanding how layers work.

The rule is simple: things that change rarely go at the top. Things that change often go at the bottom. Dependency manifests before source code. System packages before application packages. Always.

4. Multi-Stage Builds: It's Not About Size

Everyone explains multi-stage builds as "make your image smaller." That's true but it's the wrong framing. Multi-stage builds are about separation of concerns.

Your build environment needs compilers, package managers, build tools, debug symbols. Your production environment needs none of that. Every tool in your production image is a tool an attacker can use. Every extra binary is attack surface. Multi-stage builds let you use a full workshop to build, then ship only the finished product.

Here's a Go application using a multi-stage build:

FROM golang:1.22-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o server

FROM alpine:3.20
RUN adduser -D appuser
COPY --from=builder /app/server /server
USER appuser
CMD ["/server"]

Stage 1 (builder): Uses golang:1.22-alpine, which includes the full Go compiler, linker, and standard library. go.mod and go.sum are copied first so go mod download can be cached independently of source changes (the same caching logic as Node's dependency layer). CGO_ENABLED=0 produces a fully static binary with no dynamic library dependencies, so the binary runs on any Linux-based image. If your project has no external modules, drop the go.sum copy and go mod download line, but you still need go.mod (Go 1.17+ refuses to build outside a module).

Stage 2 (production): Starts fresh from alpine:3.20. Creates a non-root user. Copies only the compiled binary from stage 1. The Go compiler, source code, intermediate objects, all left behind. They never make it into the final image.

You can verify the size difference with docker images after building both. In practice, a static Go binary on top of Alpine typically weighs in around 10 to 20 MB, while a Node image carrying the full runtime and node_modules is usually well over 100 MB. The exact numbers depend on your dependencies, but the order-of-magnitude gap is consistent.

The more important win is attack surface. A Node image carries npm, a shell, a package manager, and dozens of utilities. If someone exploits a vulnerability in your app and gets code execution, they have tools to work with. A Go image on Alpine is far more restricted: running as a non-root user means no ability to install packages even though apk is present, and far fewer pre-installed utilities to abuse. For maximum restriction, use a distroless or scratch base in stage 2 and your production image won't even have a shell.

Smaller images aren't just faster to pull. They have less attack surface.

For compiled languages (Go, Rust, C, C++), multi-stage builds are non-negotiable. For interpreted languages (Node, Python, Ruby), you can't eliminate the runtime, but you can still use multi-stage to install build-time-only dependencies in a separate stage and copy only production artifacts forward.

5. docker debug: Your New Best Friend

Note: docker debug is a Docker Desktop feature and requires a paid Docker subscription (Pro, Team, or Business). It is not included in open-source Docker Engine / Docker CE. Minimum Docker Desktop version is 4.33. Check with docker debug --help before relying on it.

Here's a scenario: your multi-stage build produces a final image based on alpine:3.20 or even scratch (a completely empty image). Something's wrong at runtime. You want to shell in and look around. But there's no shell in the image. docker exec -it mycontainer sh fails.

Enter docker debug:

docker debug d3-go

docker debug attaches a debug shell to any container or image, even distroless, even scratch-based, even stopped containers. It injects a temporary toolbox with a shell, common utilities, and diagnostic tools without modifying the target image.

This is a game-changer for production debugging. You built a minimal, secure image. Now you can still inspect it when things go sideways without compromising your security posture by shipping a shell in production.

You can also use it to inspect any image directly:

docker debug myapp:latest

This drops you into the filesystem of that image. You can poke around, check what files exist, verify environment variables, and understand the exact state of what you shipped. No more adding RUN ls -la to your Dockerfile and rebuilding 14 times.

6. What Nobody Tells You

Every RUN, COPY, and ADD instruction creates a filesystem snapshot. That's not a metaphor. Docker literally takes a diff of the filesystem before and after each instruction and stores it as a compressed tar layer. Metadata instructions (ENV, EXPOSE, CMD, USER, LABEL) don't touch the filesystem; they just attach metadata to the image and appear as zero-byte entries in docker history.

This means the order of your instructions isn't just a style preference. It's an architecture decision. Reorder your instructions and build time changes by 10x. Put your COPY . . in the wrong place and you invalidate your entire cache on every single commit. Put a RUN rm -rf /tmp/build after a RUN make install and the deleted files still exist in the previous layer (they're just marked as deleted in the new one, and the image doesn't shrink).

Once you internalize that a Dockerfile is a sequence of filesystem snapshots, not a shell script, everything about Docker image optimization clicks into place. Layer ordering, cache invalidation, multi-stage builds, .dockerignore: they're all consequences of this one mental model.

7. Quick Reference

Core Dockerfile Instructions

Build Commands You'll Actually Use

docker build -t myapp .                     # standard build
docker build --no-cache -t myapp .          # force full rebuild
docker build -f Dockerfile.prod -t myapp .  # use a specific Dockerfile
docker init                                 # generate a production Dockerfile
docker history myapp                        # inspect layer sizes
docker debug mycontainer                    # shell into anything (Docker Desktop, paid sub)

The Cache Rule

Things that change RARELY   ->  top of Dockerfile    ->  cached forever
Things that change SOMETIMES ->  middle               ->  cached usually
Things that change OFTEN     ->  bottom               ->  rebuilt quickly

What's Next: Day 4

You can now build production images that are small, secure, and fast to rebuild. But containers are ephemeral. When they stop, their writable layer vanishes. Your database, your uploads, your logs: gone.

On Day 4: Docker Volumes & Persistent Storage, we tackle the hard problem: making data survive container death. Volumes, bind mounts, named volumes, and the patterns that keep your data safe when everything else is disposable.

See you tomorrow.

https://www.youtube.com/watch?v=LLuUhU3SwJo&t=4s

TL;DR, the 23 steps

1. kubectl parses argv and builds a minimal Pod object.

2. It reads ~/.kube/config for cluster, user, and context.

3. A TCP connection is opened to the API server. TLS 1.3 negotiates keys in one round trip with mutual cert auth.

4. kubectl sends POST /api/v1/namespaces/default/pods with a JSON body over HTTP/2.

5. The API server authenticates the caller (x509, bearer token, OIDC, or webhook).

6. It authorizes the request against RBAC. Can this user create pods in default?

7. Mutating admission runs. ServiceAccount injects a projected token volume, LimitRanger fills in default requests and limits, and so on.

8. The API server defaults missing fields (DNS policy, restart policy, termination grace period) and then validates against the OpenAPI schema.

9. Validating admission runs. ResourceQuota, PodSecurity, any ValidatingAdmissionWebhook, and the built in ValidatingAdmissionPolicy CEL engine (GA since 1.30).

10. The API server writes to etcd via Raft. Leader replicates, followers fsync, a majority acks, and only then does the pod exist.

11. The API server returns 201 Created. kubectl prints pod/nginx created.

12. Watch fanout. Every component holding an open watch stream (scheduler, kubelets, controllers) is notified within milliseconds.

13. The scheduler runs Filter plugins. NodeResourcesFit, NodeAffinity, TaintToleration, PodTopologySpread, VolumeBinding.

14. It runs Score plugins. NodeResourcesBalancedAllocation, ImageLocality, InterPodAffinity, NodeAffinity.

15. The winning node gets picked. Scheduler POSTs to /pods/nginx/binding, which updates spec.nodeName. One more etcd write.

16. The kubelet on that node sees the bound pod through its watch. syncPod fires.

17. Kubelet calls the container runtime over CRI (RunPodSandbox). containerd creates the pause container, PID 1, calling pause(2) and holding the pod's network namespace.

18. The CNI plugin (Calico, Flannel, Cilium, your choice) runs ADD. It creates a veth pair, allocates an IP from the pod CIDR, programs routes.

19. Image pull. containerd fetches the manifest, then the layers, verifying each with SHA-256.

20. Container create. The runtime stacks image layers with overlayfs, writes the OCI runtime spec, and asks runc to create.

21. runc takes over. clone3 with namespace flags (PID, mount, UTS, IPC), setns into the sandbox's network namespace, mount /proc, pivot_root, drop capabilities, apply the seccomp filter, execve nginx.

22. Kubelet's PLEG notices the container started. Most clusters still poll the runtime every second. Evented PLEG is the newer event stream version but it is still alpha in 1.36, so don't assume it is on.

23. The status manager patches pod.status to Running back to the API server. Done.

Setting the stage

I teach Kubernetes on the Kubesimplify YouTubeouTube channel, and I still get asked the same question in workshops. What actually happens when I run kubectl run? Most answers stop at "the API server writes to etcd and the scheduler picks a node." That is true, but it is the one line summary of a story that has twenty-three chapters.

So this post is the long form of the six-minute video I just shipped, paired with an interactive site you can scrub through step by step. If you are a platform engineer who already knows what a pod is, my goal is that by the end of this you can name the plugins, the syscalls, the admission chain order, and the CRI calls. And you should be able to point at the Kubernetes source tree when you need to go deeper.

Everything below is checked against Kubernetes 1.36.0, which shipped on April 22, 2026. Where a feature gate matters, I call the version out explicitly.

Phase 1, the client side (kubectl)

Step 1: kubectl parses your command

kubectl run is a subcommand whose job is to take sparse user input and build a valid Pod object. The code lives in staging/src/k8s.io/kubectl/pkg/cmd/run/run.go. For kubectl run nginx --image nginx, the object kubectl builds locally is roughly this.

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
    - name: nginx
      image: nginx

So notice what is not there. No restartPolicy, no dnsPolicy, no terminationGracePeriodSeconds, no serviceAccountName, no imagePullPolicy. kubectl deliberately sends a minimal object. All those fields are filled in by the API server during defaulting, which happens after admission and before validation. This is the first real insight. The object you POST and the object etcd ends up storing, they are not the same.

Step 2: Reading kubeconfig

kubectl needs to know where to send the request. It reads ~/.kube/config (or whatever $KUBECONFIG points at) and resolves three things. The cluster (API server URL, CA bundle), the user (client certs, token, exec plugin), and the context (which cluster and user pair plus a default namespace). The logic sits in client-go/tools/clientcmd. If you run kubectl --v=8, you can watch this resolution happen inline.

Step 3: TCP plus TLS 1.3 handshake

kubectl opens a TCP connection to the API server on port 6443 and runs a TLS 1.3 handshake. TLS 1.3 is important here. It negotiates keys in a single round trip (TLS 1.2 needed two), and it does so with mutual authentication when you are using a client certificate. Both sides present certs, both sides verify against a CA. Same primitives your browser uses, nothing exotic. But worth noticing because every subsequent byte rides this mTLS tunnel.

Step 4: HTTP/2 POST to the API server

kubectl serializes the pod object to JSON, not YAML. YAML is a client side convenience, the wire format is JSON by default. Then it sends POST /api/v1/namespaces/default/pods over HTTP/2. Content-Type is application/json. HTTP/2 matters because all the watch streams later in the story will multiplex over the same connection.

Step 5: Request lands at the API server

The request hits kube-apiserver. The code path is the generic API server filter chain in staging/src/k8s.io/apiserver/pkg/server/filters. Every inbound request goes through the same stack of filters in order. Panic recovery, request deadline, auditing, authentication, impersonation, authorization, admission, validation. Most of the next phase is those filters.

Phase 2, the API server gate

Step 6: Authentication, "who are you?"

So the API server asks the first question. Who are you? The API server has four authenticator backends chained together. x509 client certificates, bearer tokens (static, service account, or OIDC), OIDC directly (with JWT verification against the configured issuer), and authentication webhooks (the TokenReview API). The first one that returns a positive identity wins.

For kubectl with a standard kubeconfig, you are usually on x509. The cert you presented in the TLS handshake is reused to populate user.Info with the CN as the username and the O values as groups. Code: staging/src/k8s.io/apiserver/pkg/authentication.

Step 7: Authorization, "can you do this?"

With identity established, the next question. Can this user perform create on the resource pods in the namespace default? The default authorizer is RBAC, backed by Role, ClusterRole, RoleBinding, ClusterRoleBinding objects. Multiple authorizers can be chained. In managed clusters you will often see Node,RBAC. The Node authorizer restricts what a kubelet can ask for, RBAC handles everything else. A single "allow" is enough. Explicit denies don't exist in RBAC.

Step 8: Mutating admission

This is the fun one. Mutating admission plugins run first, before schema validation, and they are allowed to change the object. Built-in mutators that fire for a pod create include:

• ServiceAccount. Injects the projected service account token volume and the automountServiceAccountToken default.

• DefaultStorageClass, DefaultTolerationSeconds, PodNodeSelector, RuntimeClass, depending on cluster config.

• LimitRanger. Applies default resources.requests and limits when a LimitRange exists in the namespace.

• Every MutatingAdmissionWebhook you have registered. Service meshes like Istio inject their sidecar here.

• MutatingAdmissionPolicy. The CEL based in-process alternative to webhooks. This went GA (v1) in 1.36, so you no longer need a feature gate for the stable path.

Each plugin runs sequentially. The order that ships in the API server defaults matters. ServiceAccount before LimitRanger, for example. Source: plugin/pkg/admission in kubernetes/kubernetes.

Step 9: Schema validation

After mutation, the API server defaults remaining missing fields (restartPolicy: Always, dnsPolicy: ClusterFirst, terminationGracePeriodSeconds: 30, serviceAccountName: default) and validates the now complete object against the OpenAPI v3 schema published at /openapi/v3. Invalid names, empty required fields, wrong field types, all rejected here with a 422 Invalid.

Step 10: Validating admission

Validating admission is a second admission pass that cannot mutate. Built-ins include:

• ResourceQuota. Do the namespace's quotas have room for this pod's requests?

• PodSecurity. Does the pod meet the restricted, baseline, or privileged profile the namespace is labeled with?

• Every ValidatingAdmissionWebhook you have registered.

• ValidatingAdmissionPolicy. CEL based in-process validation, GA since 1.30. A great replacement for Kyverno or OPA in many cases.

So here is the subtle bit. Mutating admission runs before validating admission. If a user's webhook mutates a field, the validating chain sees the mutated value, not the original. This ordering is easy to get wrong in your head, and it matters when you are writing policy.

Step 11: etcd plus Raft quorum

Now the API server persists the pod. This is not a plain disk write. etcd is a Raft replicated key value store. The leader appends the entry to its Raft log, replicates to followers, each node fsyncs to disk, and only after a majority (3 of 5 in a typical HA setup) acks does the leader commit. The API server's storage layer blocks on that commit.

So if you ever see API latency spike, it is almost always etcd disk latency. Check etcd_disk_wal_fsync_duration_seconds. This is really, really important to know when you are debugging a slow cluster.

Step 12: 201 Created

The API server responds 201 Created with the full defaulted and mutated pod object in the body. kubectl prints:

pod/nginx created

From your terminal's perspective, it is done. From the cluster's perspective, the real work has not started.

Phase 3, the control loop hands off

Step 13: Watch fanout

Every long running component in Kubernetes holds an HTTP/2 watch stream to the API server. The scheduler watches unscheduled pods. Every kubelet watches pods bound to its node. Controllers watch their respective resources.

So when a new pod is written to etcd, the API server's watch cache broadcasts the event to all subscribers. No polling, no round trips, just a chunked HTTP/2 frame per event. Milliseconds. Source: staging/src/k8s.io/apiserver/pkg/storage/cacher.

Step 14: Scheduler, Filter

kube-scheduler receives the event. The pod has no spec.nodeName, so it is scheduler's problem. The scheduler runs a configurable pipeline of plugins, grouped into extension points. PreFilter, Filter, PostFilter, PreScore, Score, Reserve, Permit, PreBind, Bind, PostBind. For filter:

• NodeResourcesFit. The node has enough allocatable CPU, memory, and ephemeral storage for the pod's requests.

• NodeAffinity. The pod's nodeAffinity and nodeSelector match the node's labels.

• TaintToleration. The pod tolerates the node's taints.

• PodTopologySpread. The placement respects any topology spread constraints.

• VolumeBinding. All unbound PVCs can be bound to volumes reachable from this node.

• InterPodAffinity (at the filter level for hard constraints).

Any node that fails any filter is eliminated. Plugin source: pkg/scheduler/framework/plugins.

Step 15: Scheduler, Score

Surviving nodes get scored by a second set of plugins.

• NodeResourcesBalancedAllocation. Prefers nodes with balanced CPU and memory utilization, so you don't pack a CPU heavy pod onto an already CPU saturated node.

• ImageLocality. Prefers nodes that already have the container image cached locally. This saves image pull time.

• InterPodAffinity. Soft affinity and anti-affinity preferences.

• NodeAffinity. Soft (preferred) affinity terms.

• TaintToleration. Soft toleration scoring.

Each plugin returns a score 0 to 100 per node. Scores are normalized, weighted, and summed. Highest total wins. Ties are broken with a random pick using Go's rand.Int().

One thing to flag here. Kubernetes 1.36 graduated Dynamic Resource Allocation (DRA) to GA. If you are scheduling GPU workloads or other devices through DRA, the scheduler's resource claim handling is now stable. Worth reading the KEP if you are running AI workloads.

Step 16: Scheduler, Bind

The scheduler POSTs to the binding subresource. POST /api/v1/namespaces/default/pods/nginx/binding with target.name=node-1. This is what actually updates spec.nodeName in etcd. One more Raft write.

So here is a fun detail. The scheduler never writes spec.nodeName directly on the pod. It always goes through binding. This exists precisely because binding is a separate privilege you can RBAC.

Phase 4, the kubelet brings the pod to life

Step 17: Kubelet SyncPod

Kubelet on the bound node has been watching pods?fieldSelector=spec.nodeName=node-1 since startup. It sees the update, runs its pod admission checks (eviction pressure, kubelet level PodSecurityContext sanity), and calls syncPod in pkg/kubelet/kubelet.go. SyncPod is the reconciliation loop. It compares the desired pod spec with the current runtime state and issues CRI calls to bring them into alignment.

Step 18: CRI, sandbox and the pause container

Before any app container runs, the kubelet creates a pod sandbox. It calls RunPodSandbox over the CRI gRPC API on the runtime's socket (/run/containerd/containerd.sock by default). containerd launches the pause container. A tiny statically linked binary whose entire job is to call pause(2) and block forever as PID 1.

But why? Because the pause container is what owns the pod's Linux namespaces, especially the network namespace. When you add more containers to the pod, they setns into the pause container's namespaces. If an app container dies and restarts, the namespaces (and the IP) survive because pause is still there.

Step 19: CNI, pod gets networking

With the sandbox up, the runtime invokes the CNI plugin specified in /etc/cni/net.d/*.conflist (whichever is lexically first). Calico, Flannel, Cilium, Weave, the plugin you installed. CNI's contract is simple. A binary that reads JSON from stdin, takes an action (ADD, DEL, CHECK), and returns JSON to stdout. The ADD call:

1. Creates a veth pair. One end in the pod's network namespace, one end on the node.

2. Allocates an IP from the pod CIDR. IPAM is either a local store, Kubernetes IPAM, or an external controller.

3. Programs routes and iptables or eBPF rules on the host.

4. Optionally sets up sysctls inside the pod's netns.

When this returns, kubectl get pod -o wide will start showing podIP.

Step 20: Image pull

Kubelet calls PullImage over CRI. containerd resolves the reference (nginx to docker.io/library/nginx:latest), fetches the manifest, then pulls each layer in parallel, verifying SHA-256 digests on every chunk. First pull for a popular image over broadband is a few seconds. Cached? About 100 ms. containerd just revalidates the manifest and returns.

Step 21: Container create

With the image unpacked, the runtime assembles the container.

• Stacks the image layers as read only lower layers and adds a writable upper layer using overlayfs. The result is the container's rootfs.

• Writes the OCI runtime spec (config.json). A JSON document describing every mount, every namespace flag, every capability, the seccomp profile, the apparmor profile, the cgroup limits, the user, the entrypoint.

• Creates a bundle directory containing the rootfs and config.json and hands it to runc with runc create.

OCI runtime spec lives in the opencontainers/runtime-spec repo. This is the same spec Podman, CRI-O, and gVisor use. It is the portability boundary.

Phase 5, runc, namespaces, and the first breath

Step 22: runc

So this is the single coolest part of the whole pipeline. runc takes the bundle and does the following.

1. Calls clone3 with flags CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWUTS | CLONE_NEWIPC. On a modern kernel, clone3 is preferred over the older clone because it takes a structured argument and supports more namespace flags cleanly. The network namespace is not created here. Instead, runc uses setns to enter the sandbox's network namespace that CNI created earlier, so the new container shares the pod IP.

2. Inside the new process, mounts /proc for the new PID namespace.

3. pivot_root into the overlayfs rootfs, then unmounts the old root.

4. Drops Linux capabilities to the OCI spec's bounding set. The default for a non-privileged container is a tight whitelist. No CAP_SYS_ADMIN, no CAP_NET_ADMIN.

5. Applies the seccomp filter. The runtime default profile blocks around 40 syscalls, like kexec_load, certain unshare flags, and bpf without capability.

6. Joins the cgroup v2 hierarchy with the configured CPU and memory limits.

7. Calls execve on the container's entrypoint, nginx -g daemon off;. execve is the syscall that replaces the current process image with a new program while keeping the PID. This is the moment nginx is alive.

If you strace -f runc during create, you will see this whole dance. It is worth doing once.

Step 23: PLEG and the Running status

Kubelet needs to know the container started. Historically, kubelet's PLEG (Pod Lifecycle Event Generator) polled the runtime every second via ListContainers, diffed the result, and emitted events. On a big node with hundreds of pods, this was a measurable source of CPU load.

So there is a newer path called Evented PLEG. It opens a CRI event stream (ContainerEventsRequest) so containerd pushes events like CONTAINER_STARTED_EVENT and CONTAINER_STOPPED_EVENT as they happen. But here is the thing. Evented PLEG is still alpha in 1.36. It was alpha in 1.25, promoted to beta in 1.27, then reverted to alpha in 1.30 after a static pod bug. It is disabled by default. So if you are reading kubelet code today, assume the polling path is what is actually running on your cluster.

When kubelet sees a new container has started (through polling or evented), the status manager computes the pod's phase as Running and patches pod.status back to the API server via a JSON merge patch. Watchers (you, with kubectl get pod -w) see the transition. The status patch is also the signal to any controller waiting on this pod. For example, the endpoints controller, which is about to add the pod's IP to a Service's EndpointSlice.

And that is the whole journey. From argv[1] in your shell to nginx serving on port 80, about a second on a warm cluster.

Further reading

• kubernetes/kubernetes. The source tree. Start in pkg/kubelet, pkg/scheduler, staging/src/k8s.io/apiserver.

• CRI spec. The gRPC contract between kubelet and the runtime.

• CNI spec. The plugin contract for pod networking.

• OCI runtime spec. The container bundle and config format runc consumes.

• OCI image spec. Manifests, layers, and the SHA-256 content addressable model.

• KEP-3386 Evented PLEG. The design doc for the CRI event driven PLEG, still alpha in 1.36.

• kube-scheduler plugin docs. The official list of in-tree plugins and their extension points.

Watch and play

• Video (6 min): What Actually Happens When You Run kubectl run nginx (23 Steps) on the Kubesimplify YouTube channel.

• Interactive site: kubernetes-explained.vercel.app/pod. Pause, scrub, jump to any step, copy the code for yourself.

So if you liked this, the next one in the series is the scheduler deep-dive. How kube-scheduler actually decides. Subscribe on the channel so you catch it, and tell me in the comments which step surprised you. That is how I know what to unpack next.

7 Days of Docker (2026) - by Saloni Narang, Docker Captain & CNCF Ambassador

I'm a Docker Captain. I've seen hundreds of Docker tutorials explain images as "blueprints" or "templates" and then move on. That's not good enough anymore. In March 2026, the tool you use to scan for vulnerabilities was the vulnerability - attackers pushed backdoored Trivy scanner images to Docker Hub, and thousands of CI/CD pipelines had their secrets stolen before anyone noticed.

If you don't understand what an image actually is, where it comes from, and how to verify it, you're not just writing bad Dockerfiles. You're leaving the door open.

Today, we fix that.

What IS an Image? (Not "Layers Like a Cake")

Forget the analogies. Here's what actually happens.

A Docker image is an OCI artifact. It consists of:

1. A manifest - a JSON document listing references to filesystem diffs and configuration

2. Blobs - compressed tarballs containing filesystem changes

3. A config - JSON metadata (environment variables, entrypoint, exposed ports)

When you run docker pull nginx:alpine, Docker contacts a registry, downloads that manifest, then fetches the blobs it doesn't already have. That's it. There is no magic.

The OCI (Open Container Initiative) standardized this format so that images are portable across any compliant runtime — Docker, Podman, containerd, you name it. An image is not a Docker-specific thing. It's a distribution format for filesystem snapshots.

What Nobody Tells You: The word "image" is misleading. There is no single file. An image is a collection of independently addressable, content-hashed blobs assembled by a manifest. When you "pull an image," you are downloading a graph of content-addressed objects. Understanding this changes how you think about caching, sharing, and security.

Layers - The Real Story

Every instruction in a Dockerfile produces a filesystem snapshot. Docker uses a union filesystem (OverlayFS on Linux) to stack these snapshots and present them as a single coherent filesystem to the container.

Here's what this means in practice:

• Each layer is a diff. It records what files were added, modified, or deleted compared to the layer below it.

• Layers are content-addressed. Each one has a SHA256 digest. Same content = same hash = stored once.

• Copy-on-write. When a container starts, Docker adds a thin writable layer on top. Reads fall through to the image layers. Writes get captured in the writable layer. The image layers are never touched.

This is why docker pull says "Already exists" for layers you already have from another image. If nginx:alpine and redis:7-alpine share the same Alpine base layer, Docker stores it once, and both images reference it.

Let's see this with a real image. Pull nginx:alpine and then inspect its history:

docker pull nginx:alpine

alpine: Pulling from library/nginx
d17f077ada11: Pull complete
910c2a6cad6d: Pull complete
a89d14ef5abe: Pull complete
a96b658a00fe: Pull complete
10cbc192f783: Pull complete
634f1d1ce0f7: Pull complete
83fbf849ee89: Pull complete
662c8d6f6620: Pull complete
Digest: sha256:5616878291a2eed594aee8db4dade5878cf7edcb475e59193904b198d9b830de
Status: Downloaded newer image for nginx:alpine
docker.io/library/nginx:alpine

Now look at how the image was built:

docker history nginx:alpine

IMAGE          CREATED      CREATED BY                                      SIZE
7f7dcd27f920   6 days ago   RUN /bin/sh -c set -x && apkArch="$(cat ...    48.3MB
<missing>      6 days ago   CMD ["nginx" "-g" "daemon off;"]                0B
<missing>      6 days ago   STOPSIGNAL SIGQUIT                              0B
<missing>      6 days ago   EXPOSE map[80/tcp:{}]                           0B
<missing>      6 days ago   ENTRYPOINT ["/docker-entrypoint.sh"]            0B
<missing>      6 days ago   COPY 30-tune-worker-processes.sh...             4.62kB
<missing>      6 days ago   RUN /bin/sh -c set -x && addgroup -g 101...    4.51MB
<missing>      6 days ago   ENV NGINX_VERSION=1.29.8                        0B
<missing>      6 days ago   CMD ["/bin/sh"]                                 0B
<missing>      6 days ago   ADD alpine-minirootfs-3.23.3-aarch64.tar.gz    8.7MB

Read bottom-to-top. The base Alpine filesystem is 8.51 MB. The nginx install adds 40.5 MB. CMD, ENV, EXPOSE — those create 0-byte metadata-only layers. The <missing> entries mean those layers were built on a remote build server, which is completely normal for pulled images.

What Nobody Tells You: Not every Dockerfile instruction creates a layer that takes disk space. Only RUN, COPY, and ADD produce filesystem changes. Everything else (CMD, ENV, EXPOSE, LABEL, ENTRYPOINT) is metadata written into the image config JSON. When you see "0B" in docker history, that's why.

The Supply Chain Problem

Images come from registries. Registries can be compromised. And in March 2026, they were.

Attackers pushed backdoored versions of the Trivy vulnerability scanner to Docker Hub. Trivy — the tool that organizations run in their CI/CD pipelines to detect compromised images — was itself compromised. The backdoored images exfiltrated environment variables, secrets, and CI tokens from every pipeline that pulled them. Thousands of organizations were affected before the images were pulled down.

Think about that. The security tool was the attack vector.

This isn't a hypothetical. This happened. And it happened because most teams treat docker pull it like apt install - They assume the registry is trustworthy and the image is what it claims to be.

Your images are a supply chain. Every FROM in your Dockerfile, every base image you pull, every tool you run in CI — it's a link in that chain. One compromised link and everything downstream is exposed.

Docker Scout: Not Optional in 2026

After March 2026, image scanning is not a "nice to have." Docker Scout is built into the Docker CLI and gives you visibility into what's inside your images.

Start with a quick overview:

docker scout quickview nginx:alpine

 Target             │  nginx:alpine            │    0C     2H     9M     1L     1?
   digest           │  7f7dcd27f920            │
 Base image         │  nginx:1-alpine-slim     │    0C     0H     1M     0L
 Updated base image │  nginx:1.30-alpine-slim  │    0C     0H     1M     0L

Zero critical. 2 high. 9 medium. This is actually a good result -> nginx:alpine was just patched days ago. But run this same scan on an image you haven't updated in 3 months and watch the numbers explode. The point is: you wouldn't know without scanning.

Let's dig into what's still there:

docker scout cves nginx:alpine --only-severity critical,high

## Packages and Vulnerabilities

   0C     1H     0M     0L  nghttp2 1.68.0-r0

    ✗ HIGH CVE-2026-27135
      https://scout.docker.com/v/CVE-2026-27135
      Affected range : <=1.68.0-r0
      Fixed version  : not fixed

   0C     1H     0M     0L  curl 8.17.0-r1

    ✗ HIGH CVE-2026-3805
      https://scout.docker.com/v/CVE-2026-3805
      Affected range : <=8.17.0-r1
      Fixed version  : not fixed

2 vulnerabilities found in 2 packages
  CRITICAL  0
  HIGH      2

Even a freshly-pulled, just-patched image has 2 high-severity CVEs, and both say "not fixed" yet. These are zero-day-adjacent vulnerabilities in curl and nghttp2, sitting in every nginx:alpine container on the planet right now. Imagine what your 6-month-old base image looks like.

Docker Scout generates SBOMs (Software Bills of Materials), tracks CVEs across your image catalog, and integrates with CI. If you're not running this, you're flying blind.

Docker Hardened Images (DHI)

Docker released Hardened Images as a direct response to the supply chain crisis. Here's what they offer:

• 95% fewer CVEs compared to standard Docker Hub images

• Rootless by default - no process runs as root

• Distroless runtime - minimal attack surface, no shell, no package manager

• 7-day fix guarantee - critical CVEs patched within a week

• 1000+ images available and growing

• Free and open source under Apache 2.0

In practice, switching from nginx:alpine to a Docker Hardened Image means you inherit a fraction of the vulnerability surface. For anything running in production, DHI should be your default starting point.

Pro Tip: DHI images are distroless at runtime - there is no shell to docker exec into for debugging. During development, use the standard image. In your production Dockerfile stage, switch to the DHI variant. Multi-stage builds (Day 3) make this trivial.

Image Naming - More Than You Think

The full format of an image reference:

registry/namespace/repository:tag@sha256:digest

These are all equivalent:

nginx
nginx:latest
library/nginx:latest
docker.io/library/nginx:latest

Tags are mutable. Digests are immutable. This distinction matters more than almost anything else in this post.

What Nobody Tells You: An image tagged :latest today and :latest tomorrow can be completely different binaries. Tags are pointers that can be moved. Anyone with push access can retag an image at any time. Digests are content hashes that cannot be faked. In production, pin sha256 digests. In the March 2026 Trivy attack, the malicious images were pushed under existing tags — anyone pulling by tag got the backdoor. Anyone pinning by digest was unaffected.

Hands-On: Inspect, Scan, Understand

Let's put it all together. Run these commands and understand what each one tells you.

Check what's on disk:

docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}" | head -15

REPOSITORY    TAG         SIZE
node          20-alpine   136MB
nginx         alpine      53.4MB
ubuntu        latest      101MB
hello-world   latest      5.2kB
alpine        latest      8.51MB
redis         7-alpine    41.7MB

Alpine is 8.51 MB. Ubuntu is 101 MB. For a base image, that's a 12x difference in attack surface before you've installed anything.

Check disk usage across all Docker resources:

docker system df

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          86        17        22.49GB   5.294GB (23%)
Containers      41        0         366MB     366MB (100%)
Local Volumes   14        5         1.721GB   1.676GB (97%)
Build Cache     56        0         1.926GB   278.2kB

22.49 GB in images. 41 stopped containers taking 366 MB. 1.9 GB of build cache. Run docker system prune -a periodically, or this grows without bound.

Pull process visualized:

When you docker pull nginx:alpine:

1. Docker resolves nginx:alpine to docker.io/library/nginx:alpine

2. Contacts the registry, fetches the manifest

3. Checks each layer against local storage

4. Downloads missing layers in parallel (compressed)

5. Verifies every layer's SHA256 digest

6. Assembles the image locally

If any digest doesn't match, the pull fails. This is content-addressable storage doing its job - but it only protects you against tampering in transit, not a compromised image at the source.

Pro Tip: Need to pull for a different architecture? Use --platform: docker pull --platform linux/amd64 nginx:alpine. This is common when building on Apple Silicon for amd64 deployment targets.

Quick Reference

What's Next: Day 3

You now know what images are, how the supply chain works, and why scanning is non-negotiable. But you've only pulled images other people built.

On Day 3: Building Images with Dockerfiles, you'll write your own. You'll learn the build cache, layer ordering, multi-stage builds, and how to produce images that are small, fast, and don't ship with a shell an attacker can use.

See you tomorrow.

7 Days of Docker (2026) — A Docker Captain's guide. Not your average tutorial.

I'm a Docker Captain. I've seen many Docker tutorials on the internet. And they all start the same way:

"Docker is like a virtual machine, but lighter..."

No. Let's stop doing that.

I'm going to explain Docker the way I wish someone had explained it to me — from the inside out. No VM comparisons. No shipping container analogies. Just what's actually happening on your machine right now.

A Container Is Just a Process

That's it. That's the tweet.

When you run docker run nginx, you're not spinning up a virtual machine. You're not creating a miniature computer. You're starting a Linux process — the nginx binary — with two restrictions:

1. It can only SEE certain things (namespaces)

2. It can only USE certain resources (cgroups)

That's a container. A restricted process. Let me prove it.

docker run -d --name my-nginx nginx:alpine

Now let's look at it from the host's perspective:

docker top my-nginx

UID    PID    PPID   C   STIME   TTY   TIME       CMD
root   15290  15267  0   09:31   ?     00:00:00   nginx: master process nginx -g daemon off;
statd  15327  15290  0   09:31   ?     00:00:00   nginx: worker process
statd  15328  15290  0   09:31   ?     00:00:00   nginx: worker process
statd  15329  15290  0   09:31   ?     00:00:00   nginx: worker process
statd  15330  15290  0   09:31   ?     00:00:00   nginx: worker process
statd  15331  15290  0   09:31   ?     00:00:00   nginx: worker process
statd  15332  15290  0   09:31   ?     00:00:00   nginx: worker process
statd  15333  15290  0   09:31   ?     00:00:00   nginx: worker process
statd  15334  15290  0   09:31   ?     00:00:00   nginx: worker process

See those PIDs? Those are regular processes on your machine. One master, eight workers (one per CPU core on this M2). They show up in the process table just like any other program. They're not in a VM. They're not in a sandbox. They're processes.

So what makes it a "container"?

The Two Ingredients: Namespaces and Cgroups

Namespaces: What the process can SEE

Linux namespaces are walls. They control what a process is allowed to perceive:

When you're "inside" a container and type ps aux, you see maybe 2-3 processes. On the host, there are hundreds. The container doesn't know that. Its PID namespace hides everything else.

The container isn't isolated because it's in a separate machine. It's isolated because the kernel lies to it.

Cgroups: What the process can USE

Control groups limit resources:

docker run --memory=128m --cpus=0.5 nginx:alpine

This process can never use more than 128MB of RAM or half a CPU core. The kernel enforces this. The container starts and runs fine — nginx barely uses 10MB. But if it ever tries to allocate beyond 128MB, the kernel OOM-kills it. The limit is a ceiling, not a cage. Most processes never hit it. But a memory leak or a traffic spike that exhausts the limit? The process dies instantly. That's the point — one runaway container can't take down the entire host.

$ docker run --rm --memory=128m alpine cat /sys/fs/cgroup/memory.max
134217728

134,217,728 bytes. Exactly 128 megabytes. The kernel sets this limit at the cgroup level, and the process can never escape it.

That's it. Namespaces + Cgroups = Container. Docker didn't invent containers. Docker made them usable.

So What Does Docker Actually Do?

If containers are just kernel features, why do we need Docker?

Because doing this manually is miserable. Without Docker, to "containerize" a process, you'd need to:

• Call unshare() and clone() system calls to create namespaces

• Set up cgroup hierarchies in /sys/fs/cgroup/

• Build a root filesystem by hand

• Configure networking with veth pairs and iptables

• Handle image distribution yourself

Docker wraps all of this into one command:

docker run -d -p 8080:80 nginx:alpine

One line. Docker:

1. Pulls the nginx:alpine image (a packaged filesystem)

2. Creates namespaces (PID, NET, MNT, UTS, IPC)

3. Sets up cgroups for resource limits

4. Configures a virtual network interface

5. Mounts the image as a layered filesystem

6. Starts the process

$ docker run -d --name day1-nginx -p 8080:80 nginx:alpine
a45dc5a898449edc5e4ecfbca6fc3c22db6f77901e1114f26057c45ff3dacaa7

$ curl -s http://localhost:8080 | head -3
<!DOCTYPE html>
<html>
<head>

A web server. Running. In under a second. That hash? It's the container ID — a unique identifier for this particular set of namespaces and cgroups.

"But Wait — I'm on a Mac"

Here's something most tutorials never tell you:

Containers are a Linux kernel feature. macOS doesn't have a Linux kernel.

So how is Docker running on your Mac right now?

$ docker version
Client:
 Version:    29.1.5
 OS/Arch:    darwin/arm64

Server: Docker Desktop 4.58.1
 Engine:
  Version:   29.1.5
  OS/Arch:   linux/arm64

See that? Client: darwin/arm64 (your Mac). Server: linux/arm64 (not your Mac).

Docker Desktop runs a lightweight Linux virtual machine in the background. On Apple Silicon Macs, you get two choices for the Virtual Machine Manager (VMM):

• Apple Virtualization framework — the stable, well-established option that leverages Apple's native hypervisor. This is what most people use (and what I'm running right now).

• Docker VMM (Beta) — Docker's own container-optimized hypervisor, built specifically for Apple Silicon. Promises better performance for container workloads.

Both options also let you enable Rosetta for x86_64/amd64 emulation — so you can run Intel-based images on your ARM Mac without QEMU. And for file sharing between Mac and the Linux VM, you choose between VirtioFS (fastest, recommended), gRPC FUSE, or osxfs (legacy).

Your containers live inside this Linux VM. The flow:

1. You type docker run on your Mac (the client)

2. The command goes to the Docker daemon inside the Linux VM (the server)

3. The daemon creates namespaces and cgroups in that Linux kernel

4. Your container runs inside the VM

You never see the VM. You never configure it. But it's there. On a Linux server? No VM needed — containers talk directly to the host kernel.

What Nobody Tells You: Your Mac's fan spinning during Docker builds? That's the Linux VM doing the work. The VM has allocated CPU and memory from your Mac. If you're running 10 containers, they're all sharing one VM — not 10 separate machines. This is why Docker on Mac will never be as fast as Docker on Linux. The VM is the tax you pay.

2026 Update: Apple released their own container tool — Apple Containers — which takes the opposite approach: one lightweight VM per container instead of one shared VM. It boots in sub-second, has stronger isolation, and is written in Swift. It's interesting for macOS-native workflows, but Docker remains the standard for production, CI/CD, and cross-platform builds.

Your First Containers (For Real This Time)

Let's actually do this. Not hello-world — that's a toy. Let's run real software.

Run an entire operating system

docker run --rm ubuntu bash -c "cat /etc/os-release && uname -a"

PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
Linux d50e9bad40e9 6.12.65-linuxkit #1 SMP Thu Jan 15 14:58:53 UTC 2026 aarch64 aarch64 aarch64 GNU/Linux

Look at that kernel version: 6.12.65-linuxkit. That's the Linux VM's kernel. The Ubuntu "operating system" inside the container is sharing this kernel. There's no second kernel. Ubuntu here is just a filesystem — the binaries, libraries, and configs that make Ubuntu Ubuntu. The kernel comes from the host (or the VM on Mac).

This is why containers start in milliseconds instead of minutes. There's no OS to boot. It's just a process with a different filesystem view.

Run a web server

docker run -d --name web -p 8080:80 nginx:alpine

$ docker ps
CONTAINER ID   IMAGE          STATUS         PORTS                                     NAMES
a45dc5a89844   nginx:alpine   Up 26 seconds  0.0.0.0:8080->80/tcp, [::]:8080->80/tcp   web

curl -s http://localhost:8080 | head -5

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>

That -p 8080:80 is port mapping. The container's nginx listens on port 80 inside its network namespace. Docker maps your host's port 8080 to it. Traffic flows: your browser → host port 8080 → Docker networking → container port 80 → nginx process.

Check what's happening inside

docker logs web

/docker-entrypoint.sh: Configuration complete; ready for start up
2026/04/12 08:10:22 [notice] 1#1: using the "epoll" event method

Logs. Real logs from the nginx process. Because that's all a container is — a process.

Clean up

docker stop web && docker rm web

Container stopped (process received SIGTERM). Container removed (namespaces and cgroups cleaned up). The filesystem? Gone too, unless you used a volume. We'll cover that on Day 4.

2026: Docker Isn't Just Containers Anymore

Here's what separates learning Docker today from learning it in 2020.

docker init — Never write a Dockerfile from scratch

cd your-project/
docker init

Docker scans your project, detects the language (Python, Node.js, Go, Java, .NET), and generates a production-ready Dockerfile, compose.yaml, and .dockerignore. Multi-stage builds. Non-root users. Health checks. All generated in seconds.

docker ai — Ask Gordon

docker ai "How do I containerize a Flask app with Redis?"

Gordon is Docker's built-in AI assistant. It reads your project structure, your Dockerfile, your running containers — and gives you answers that actually understand your context. Not generic advice. Specific to your setup.

docker model — Run LLMs locally

$ docker model status
Docker Model Runner is running

$ docker model list
MODEL NAME  PARAMETERS  QUANTIZATION    ARCHITECTURE  SIZE
smollm2     361.82 M    IQ2_XXS/Q4_K_M  llama         256.35 MiB

Docker Model Runner lets you pull and run AI models the way you pull Docker images. Powered by llama.cpp, exposed via OpenAI-compatible API, GPU-accelerated on Apple Silicon. Think of it as docker run but for LLMs. We'll deep-dive on Day 6.

docker scout — Know your vulnerabilities

docker scout quickview nginx:latest

Every image has dependencies. Dependencies have CVEs. Docker Scout scans layers, generates SBOMs (Software Bill of Materials), and shows you what's exposed. In March 2026, compromised Trivy images on Docker Hub stole CI/CD secrets from thousands of pipelines. Security scanning isn't optional anymore.

docker debug — Shell into anything

docker debug my-broken-container

Container has no shell? No curl? No debugging tools? docker debug injects a full toolbox into any container or image — even distroless, even crashing. It brings vim, curl, htop, and more. You can install additional tools on the fly.

What You Actually Learned Today

Not "Docker is like a lightweight VM." You learned:

• A container is a Linux process restricted by namespaces (what it can see) and cgroups (what it can use)

• Docker automates the creation of these restrictions, plus image management, networking, and storage

• On Mac, Docker runs a Linux VM because macOS has no Linux kernel — your containers live inside that VM

• In 2026, Docker is also an AI platform: Model Runner for local LLMs, Gordon for AI assistance, Scout for security scanning, Debug for troubleshooting, Init for project scaffolding

Quick Reference

Tomorrow: Day 2

We're looking at images — not just "layers" but what they actually are: OCI artifacts containing filesystem snapshots. Why your images probably have dozens of vulnerabilities. How Docker Scout and Hardened Images change the game. And why the Trivy supply chain attack of March 2026 means security scanning is no longer optional.

I recently got my hands on an NVIDIA DGX Spark, and the first thing I wanted to figure out was: how do I access this thing from anywhere? Whether I'm at a coffee shop, at a conference, or on a different network entirely — I want to just ssh in and get to work.

The answer? Tailscale. It took me about 10 minutes to set up, and now I can SSH into my Spark from any device, on any network, anywhere in the world. I even set up a friend with access — simultaneously — without giving them my credentials. Here's exactly how I did it.

Why Tailscale?

Tailscale creates a private mesh network (called a "tailnet") between your devices. No port forwarding, no static IPs, no VPN server to maintain. You install it on your devices, log in with the same account, and they can talk to each other. It's built on WireGuard, so it's fast and encrypted.

For the DGX Spark, this means:

• No need to be on the same WiFi network

• No need to mess with your router settings

• Works behind NATs and firewalls

• Encrypted end-to-end

Prerequisites

Before starting, make sure your DGX Spark:

• Is running Ubuntu 24.04 or newer

• Has internet connectivity

• You have sudo access

Here's what my system looked like:

$ lsb_release -a
No LSB modules are available.
Distributor ID:    Ubuntu
Description:    Ubuntu 24.04.3 LTS
Release:    24.04
Codename:    noble

A quick ping to confirm internet:

$ ping -c 3 google.com
64 bytes from tzdela-ba-in-x0e.1e100.net: icmp_seq=1 ttl=118 time=15.3 ms
64 bytes from tzdela-ba-in-x0e.1e100.net: icmp_seq=2 ttl=118 time=13.7 ms
64 bytes from tzdela-ba-in-x0e.1e100.net: icmp_seq=3 ttl=118 time=17.2 ms

--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss

And verify sudo access:

$ sudo whoami
root

Good to go.

Step 1: Install Tailscale on the DGX Spark

SSH into your Spark (or use a directly connected keyboard/monitor) and run:

# Update package list and install prerequisites
sudo apt update
sudo apt install -y curl gnupg

# Add Tailscale signing key
curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/noble.noarmor.gpg | \
  sudo tee /usr/share/keyrings/tailscale-archive-keyring.gpg > /dev/null

# Add Tailscale repository
curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/noble.tailscale-keyring.list | \
  sudo tee /etc/apt/sources.list.d/tailscale.list

# Install Tailscale
sudo apt update
sudo apt install -y tailscale

You'll see the repository being added and the package installing:

# Tailscale packages for ubuntu noble
deb [signed-by=/usr/share/keyrings/tailscale-archive-keyring.gpg] https://pkgs.tailscale.com/stable/ubuntu noble main
...
Setting up tailscale (1.94.2) ...
Created symlink /etc/systemd/system/multi-user.target.wants/tailscaled.service → /usr/lib/systemd/system/tailscaled.service.

Verify the installation:

$ tailscale version
1.94.2
  tailscale commit: 0a29cf18b56e478b9cd33af07755fcae90d5171a
  long version: 1.94.2-t0a29cf18b-g3f044c9f6
  go version: go1.25.5

Check the service is running:

saiyam@spark-5223:~$ sudo systemctl status tailscaled --no-pager
[sudo] password for saiyam: 
● tailscaled.service - Tailscale node agent
     Loaded: loaded (/usr/lib/systemd/system/tailscaled.service; enabled; preset: enabled)
     Active: active (running) since Tue 2026-04-07 11:13:14 UTC; 9min ago
       Docs: https://tailscale.com/docs/
   Main PID: 2410 (tailscaled)
     Status: "Connected; saiyam911@gmail.com; 100.120.233.78 fd7a:115c:a1e0::f83a:e94e"
      Tasks: 22 (limit: 153561)
     Memory: 45.4M (peak: 53.7M)
        CPU: 615ms
     CGroup: /system.slice/tailscaled.service
             └─2410 /usr/sbin/tailscaled --state=/var/lib/tailscale/tailscaled.…

The status says "Needs login" — that's expected. We'll authenticate next.

Step 2: Connect the Spark to Your Tailnet

This is the magic step:

$ sudo tailscale up

To authenticate, visit:

    https://login.tailscale.com/a/1ff5e3e9017787

Open that URL in any browser, log in with your account (Google, GitHub, Microsoft — whatever your org uses), and you'll see:

Login successful. Your device spark-5223 is logged in

Back on the Spark terminal, you'll see:

Success.
Some peers are advertising routes but --accept-routes is false

That's it on the Spark side. Your DGX Spark is now part of your private Tailscale network with the hostname spark-5223.

Note: The --accept-routes message is harmless for SSH access. You can ignore it. If you ever need subnet routing, run sudo tailscale up --accept-routes.

Step 3: Install Tailscale on Your Laptop

macOS

• Option A: Download from the Mac App Store (search "Tailscale")

• Option B: Download the .pkg from tailscale.com/download

Open the app, click Log in, and sign in with the same account you used on the Spark.

Windows

1. Download the installer from tailscale.com/download

2. Run the .msi file

3. Launch Tailscale from the system tray

4. Log in with the same account

Linux

Same commands as the Spark:

sudo apt update
sudo apt install -y curl gnupg

curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/noble.noarmor.gpg | \
  sudo tee /usr/share/keyrings/tailscale-archive-keyring.gpg > /dev/null

curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/noble.tailscale-keyring.list | \
  sudo tee /etc/apt/sources.list.d/tailscale.list

sudo apt update
sudo apt install -y tailscale
sudo tailscale up

Step 4: SSH Into Your Spark From Anywhere

First, confirm both devices see each other:

$ tailscale status
100.104.142.22  spark-5223           saiyamxxx@  linux  -
100.108.115.75  saiyams-macbook-pro  saiyam9xxx@  macOS  -

You should see your Spark listed. Now, simply:

ssh saiyam@spark-5223

That's it. Tailscale's MagicDNS resolves spark-5223 to the right Tailscale IP automatically. No need to remember IP addresses.

If MagicDNS isn't working for some reason, use the Tailscale IP directly:

# Find the IP
tailscale status
# Look for spark-5223 and note the 100.x.x.x address

ssh saiyam@100.104.142.22

Setting Up SSH Key Authentication

For passwordless SSH access, set up key-based authentication. If you already have an SSH key (check ~/.ssh/id_ed25519.pub or ~/.ssh/id_rsa.pub), add it to the Spark:

# Copy your public key to the Spark (will ask for password once)
ssh-copy-id saiyam@spark-5223

Or manually add it on the Spark:

# On the Spark — append the public key
echo "your-public-key-here" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
chmod 700 ~/.ssh

After that, SSH works without a password prompt.

Note: Password authentication still works alongside SSH keys. You don't have to choose one or the other.

What About My Second Laptop?

This is the beauty of Tailscale — just install and log in:

1. Install Tailscale on the second laptop (using the steps above for your OS)

2. Log in with the same account

3. Run ssh saiyam@spark-5223

No extra configuration on the Spark. Every device on your tailnet can reach every other device automatically.

Sharing Your Spark With a Friend

What if a friend also needs SSH access to your Spark — simultaneously, from their own laptop? You don't need to create a new Tailscale account for them. Use a pre-auth key to add their device to your tailnet.

Generate a Pre-Auth Key

1. Go to the Tailscale Admin Console

2. Click "Generate auth key..."

3. Enable Reusable if you want it to work for multiple devices

4. Set an expiration as needed

5. Copy the key (starts with tskey-auth-...)

Your Friend's Setup (macOS)

1. Install Tailscale from the Mac App Store

2. Important: If they're already logged in to their own Tailscale account, they need to leave it first:

   sudo tailscale logout
   

3. Join your tailnet using the pre-auth key:

   sudo tailscale up --auth-key=tskey-auth-xxxxxxxxxxxx
   

4. That's it — their Mac is now on your tailnet. No login, no email needed.

Add Their SSH Key to the Spark

Your friend should generate an SSH key on their Mac (if they don't have one):

ssh-keygen -t ed25519

Then share their public key with you (the contents of ~/.ssh/id_ed25519.pub). On the Spark, add it:

echo "ssh-ed25519 AAAA...their-key-here... friend@hostname" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

Now your friend can SSH in directly:

ssh saiyam@spark-5223

No password prompt — the key handles authentication automatically. SSH automatically tries keys from the default location (~/.ssh/id_ed25519), so your friend does not need to use ssh -i.

Verify it all works:

$ tailscale status
100.104.142.22  spark-5223           saiyamxxx@  linux  -
100.67.209.38   rohits-macbook-pro   saiyamxxx@  macOS  -
100.108.115.75   saiyams-macbook-pro  saiyamxxx@  macOS  -

Three devices, one tailnet, simultaneous SSH access.

Tip: You can manage access from the Tailscale Admin Console. To revoke someone's access, remove their device from the console and delete their key from ~/.ssh/authorized_keys on the Spark.

Troubleshooting

"No Matching Peer" Error

If your friend gets a "no matching peer" error when trying to SSH, it means they're on a different tailnet — not yours.

The 100.x.x.x Tailscale IPs are only reachable between devices on the same tailnet. The fix:

# Friend logs out of their own tailnet
sudo tailscale logout

# Friend joins YOUR tailnet with your pre-auth key
sudo tailscale up --auth-key=tskey-auth-xxxxxxxxxxxx

SSH Connection Timeout

If tailscale ping works but SSH times out:

# On the Spark — check SSH is running
sudo systemctl status ssh

# Check firewall isn't blocking
sudo ufw status

# If SSH isn't running
sudo systemctl start ssh

# If firewall is active and blocking
sudo ufw allow 22/tcp

Also check SSH is listening on all interfaces:

$ ss -tlnp | grep 22
LISTEN  0  4096  0.0.0.0:22  0.0.0.0:*  users:(("sshd",...))
LISTEN  0  4096     [::]:22     [::]:*  users:(("sshd",...))

If SSH is only listening on a specific IP, edit /etc/ssh/sshd_config to ensure ListenAddress is not restricted, then sudo systemctl restart ssh.

Permission Denied (publickey, password)

This means SSH connected but authentication failed. Either:

• Your SSH key isn't in ~/.ssh/authorized_keys on the Spark

• You're using a non-default key path (use ssh -i /path/to/key)

• Password authentication is disabled in sshd_config

Check the authorized keys on the Spark:

cat ~/.ssh/authorized_keys

Make sure your public key is listed there.

Useful Commands Cheat Sheet

Pro Tips

1. Tailscale starts on boot — the tailscaled service is enabled by default, so your Spark will rejoin the tailnet automatically after a reboot.

2. Forward ports for Jupyter — if you run JupyterLab on your Spark:

   ssh -L 8888:localhost:8888 saiyam@spark-5223
   

   Then open http://localhost:8888 in your browser.

3. File transfers work too:

   scp model.bin saiyam@spark-5223:~/models/
   

4. Check who's connected — on the Spark, see active SSH sessions:

   who
   

5. Tailscale admin console — monitor all devices, manage keys, and remove devices at login.tailscale.com/admin.

Cleanup (If Needed)

If you ever want to remove Tailscale from your Spark:

sudo tailscale down
sudo apt remove --purge tailscale
sudo rm /etc/apt/sources.list.d/tailscale.list
sudo rm /usr/share/keyrings/tailscale-archive-keyring.gpg
sudo apt update

To restore: re-run installation steps 1-2.

Wrapping Up

The whole setup took me about 10 minutes. Now I can SSH into my DGX Spark from my MacBook at home, my second laptop on the go, and even my friend can access it simultaneously from his MacBook — all without any port forwarding, static IPs, or VPN servers.

The key takeaways:

• For yourself: Install Tailscale on both devices, log in with the same account, ssh in

• For friends: Generate a pre-auth key, have them join your tailnet, add their SSH public key to the Spark

• Troubleshooting: Make sure all devices are on the same tailnet, SSH is running, and keys are in authorized_keys

Just ssh saiyam@spark-5223 — from anywhere in the world.

I also used this at 30000 feet in the air!

https://x.com/SaiyamPathak/status/2032098978213528037?s=20

Yesterday, Anthropic accidentally published a 59.8 MB source map file inside version 2.1.88 of their @anthropic-ai/claude-code npm package. The build pipeline was configured to generate source maps, and the packaging step — whether it was a missing .npmignore rule or a misconfigured files field in package.json — failed to exclude them. One packaging oversight, and ~512,000 lines of TypeScript source were public. Anthropic's DMCA notice eventually took down over 8,100 GitHub repositories. A clean-room rewrite called claw-code hit 50,000 stars in about two hours and now about 100k+.

Within last 24 hours, the internet was flooded with hot takes. Architecture diagrams. The virtual pet system. Thread after thread of "I read the entire codebase and here's what I found." Sites like ccunpacked.dev did genuinely good visual walkthroughs of the high-level structure.

But let's be real — nobody read 512,000 lines of TypeScript. I certainly didn't, and I'm skeptical of anyone who claims they did. What I did do was feed the source into Claude, systematically analyzed the key modules, cross-referenced what I found against public documentation and other analyses, and verified the claims I'm about to make against the actual code. If you've read other "I analyzed the leak" posts, they probably all used a similar workflow. The difference I'm going for here is honesty about the process.

The Core Loop Is a State Machine, and That's the Whole Point

The agent loop lives in query.ts. It's exactly 1,729 lines (I checked), structured as an async generator function called queryLoop wrapping a while(true) loop. The code itself, in an internal comment, references "7 continue sites" — seven distinct points where the loop yields control and decides what to do next.

The actual function signature:

async function* queryLoop(
  params: QueryParams,
  consumedCommandUuids: string[],
): AsyncGenerator<
  StreamEvent | RequestStartEvent | Message | TombstoneMessage | ToolUseSummaryMessage,
  Terminal
>

Why does this matter? Because most agent frameworks treat the LLM call as the center of gravity. Send a prompt, get a response, run a tool, repeat. That works fine for demos. It falls apart the moment you need to pause a session, resume it later, serialize state, handle errors mid-turn, or compose multiple agents together.

The generator pattern makes every loop iteration an explicit state transition. You can yield control at each of the seven points without losing state. You can test individual stages. You can add compaction, permission checks, or budget tracking as stages rather than side effects bolted onto a callback chain.

If you're building an agent and your core loop is a simple while with await model.chat() in the middle, this is the pattern to study.

Five Compaction Strategies (Not a Neat Stack)

Every long-running agent eventually fills its context window. Most frameworks handle this by truncating old messages. Claude Code has five distinct strategies — though I want to be clear, these aren't a clean "Layer 1 through 5" hierarchy like some other posts have described. They're composable strategies that kick in under different conditions:

Snip prunes older messages for quick headroom. Fast and lossy.

Microcompact targets tool outputs specifically. A 5,000-line file read gets saved to disk; the model sees a summary with a reference. Two implementations handle this: microCompact.ts and apiMicrocompact.ts. This alone is a big deal — a single uncompressed tool output can eat half your context window.

Context Collapse progressively compresses older conversation segments while keeping recent context sharp. It's still behind a CONTEXT_COLLAPSE feature flag, with dedicated persistence types (ContextCollapseCommitEntry, ContextCollapseSnapshotEntry) to survive session restarts. Not yet fully shipped.

Autocompact is full-conversation summarization at configurable token thresholds. Replaces older history with a summary.

Reactive Compact is the emergency brake — behind the REACTIVE_COMPACT feature flag. When the API returns a 413 (payload too large), this aggressively compacts everything so your session doesn't die. Without this, one bad tool output would brick the conversation.

Now, I've seen posts claiming "no other framework has this." That was arguably true in 2025, but it's not true now. Microsoft's Agent Framework has composable multi-strategy compaction pipelines. LangChain Deep Agents (shipped March 15, 2026) does filesystem offloading plus multi-frequency summarization. Google ADK has sliding window with summarization.

What sets Claude Code apart isn't that it has compaction — it's the granularity. Five strategies, two of them still being iterated on behind feature flags. That reflects the kind of edge cases you only discover at scale.

Deferred Tool Loading

This is probably the most practical pattern in the codebase for anyone building agents.

When you connect MCP servers, you might have 200+ tools available. Sending all those schemas on every API call wastes thousands of tokens. Claude Code's solution: mark tools with defer_loading: true. The model doesn't see them. Instead, it has a single meta-tool called ToolSearch (the internal class is ToolSearchTool, but the model-facing name is ToolSearch — defined as TOOL_SEARCH_TOOL_NAME = 'ToolSearch' in the constants). When the model needs a capability, it calls ToolSearch with a query:

User: "Deploy this to my Kubernetes cluster"

Model calls ToolSearch("kubernetes deploy")
  -> System fuzzy-matches deferred tool descriptions
  -> Injects matching schemas into the conversation
Model now has the tools it needs.

The model goes from ~20 core tools to access to hundreds, without the upfront token cost.

This pattern has spread. The OpenAI Agents SDK now has deferLoading: true with tool search (requires GPT-5.4+). ZeroClaw implements nearly identical deferred loading. CrewAI 1.10.2a1 (March 2026) added dynamic tool injection via Anthropic's tool search API.

But there's still no framework-agnostic library for this. The core is straightforward — fuzzy matching over tool descriptions, schema injection on demand, MCP compatibility. If someone built this as a standalone package, it'd be useful immediately.

Default-Deny Permissions With a Graceful Fallback

Claude Code's permission system is built on default-deny. Every tool has two permission-relevant properties defined in Tool.ts:

• isReadOnly — defaults to false (assume the tool writes)

• isDestructive — defaults to false

Tools must explicitly declare their risk profile. The permission system then layers rule-based checks (alwaysAllow/alwaysDeny rules), pre-tool-use hooks (which can modify input, block execution, or log), and an auto-mode safety classifier.

The part I found most interesting is the denial tracking in denialTracking.ts — just 46 lines:

3 consecutive denials → shouldFallbackToPrompting() returns true
20 total denials in a session → same result

If the user keeps saying "no," the system stops running in auto-mode and starts asking for explicit permission on every action. Most agent frameworks either keep retrying or hard-stop. Claude Code's approach gracefully degrades: "You're uncomfortable with what I'm doing, so I'll check before each step."

Small file, big principle.

The Cost Engineering

These are the details that only matter at Anthropic's scale, but they're instructive:

Sticky-on latches. When a feature flag activates during a session, it stays on for the rest of that session. Flipping it back would change the system prompt, which busts the prompt cache. The promptCacheBreakDetection.ts file tracks 14 distinct state fields that can invalidate the cache — system prompt hash, tool schema hashes, model changes, beta headers, effort values, and more. Sticky latches prevent unnecessary cache invalidation from mode toggles.

Tool result persistence. Large outputs get written to disk; the model sees a preview. This isn't just context management — it keeps the cache prefix stable.

Schema stability. Tool schemas assembled once at session start, held stable throughout. MCP tools can come and go, but the core schema block doesn't change.

At scale, these optimizations compound significantly.

What I Actually Took Away From This

I'm not going to pretend the leak is a startup idea list or that I discovered things nobody else saw. But analyzing the code did crystallize a few things:

Context management is harder than it looks. Five strategies, two still behind feature flags, a dedicated promptCacheBreakDetection.ts tracking 14 vectors. This is not a solved problem, even for Anthropic.

Deferred tool loading is becoming table stakes. Claude Code, OpenAI, ZeroClaw, CrewAI — multiple teams independently arrived at the same pattern. If you're building an agent with more than ~20 tools and you're not doing this, you're wasting tokens.

Permission design matters more than permission features. The denial tracking system is 46 lines. The principle it encodes — "degrade gracefully when the user loses trust" — is more important than any specific implementation detail.

The real work is in the orchestration. The model call is one stage out of seven in the main loop. Everything else — state management, compaction, tool loading, permissions, cost optimization — is where the engineering actually lives.

Being Honest About the Process

Every blog you've read about this leak was written with AI assistance. This one included. I used Claude to analyze the source modules, identify patterns, and draft the initial structure. I then fact-checked every claim — cross-referencing the actual source code, public documentation, news coverage, and other technical analyses. Where I found errors in my initial draft (and there were several), I corrected them.

The value isn't in pretending I manually read half a million lines of code. It's in doing the verification work to make sure what I'm telling you is actually true. In a sea of AI-generated analysis of AI-generated code, accuracy is the differentiator.

If you spot something wrong, tell me — I'd rather correct it than let it stand.

I write about cloud-native, AI engineering, and the infrastructure that makes modern software work. Find me on Twitter or LinkedIn.

Since then, ingress-nginx was officially archived (March 24, 2026). March 31 is end of life -- zero security patches after that date.

Based on community feedback from KubeCon, this is the biggest update yet: 119 annotations (up from 50), Gateway API with Traefik as the provider (the #1 request), and impact ratings on every annotation so you know exactly what matters.

This post walks through a complete end-to-end migration on a vind cluster with actual command outputs.

Why You Need to Migrate Now

• Nov 11, 2025: Kubernetes SIG Network announces ingress-nginx retirement

• Jan 29, 2026: Joint statement from Kubernetes Steering + Security Response Committees urging immediate migration

• Mar 24, 2026: GitHub repository archived (read-only)

• Mar 31, 2026: End of life -- zero support from this date

Chainguard maintains a fork for CVE-level fixes only -- no features, no community PRs, no pre-built images. You're on your own.

The Three Migration Paths

The Annotation Problem

The real complexity isn't swapping controllers -- it's the annotations. A typical production Ingress has 10-15 NGINX annotations for SSL, auth, rate limiting, CORS, session affinity, and more.

ing-switch maps 119 annotations with impact ratings:

Every unsupported annotation gets an impact rating: NONE (safe to ignore), LOW (better defaults), MEDIUM (needs workaround), or VARIES (review your snippets). Most teams discover 70%+ of "unsupported" annotations are safe to ignore.

End-to-End Demo: vCluster + ing-switch

Let's walk through a complete migration on a real cluster. We'll use vCluster to spin up a Kubernetes cluster in Docker, deploy 3 services with NGINX annotations, and migrate them to Gateway API with Traefik.

Step 1: Create a Cluster

vcluster create demo --driver docker

Output:

info  Using vCluster driver 'docker' to create your virtual clusters
info  Ensuring environment for vCluster demo...
done  Created network vcluster.demo
info  Starting vCluster standalone demo
done  Successfully created virtual cluster demo
info  Waiting for vCluster to become ready...
done  vCluster is ready
done  Switched active kube context to vcluster-docker_demo

Verify:

kubectl get namespaces

NAME                 STATUS   AGE
default              Active   16s
kube-flannel         Active   6s
kube-node-lease      Active   16s
kube-public          Active   16s
kube-system          Active   16s
local-path-storage   Active   6s

Step 2: Install Ingress NGINX

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --create-namespace \
  --set controller.service.type=ClusterIP \
  --set controller.admissionWebhooks.enabled=false \
  --wait --timeout 120s

NAME: ingress-nginx
LAST DEPLOYED: Sun Mar 29 11:15:57 2026
NAMESPACE: ingress-nginx
STATUS: deployed

kubectl get pods -n ingress-nginx

NAME                                        READY   STATUS    RESTARTS   AGE
ingress-nginx-controller-5486dbd97f-vc9wv   1/1     Running   0          54s

Step 3: Deploy 3 Apps with NGINX Annotations

We deploy three services, each with different annotation patterns:

App 1 -- Basic web app (SSL redirect + timeouts):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-app
  namespace: demo
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "60"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "10"
spec:
  ingressClassName: nginx
  rules:
  - host: web.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web-app
            port:
              number: 80

App 2 -- API with CORS + rate limiting (10 annotations):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-cors
  namespace: demo
  annotations:
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-origin: "https://app.example.com,https://admin.example.com"
    nginx.ingress.kubernetes.io/cors-allow-methods: "GET, POST, PUT, DELETE, OPTIONS"
    nginx.ingress.kubernetes.io/cors-allow-headers: "Content-Type, Authorization, X-API-Key"
    nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
    nginx.ingress.kubernetes.io/cors-max-age: "86400"
    nginx.ingress.kubernetes.io/limit-rps: "50"
    nginx.ingress.kubernetes.io/limit-burst-multiplier: "3"
    nginx.ingress.kubernetes.io/proxy-body-size: "5m"
spec:
  ingressClassName: nginx
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /v1
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 80

App 3 -- Auth-protected dashboard (external auth + IP allowlist + session affinity):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dashboard
  namespace: demo
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/auth-url: "https://auth.example.com/verify"
    nginx.ingress.kubernetes.io/auth-response-headers: "X-User-ID,X-User-Email"
    nginx.ingress.kubernetes.io/whitelist-source-range: "10.0.0.0/8,172.16.0.0/12"
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "dashboard-session"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "3600"
spec:
  ingressClassName: nginx
  rules:
  - host: dashboard.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: dashboard
            port:
              number: 80

After applying all three:

kubectl get ingress -n demo

NAME        CLASS   HOSTS                   ADDRESS   PORTS   AGE
api-cors    nginx   api.example.com                   80      5s
dashboard   nginx   dashboard.example.com             80      5s
web-app     nginx   web.example.com                   80      5s

kubectl get pods -n demo

NAME                           READY   STATUS    RESTARTS   AGE
api-service-5f99b6d99d-x7vmn   1/1     Running   0          24s
dashboard-9ddbf867-7dbgf       1/1     Running   0          24s
web-app-969c76b7c-7wqw5        1/1     Running   0          24s

3 ingresses, 20 NGINX annotations, 3 services running. Now let's see what ing-switch makes of this.

Step 4: Scan the Cluster

ing-switch scan

  ing-switch -- Cluster Scan Results
  Cluster: vcluster-docker_demo

  Ingress Controller Detected
  Type:      ingress-nginx
  Version:   unknown
  Namespace: ingress-nginx

  Found 3 Ingress resource(s)

  NAMESPACE   NAME        HOSTS                   ANNOTATIONS   TLS   COMPLEXITY
  ---------   ----        -----                   -----------   ---   ----------
  demo        api-cors    api.example.com         10            no    unsupported
  demo        dashboard   dashboard.example.com   7             no    complex
  demo        web-app     web.example.com         3             no    complex

ing-switch detected the NGINX controller and found all 3 ingresses with their annotation counts and complexity scores.

Step 5: Analyze Compatibility

Let's compare all three targets:

Traefik v3:

ing-switch analyze --target traefik

  Summary
  -------
  Total ingresses:      3
  Fully compatible:     1
  Needs workarounds:    2
  Has unsupported:      0

Gateway API (Envoy):

ing-switch analyze --target gateway-api

  Summary
  -------
  Total ingresses:      3
  Fully compatible:     0
  Needs workarounds:    3
  Has unsupported:      0

Gateway API (Traefik):

ing-switch analyze --target gateway-api-traefik

  Summary
  -------
  Total ingresses:      3
  Fully compatible:     0
  Needs workarounds:    3
  Has unsupported:      0

Key insight: Traefik is the highest-compatibility target for this workload (1 fully compatible out of 3). The CORS annotations map directly to Traefik's Headers middleware. For Gateway API, CORS is now also fully supported thanks to the native CORS filter in Gateway API v1.5.

Here's the detailed annotation mapping for the API with CORS:

  demo/api-cors
  -------------
  ANNOTATION               STATUS        TARGET RESOURCE                    NOTES
  enable-cors              [supported]   HTTPRoute (CORS filter)            Native CORS filter (GA in Gateway API v1.5)
  cors-allow-origin        [supported]   HTTPRoute (CORS filter)            allowOrigins in CORS filter
  cors-allow-methods       [supported]   HTTPRoute (CORS filter)            allowMethods in CORS filter
  cors-allow-headers       [supported]   HTTPRoute (CORS filter)            allowHeaders in CORS filter
  cors-allow-credentials   [supported]   HTTPRoute (CORS filter)            allowCredentials in CORS filter
  cors-max-age             [supported]   HTTPRoute (CORS filter)            maxAge in CORS filter
  force-ssl-redirect       [supported]   HTTPRoute (RequestRedirect filter) 301 redirect to HTTPS
  limit-rps                [partial]     BackendTrafficPolicy (RateLimit)   Envoy Gateway BackendTrafficPolicy
  limit-burst-multiplier   [partial]     BackendTrafficPolicy (RateLimit)   Burst configurable but uses tokens
  proxy-body-size          [partial]     BackendTrafficPolicy               requestBuffer.limit

7 out of 10 annotations are fully supported. The 3 "partial" ones work -- they just use a slightly different API.

Step 6: Generate Migration Files

ing-switch migrate --target gateway-api-traefik --output-dir ./migration

  ing-switch -- Generating Migration Files
  Target:     gateway-api-traefik
  Output dir: ./migration

  + 00-migration-report.md
  + 01-install-gateway-api-crds/install.sh
  + 02-install-traefik-gateway/helm-install.sh
  + 02-install-traefik-gateway/values.yaml
  + 03-gateway/gatewayclass.yaml
  + 03-gateway/gateway.yaml
  + 04-httproutes/demo-api-cors.yaml
  + 04-httproutes/demo-dashboard.yaml
  + 04-httproutes/demo-web-app.yaml
  + 05-policies/demo-api-cors-ratelimit.yaml
  + 05-policies/demo-dashboard-forwardauth.yaml
  + 05-policies/demo-dashboard-ipallowlist.yaml
  + 06-verify.sh
  + 07-cleanup/remove-nginx.sh
  Generated 13 files in ./migration/

Step 7: Inspect the Generated YAML

GatewayClass -- points to Traefik, not Envoy:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: traefik
spec:
  controllerName: traefik.io/gateway-controller

HTTPRoute with native CORS filter (no more ResponseHeaderModifier hacks):

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-cors
  namespace: demo
spec:
  parentRefs:
  - name: ing-switch-gateway
    namespace: default
  hostnames:
  - "api.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: "/v1"
    filters:
    - type: CORS
      cors:
        allowOrigins:
        - type: Exact
          value: "https://app.example.com"
        - type: Exact
          value: "https://admin.example.com"
        allowMethods:
        - "GET"
        - "POST"
        - "PUT"
        - "DELETE"
        - "OPTIONS"
        allowHeaders:
        - "Content-Type"
        - "Authorization"
        - "X-API-Key"
        allowCredentials: true
        maxAge: "86400s"
    backendRefs:
    - name: api-service
      port: 80

Traefik Middleware CRDs (not Envoy-specific policies):

# Rate Limiting
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: demo-api-cors-ratelimit
  namespace: demo
spec:
  rateLimit:
    average: 50
    burst: 3

# ForwardAuth (external authentication)
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: demo-dashboard-forwardauth
  namespace: demo
spec:
  forwardAuth:
    address: "https://auth.example.com/verify"
  authResponseHeaders:
    - "X-User-ID"
    - "X-User-Email"

# IP AllowList
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: demo-dashboard-ipallowlist
  namespace: demo
spec:
  ipAllowList:
    sourceRange:
    - "10.0.0.0/8"
    - "172.16.0.0/12"

Step 8: Review the Migration Report

The migrate command automatically generates 00-migration-report.md in the output directory. Open it to see the full summary:

cat ./migration/00-migration-report.md

# ing-switch Migration Report
**Target Controller:** gateway-api-traefik

## Summary
| Metric | Count |
|--------|-------|
| Total Ingresses | 3 |
| Fully Compatible | 0 |
| Needs Workarounds | 3 |
| Has Unsupported Annotations | 0 |

## demo/api-cors -- Needs workaround
| Annotation | Status | Target Resource | Notes |
|-----------|--------|-----------------|-------|
| enable-cors | OK | HTTPRoute (CORS filter) | Native CORS filter (GA in v1.5) |
| cors-allow-origin | OK | HTTPRoute (CORS filter) | allowOrigins in CORS filter |
| limit-rps | WARN | BackendTrafficPolicy | Envoy Gateway BackendTrafficPolicy |
...

Step 9: Apply (Dry-Run First)

# Install Gateway API CRDs
bash ./migration/01-install-gateway-api-crds/install.sh

# Install Traefik with Gateway API provider
bash ./migration/02-install-traefik-gateway/helm-install.sh

# Dry-run all resources first
kubectl apply -f ./migration/03-gateway/ --dry-run=server
kubectl apply -f ./migration/04-httproutes/ --dry-run=server

# If dry-run passes, apply for real
kubectl apply -f ./migration/03-gateway/
kubectl apply -f ./migration/04-httproutes/
kubectl apply -f ./migration/05-policies/

At this point, both NGINX and Traefik are running side by side. DNS still points to NGINX. Production traffic is untouched.

Step 10: Verify and Cutover

# Run the generated verification script
bash ./migration/06-verify.sh

# Once verified, update DNS to Traefik's IP
# Then clean up NGINX
bash ./migration/07-cleanup/remove-nginx.sh

Step 11: Use the Web UI

For teams that prefer a visual workflow:

ing-switch ui
# Opens http://localhost:8080

The dashboard provides four pages:

Detect -- Scan your cluster and see all ingresses with annotation counts and complexity:

Analyze -- Choose between 3 targets and see the full annotation compatibility matrix:

Migrate -- One-click generation with step-by-step checklist and dry-run buttons:

View all generated files inline with syntax highlighting:

See migration gaps with impact ratings and fix instructions:

Validate -- Run live cluster checks to confirm your migration phase:

Cleanup

vcluster delete demo --driver docker

done  Successfully deleted virtual cluster demo

What Makes ing-switch Different

The Ecosystem Is Ready

• Gateway API v1.5 -- CORS filter, TLSRoute, BackendTLSPolicy all GA

• ingress2gateway v1.0 -- Official tool with emitter architecture

• Traefik v3.7 -- Native NGINX annotation provider (80+ annotations)

• Envoy Gateway v1.7 -- XListenerSet, enhanced policies

• cert-manager v1.20 -- Gateway API ListenerSet support

• Kubernetes 1.36 -- Ships April 22, first release post-NGINX archival

The tools exist. The standards are stable. The only thing left is to actually run the migration.

Star it, fork it, migrate today: github.com/saiyam1814/ing-switch

ing-switch is open source under the MIT license. PRs welcome.

OpenClaw has 314,000+ GitHub stars. It is the most popular open-source AI agent out there. It connects to WhatsApp and Telegram, does deep research, manages files, writes code, handles voice notes, and genuinely works as a personal assistant. The catch is that setting it up with a local LLM on NVIDIA hardware is a long process with security in place.

I spent some time getting it right on a DGX Spark. Then I automated the entire thing into one command. Along the way I found nine bugs, wrote three source patches, fought Ubuntu's managed Python, debugged WhatsApp's device linking protocol, and integrated a hardware-aware model selection engine. This is the full story.

The Problem

NVIDIA's DGX Spark is a desktop AI supercomputer. GB10 Grace Blackwell chip, 128GB unified memory, 1 PFLOP of AI compute, 20-core ARM Cortex-A725 CPU. It sits on your desk, runs quietly, and has enough memory to load models that actually compete with cloud APIs. The hardware is not the bottleneck anymore.

The bottleneck is setup. If you want to run OpenClaw with a local model on DGX Spark, here is what you need to do manually:

1. Install Node.js 22+

2. Install OpenClaw via npm

3. Install Ollama

4. Figure out which model fits your hardware (there are hundreds)

5. Pull the model (can be 20-80GB)

6. Configure OpenClaw to point at your local Ollama instance

7. Set the correct environment variables (OLLAMA_API_KEY, OLLAMA_BASE_URL)

8. Run onboard, which sets half your config to wrong defaults

9. Fix tools.profile from "messaging" to "full"

10. Start the gateway, then start a separate Node Host process

11. Pair the Node Host to the gateway

12. Link WhatsApp via QR code

13. Patch three different JavaScript files in OpenClaw's dist folder

14. Install skills

15. Harden security

16. Set up voice transcription

Miss one step and things fail silently. The gateway starts, the model loads, but your agent can only send text messages because it has 5 tools instead of 15. Or WhatsApp linking fails because the browser identification string gets rejected. Or group messages never arrive because history sync is disabled.

NVIDIA's own docs at build.nvidia.com recommend gpt-oss-120b for DGX Spark and describe a manual multi-step process using Ollama or LM Studio. Their guide covers the inference setup but not WhatsApp integration, not voice transcription, not security hardening, and not the Node Host that the agent actually needs to do useful work. clawspark automates all of this, including the parts NVIDIA's guide does not cover.

What clawspark Does

One command:

curl -fsSL https://clawspark.dev/install.sh | bash

That is it. The installer runs 14 steps. It detects your hardware, recommends a model, asks a few questions, then installs and configures everything. Here is the full flow:

Step 1-2: Hardware detection. The script probes your GPU via nvidia-smi, reads DMI product name for DGX Spark identification, checks for Tegra signatures on Jetson, and measures total system memory. It classifies your hardware into one of four tiers: DGX Spark (128GB unified), Jetson AGX (64GB), RTX high-end (24GB+ VRAM), or RTX standard (8-24GB).

Step 3: Model selection. This is where it gets interesting. For DGX Spark, I curated a list of 5 models ranked by llmfit score and verified on real hardware:

For non-DGX-Spark hardware (RTX, Jetson, anything else), the installer uses llmfit to analyze your specific hardware, score hundreds of models, map the results to Ollama model IDs, verify each candidate actually exists on the Ollama library, and present the top 5 that fit. No hardcoded lists. Your GPU, your recommendations.

Step 4-5: Deployment and messaging. Choose local-only or hybrid (cloud fallback). Choose WhatsApp, Telegram, both, or skip messaging entirely. The web UI at /__openclaw__/canvas/ always works regardless.

Step 6-14: The actual installation. Ollama install and model pull. Node.js 22 if needed. OpenClaw npm install. Config generation with correct Ollama endpoints. Onboard with overrides for all the wrong defaults. Three source patches (more on these below). Skills installation. Whisper voice setup. WhatsApp QR linking. Optional Tailscale for remote access. ClawMetry dashboard. Security hardening. Final verification.

After installation, you get the clawspark CLI tool for day-to-day management: clawspark status, clawspark benchmark, clawspark restart, clawspark skills sync, clawspark airgap on/off, and more.

The Architecture

Here is how the pieces fit together once everything is running:

WhatsApp / Telegram / Web UI (Canvas)
              |
    OpenClaw Gateway (port 18789)
     |            |            |
   Agent      Node Host     Baileys
   (LLM)      (Tools)      (WhatsApp Web)
     |            |
   Ollama      15 tools:
  (port 11434)  exec, read, write, edit,
     |          web_fetch, message, canvas,
   Model        process, cron, nodes,
   (GPU)        sessions_spawn, vision,
                transcribe, memory_search,
                memory_store

The Gateway is the central process. It manages the agent, routes messages from WhatsApp/Telegram/Web, and coordinates tool calls. The Agent is the LLM reasoning loop that decides what to do. The Node Host is a separate process that provides the actual tool implementations -- reading files, fetching web pages, executing code. Without the Node Host, the agent has only 5 basic messaging tools instead of the full 15.

Baileys is the WhatsApp Web client library that OpenClaw uses under the hood. It connects to WhatsApp's servers using a linked device session, the same way WhatsApp Web works in your browser. Messages flow from WhatsApp through Baileys to the Gateway, which sends them to the Agent, which calls tools on the Node Host, and the response flows back the same way.

The Bugs I Found and Fixed

This section is the reason I wrote this post. These are all real issues I hit on real hardware, and some of them took hours to diagnose. If you are setting up OpenClaw manually, this list might save you a lot of time.

1. tools.profile does not default to "full"

When you run openclaw onboard, it does not set tools.profile to "full". In v2026.3.2 it defaulted to "messaging" (5 tools only). This was partially fixed in v2026.3.7, which changed the default to "coding" -- better, but still missing tools like exec, process, cron, and nodes. The agent looks like it is working, but it cannot do everything it should.

The fix: openclaw config set tools.profile full after onboard completes. clawspark does this automatically.

2. Node Host is required but not documented

The Gateway alone does not provide execution tools. You need a separate "Node Host" process (openclaw node run) that connects to the Gateway and provides filesystem, browser, and execution capabilities. Without it, even with tools.profile full, the agent has no tools to call. The Node Host also needs to be paired with the Gateway (device approval), which is another step that is easy to miss.

clawspark starts the Node Host, detects pending pairing requests, auto-approves them, and restarts the Node Host with the pairing token.

3. Baileys browser string rejected by WhatsApp

OpenClaw's WhatsApp integration uses Baileys, which identifies itself as ["openclaw", "cli", VERSION] to WhatsApp's servers. WhatsApp rejects this during device linking. The QR code scan works, but the connection fails silently.

The fix is a source patch: replace the browser identification with ["Ubuntu", "Chrome", "22.0"], which WhatsApp accepts. This requires patching the compiled JavaScript in OpenClaw's dist folder. clawspark finds the relevant session files and applies the patch automatically.

4. web_search requires a Brave API key

OpenClaw's built-in web_search tool requires a Brave Search API key. For a local setup, requiring an external API key defeats the purpose. clawspark works around this by configuring the agent's TOOLS.md to use DuckDuckGo Lite via web_fetch instead:

web_fetch with url="https://lite.duckduckgo.com/lite/?q=YOUR+QUERY"

This gives the agent web search capabilities without any API keys or external dependencies.

5. Agent narrates tool usage on WhatsApp

When you ask the agent a question on WhatsApp, it sends messages like "Let me search for that..." and "The search returned these results..." before giving you the actual answer. On WhatsApp, this means three or four notification buzzes for one question.

The fix is SOUL.md rules: explicit instructions to never narrate tool usage, use tools silently, and respond with one clean message.

6. syncFullHistory breaks group messages

OpenClaw defaults to syncFullHistory: false in its Baileys configuration. This means after a fresh WhatsApp link, Baileys never receives group sender keys. The result: groups are completely silent. No messages arrive, no errors are logged. It just looks like nobody is talking.

The fix: patch syncFullHistory: false to syncFullHistory: true in the compiled session files. clawspark finds and patches all relevant files automatically.

7. Mention detection has an early return that blocks text @mentions

OpenClaw's mention detection in group chats has a return false early exit when JID (WhatsApp ID) mentions exist but do not match the bot's JID. The problem is that WhatsApp resolves @mentions to JIDs, and sometimes the resolved JID does not match the bot's linked phone JID. The early return prevents the text-pattern fallback from ever running, so typing @botname in a group never triggers the bot.

The fix: remove the return false line so the text-pattern fallback always has a chance to match. Another source patch to the compiled JavaScript.

8. Systemd service missing Ollama environment variables

When OpenClaw's gateway runs as a systemd service, it does not inherit the shell environment. The OLLAMA_API_KEY and OLLAMA_BASE_URL variables are missing, so the gateway cannot reach Ollama. The model appears to load, but every inference call fails.

clawspark writes the environment variables to a gateway.env file, adds them to the user's shell profile (.bashrc or .zshrc), and sources them before starting any OpenClaw process.

9. OpenClaw bindings schema changed between versions

This one cost me an entire evening. Earlier versions of OpenClaw supported a bindings config for routing different message sources to different agents (e.g., full tools in DMs, restricted tools in groups). Starting with v2026.3.2, the bindings schema changed and the old format causes a validation error at startup: Invalid config: bindings.0: Invalid input. This is not a bug per se -- the schema evolved -- but any guide or config from earlier versions will break silently.

The fix: remove bindings entirely and use a single agent with context-aware rules. SOUL.md and TOOLS.md contain explicit sections for DM context (full tools) and group context (Q&A only). The agent enforces the boundary at the prompt level. Groups also use requireMention: true and groupPolicy: open at the config level so the bot only responds when @mentioned.

Security

Running a local AI agent is not automatically secure. clawspark applies multiple layers:

Gateway binding. The OpenClaw gateway binds to localhost only. It is not accessible from other machines on your network unless you explicitly set up Tailscale.

Firewall rules. UFW is configured to deny all incoming connections except SSH. Outgoing traffic is allowed by default, or blocked entirely in air-gap mode.

Token authentication. A random 256-bit token is generated during installation. Only clients with this token can talk to the gateway API.

Context-aware tool restrictions. In direct messages, the owner gets full access to all 15 tools. In group chats, the agent restricts itself to Q&A only (message, web_fetch, memory). This is enforced at the prompt level via SOUL.md, which contains explicit rules for each context. Groups also require @mention to activate.

SOUL.md and TOOLS.md. These workspace files contain the agent's identity, capabilities, and absolute rules. No credential disclosure (applies to all users, including the owner). No system information in groups. No self-modification. Both files are set to chmod 444 (read-only) so the agent cannot edit its own rules.

Air-gap mode. For maximum isolation, clawspark airgap on blocks all outbound internet traffic via UFW. Only local network and loopback traffic is allowed. The model, the agent, and all tools run entirely offline.

One honest caveat: local models do not have the same safety filters that cloud providers build into their APIs. That is both a feature (no arbitrary refusals) and a responsibility. You should think carefully about who has access to message your bot.

Real Performance Numbers

All numbers below are from an actual DGX Spark running Linux 6.14.0-1015-nvidia (arm64), Ollama 0.17.7, Node.js v22.22.1, OpenClaw v2026.3.13, with Qwen 3.5 35B-A3B:

The 59 tok/s generation speed is fast enough to feel responsive. You send a question on WhatsApp and the response arrives in 15-45 seconds depending on complexity. The cold load penalty only hits on the first query after a restart. After that, the model stays in memory.

To put this in perspective: 59 tok/s means the model generates roughly 45-50 words per second. A typical response of 200 words takes about 4 seconds of pure generation time. The rest of the 15-45 second latency comes from the WhatsApp message routing, tool calls (if the agent needs to search the web or read a file), and response formatting.

Is this as fast as GPT-4o or Claude via cloud API? No. Cloud inference on dedicated hardware with massive batching will always be faster for raw token throughput. But it is fast enough for practical use, and your data never leaves your desk. That is the tradeoff.

For the 122B-A10B MoE model (the highest-ranked by llmfit), expect roughly 45 tok/s. Slightly slower but you get access to the full 122B model's knowledge with only 10B active parameters. The DGX Spark's 128GB unified memory can comfortably hold this model (33GB) with plenty of room for the KV cache.

Hardware-Aware Model Selection with llmfit

One of the hardest problems with local AI is knowing which model to use. There are hundreds of models on Ollama, each with different sizes, quantizations, and performance characteristics. Picking the wrong one means either wasting memory (model too small for the hardware) or crashing on load (model too big).

I integrated llmfit to solve this. llmfit is a Rust-based CLI tool that detects your hardware (GPU, VRAM, RAM, CPU), scores every model in its database for fit, speed, and quality, and tells you which ones will actually run well.

For DGX Spark, I ran llmfit on the real hardware and it correctly detected the NVIDIA GB10 with 119.7 GB unified memory and CUDA backend. I then cross-referenced its top recommendations against the Ollama library to verify each model is actually pullable. The result is the curated list of 5 models you see during installation.

For all other hardware, the installer runs llmfit live:

1. Install llmfit (one curl command, Rust binary)

2. Run llmfit recommend --json -n 20 --min-fit good

3. Map each HF-style model name to an Ollama model ID (40+ regex patterns)

4. Verify each candidate exists on Ollama's library (HTTP check against ollama.com)

5. Present the top 5 verified models with score, estimated tok/s, and fit level

If llmfit is not available or fails, the installer falls back to curated lists per hardware tier. No hardcoded guessing. Your GPU, your recommendations.

WhatsApp Integration: The Deep Cut

Getting WhatsApp working reliably was the hardest part of this entire project. Here is why.

OpenClaw uses the Baileys library, which is an unofficial WhatsApp Web client. It works by emulating a linked device session, the same protocol that WhatsApp Web uses in your browser. The connection is end-to-end encrypted and goes through WhatsApp's servers. There is no official API involved.

This creates three categories of problems:

Protocol issues. WhatsApp regularly changes its protocol, and Baileys has to keep up. The browser string rejection (bug #3) is an example. WhatsApp started rejecting non-standard browser identifiers at some point, and OpenClaw's default string got caught.

Group message handling. WhatsApp groups use Signal's sender keys protocol. When you first link a device, it needs to receive sender keys from all group participants before it can decrypt group messages. Setting syncFullHistory: false (bug #6) prevents this initial key exchange, making groups completely silent.

Mention routing. In WhatsApp groups, @mentions get resolved to JIDs (WhatsApp internal user IDs). The bot's JID might not match its linked phone number's JID in all cases. The early return in mention detection (bug #7) means the bot never sees @mentions in groups unless you patch the code.

clawspark applies all three patches automatically and re-applies them after updates (since npm update overwrites the dist files). Groups require @mention to activate, and the agent is restricted to Q&A only in group context. Full tool access is reserved for direct messages with the owner.

Voice notes work through the local-whisper skill, which runs Whisper (OpenAI's open-source speech-to-text model) locally on the GPU. On DGX Spark, it uses the large-v3 model for maximum transcription accuracy. On Jetson, it drops to the small model. On RTX, it scales based on available VRAM. The audio never leaves your machine.

Skills

clawspark installs 10 skills by default, verified against the OpenClaw skill registry:

Web search works without any API keys. The agent uses DuckDuckGo Lite via web_fetch, fetches result URLs, and composes answers from the content. No Brave API key, no Google API key, no external dependencies.

You can add or remove skills with clawspark skills add <name> and clawspark skills remove <name>. The skills.yaml file in your config directory is the source of truth, and clawspark skills sync reads it and installs everything.

Getting Started

Tested and verified on DGX Spark. Should also work on Mac (Apple Silicon M1/M2/M3/M4), RTX desktops, and Jetson. The installer has fallbacks for all platforms -- macOS uses Homebrew, Ollama runs natively on Apple Silicon, and llmfit handles model selection. These platforms have not been end-to-end tested yet. Community testing welcome -- open an issue if you try it on different hardware.

curl -fsSL https://clawspark.dev/install.sh | bash

Or with specific options:

bash install.sh --model=qwen3.5:122b-a10b --messaging=whatsapp

After installation:

clawspark status      # Check all components
clawspark benchmark   # Run a performance benchmark
clawspark logs        # Tail the gateway logs
clawspark restart     # Restart all services
clawspark update      # Update OpenClaw and re-apply patches
clawspark airgap on   # Enable air-gap mode

The web UI is at http://localhost:18789/__openclaw__/canvas/. The metrics dashboard is at http://localhost:8900.

Source code and documentation: clawspark.dev

What is Next

A few things I am working on:

Multi-model routing. Use the fast 35B model for simple queries and automatically route complex reasoning tasks to the 122B model. The hardware can handle both loaded simultaneously since 128GB is enough for both.

Better metrics. ClawMetry currently shows basic gateway stats. I want per-query latency tracking, token usage by model, and cost-equivalent comparisons (how much this query would have cost on cloud APIs).

More hardware testing. The Jetson and RTX paths in clawspark are written and should work (hardware detection, llmfit model selection, Ollama setup), but I have only done full end-to-end testing on DGX Spark. Jetson AGX Orin is next. For RTX desktops, open a terminal and run the install command directly -- no SSH needed. If you try it on your hardware, please open an issue with your results.

Upstream patches. The three source patches I wrote are necessary because of bugs in OpenClaw's compiled code. Ideally these get fixed upstream so the patches become unnecessary. I plan to submit them.

Closing Thoughts

Two years ago, running a capable AI model locally meant a server rack, a cooling system, and deep knowledge of CUDA. Today it means a quiet box on your desk and a bash script.

The gap between local and cloud AI is closing fast. Not because local is getting as fast as cloud (dedicated data center hardware will always win on raw throughput), but because local is getting good enough. 59 tokens per second from a 35B-parameter MoE model on a desktop machine is good enough for a personal assistant. The tradeoff is straightforward: you get complete data privacy and zero ongoing cost in exchange for slightly higher latency.

clawspark is just the glue. It takes hardware that is already capable (DGX Spark), software that is already good (OpenClaw, Ollama), a model that is already smart (Qwen 3.5), and a tool selector that actually knows your hardware (llmfit), and removes the friction between them. The one-click part is not the innovation. The innovation is all the edge cases, patches, and defaults that make it actually work when you run it for the first time on real hardware.

If you have a DGX Spark, an RTX GPU, a Mac with Apple Silicon, or a Jetson, give it a try. One command, a few questions, and you have a private AI assistant that genuinely never phones home.

GitHub: github.com/saiyam1814/claw-spark Website: clawspark.dev

That’s what running NVIDIA’s Nemotron 3 Super on the DGX Spark feels like. After spending time with it, I think this model needs more attention than it’s getting. Not because of one benchmark number, but because of the engineering choices behind it. These choices show you exactly where AI inference is going.

Let me walk you through what I found.

The Headline Numbers (And Why They’re Misleading)

When NVIDIA drops a model, they lead with the big stats: 120 billion parameters, 1 million token context, 5x throughput. These numbers are real, but they hide the real story.

The number that actually matters is 12.7 billion. That’s how many parameters fire per token. Out of 120.6 billion total, only about a tenth light up for any given input. The rest sit there, waiting until the right token needs their skill.

This roughly 10:1 ratio is the whole story. It’s why the model runs on desktop hardware. It’s why it’s fast. It’s why NVIDIA built it this way. Everything else follows from this one design choice.

The second thing that matters is the layer mix. The model has 88 layers total. Most of them are Mamba-2 layers. This is a completely different architecture from transformers and it doesn’t need to store growing key-value caches. Only a small number are traditional transformer attention layers. NVIDIA interleaves them in a repeating pattern: groups of Mamba-2 blocks paired with Latent MoE layers, with attention layers placed at key depths. We’ll come back to why this split is so important.

Three Architectures in a Trenchcoat

Nemotron 3 Super isn’t one architecture. It’s three, stacked together so each one does what it’s best at. Once you get this stack, you get why the model works the way it does.

Mamba-2: The Workhorse

The majority of the 88 layers are Mamba-2 blocks. Mamba is a state-space model. Think of it like a recurrent architecture that keeps a compact, fixed-size state and updates it as each new token comes in.

The key thing: Mamba runs in linear time. Double the sequence length, you roughly double the compute. Compare that with transformer attention, where doubling the sequence quadruples the compute.

This is why Nemotron 3 Super can actually deliver a 1-million-token context window in practice. With most layers being Mamba, the bulk of the model doesn’t care how long your prompt is. Compute grows linearly, and memory doesn’t grow at all. Mamba’s state stays the same size no matter the sequence length.

Transformer Attention: The Precision Tool

A small number of layers in the stack are traditional transformer attention layers, using Grouped Query Attention with 32 query heads, 2 KV heads, and a head dimension of 128. These are confirmed specs from the technical report.

Why keep any attention at all? Because Mamba has a known gap. It struggles with precise associative recall, like connecting a specific detail from position 1,000 with something at position 500,000. The fixed-size state means some info gets compressed away over very long sequences.

NVIDIA’s fix: place attention layers at carefully chosen depths through the 88-layer stack. They act like precision tools, handling the long-range connections that Mamba would miss, while Mamba does everything else fast.

The result: the vast majority of the model’s compute happens in linear time. Quadratic attention is used only where it’s needed.

The KV Cache Payoff

This design has a huge side effect that matters a lot for hardware like the DGX Spark.

In a transformer, the KV cache grows with sequence length. Every attention layer stores key and value tensors for every token it has seen. For a model like Qwen 3.5-122B with 12 full attention layers, head dimension 256, and 2 KV heads, that adds up to about 22.9 GiB at 1 million tokens in BF16. (The math: 12 layers x 2 KV heads x 256 dim x 2 bytes x 2 for K+V x 1M tokens.)

Nemotron 3 Super has far fewer attention layers, each with head dimension 128 and 2 KV heads. Because Mamba layers use a fixed-size state (no KV cache growth), only the attention layers add to the KV cache. The bottom line: the KV cache is roughly 3x smaller than Qwen’s at the same context length. On the DGX Spark’s 128 GB of unified memory, you load the 87 GB model, add a relatively small KV cache even at very long contexts, and you still have plenty of room to spare.

For practical purposes, the KV cache almost doesn’t matter with this model.

Latent MoE: Getting 4x More Experts for Free

The Mixture of Experts layer is where things get really clever.

In a standard MoE, each token is routed to one or two “expert” sub-networks from a larger pool. The idea is simple: different experts specialize in different things, and the router learns which expert to call for each token.

The problem is cost. Routing happens at the model’s full hidden dimension, and each expert operates at that same dimension. If you want more experts (for better specialization), routing gets more expensive. If you want to activate more experts per token (for better accuracy), inference gets slower.

Latent MoE solves this with a compression trick. Before routing, token embeddings are projected from the full hidden dimension down to a smaller latent dimension. The router operates in this compressed space, which is much cheaper. Experts also operate on the compressed representations.

The compute you save on compression doesn’t disappear. It gets reinvested. NVIDIA uses it to increase both the total number of experts and the number of experts active per token by the same factor. The result: 4x more experts consulted per token, at approximately the same inference cost as a standard MoE with fewer experts.

Each token effectively gets a committee of 4 specialists deliberating instead of a single expert making a snap judgment. The accuracy improvement is significant, and you don’t pay for it in latency.

Multi-Token Prediction: Built-In Speculative Decoding

Standard language models predict one token at a time. Generate position N, feed it back in, generate position N+1, repeat. This sequential nature is the fundamental bottleneck in text generation speed.

Nemotron 3 Super predicts multiple future tokens from each position simultaneously. The model has shared-weight prediction heads that project from the same internal representation to predict not just the next token, but several tokens ahead.

This serves two purposes. During training, it forces the model to learn longer-range dependencies. You can’t predict three tokens ahead without understanding the broader context. This makes the model smarter.

During inference, it works like built-in speculative decoding. Instead of generating one token per forward pass, the model proposes multiple tokens, verifies them, and keeps the correct ones. For structured output like code and tool calls, where the next few tokens are often very predictable, NVIDIA reports up to 3x wall-clock speedup. General chat won’t see the full 3x, but code generation benefits a lot.

The nice thing: you don’t need a separate draft model. The speculation is built right into the model.

How It Was Trained: The Three-Phase Pipeline

This is where NVIDIA’s openness really shines. They didn’t just release weights. They published the complete methodology, and the numbers are big.

Phase 1: Pretraining. 25 trillion tokens total (10 trillion unique), plus 10 billion additional tokens focused specifically on reasoning, plus 15 million coding problems. The majority of compute ran in NVFP4, which is NVIDIA’s native 4-bit floating point format. This is unusual and important: most models train in higher precision and quantize down later, losing accuracy. Nemotron 3 Super was born in FP4.

Phase 2: Supervised Fine-Tuning. 7 million carefully selected samples from a corpus of 40 million. Coverage spans reasoning, instruction following, coding, safety, and critically, multi-step agent task completion. This phase is where the model learns to be useful, not just knowledgeable.

Phase 3: Reinforcement Learning. 1.2 million environment rollouts across 21 different configurations, using NeMo Gym and NeMo RL frameworks with 37 datasets. This is where the model learns to reason through complex, multi-step problems. This is the kind of thinking that makes it useful as an autonomous agent.

NVIDIA released around 10 of the pretraining datasets publicly, 15 RL training environments, and about 10 of the 37 RL datasets, along with complete training recipes. The Artificial Analysis Openness Index scored this release at 83 out of 100. Only two research labs (Ai2 and MBZUAI) score higher, and their models aren’t anywhere near this performance level.

This kind of transparency is new for a model this good. With enough compute, you could follow their recipe and reproduce the training run yourself.

Why NVIDIA Built This for Agents

The model wasn’t designed for chatbots. Every architectural decision points toward one use case: long-running autonomous agents. The 1M context, the sparse activation, the MoE efficiency. All of it.

When you run a multi-agent system, token consumption explodes. Each agent interaction requires sending the full conversation history, tool outputs, intermediate reasoning steps, and results from other agents. NVIDIA’s numbers suggest multi-agent workflows generate up to 15x more tokens than standard chat.

This creates two problems that kill most models:

Context overflow. At 128K tokens, even large models run out of context in extended agent sessions. The agent either loses early context (and the original goal with it), or you implement complex summarization/RAG schemes that lose fidelity. Nemotron 3 Super’s million-token window means the agent can hold the entire workflow state. Every tool call, every intermediate result, every reasoning step stays in memory without ever truncating.

The cost of thinking. Agents need to reason at every step. If each reasoning call costs as much as a full 120B forward pass, running thousands of agent subtasks gets very expensive very fast. With only 12B active parameters, each call through Nemotron 3 Super costs a fraction of what a dense 120B model would. You get the intelligence of a large model with the economics of a small one.

The benchmarks back this up. On PinchBench, which tests models as actual coding agents (not just answering coding questions), it scores 85.6%. That’s the best open model out there. On DeepResearch Bench, which tests multi-step research over large document sets, NVIDIA’s AI-Q multi-agent system took the number one position. AI-Q is built on top of a fine-tuned Nemotron 3 Super. Worth noting: AI-Q is a full multi-agent system with orchestrator, planner, and researcher sub-agents. It’s not just the base model running solo. But Nemotron 3 Super is the reasoning engine at its core.

Why Sparse MoE is Perfect for the DGX Spark

Here’s something that sounds wrong: on the DGX Spark, a 120B MoE model runs way faster than a smaller 70B dense model. The bigger model is faster. How?

Running It: Three Paths from Zero to Inference

I tested three ways to get Nemotron 3 Super running on the DGX Spark. Here they are, from simplest to most configurable.

Path 1: Ollama (Two Commands)

The fastest possible path. If Ollama is installed on your Spark (it comes pre-installed on DGX OS):

ollama pull nemotron-3-super

ollama run nemotron-3-super

saiyam@spark-5385:~$   ollama pull nemotron-3-super
pulling manifest 
pulling 0fc53cc990a2: 100% ▕███████████████████████████████████████████████████████████████████████████████████████▏  86 GB                         
pulling d02d998e5ae6: 100% ▕███████████████████████████████████████████████████████████████████████████████████████▏  23 KB                         
pulling 02897ca0d6a3: 100% ▕███████████████████████████████████████████████████████████████████████████████████████▏   31 B                         
pulling 9c35241878aa: 100% ▕███████████████████████████████████████████████████████████████████████████████████████▏  509 B                         
verifying sha256 digest 
writing manifest 
success 


saiyam@spark-5385:~$ ollama list

NAME                       ID              SIZE     MODIFIED               
nemotron-3-super:latest    95acc78b3ffd    86 GB    Less than a second ago    
qwen3.5:35b-a3b            3460ffeede54    23 GB    5 days ago                
saiyam@spark-5385:~$  ollama run nemotron-3-super --verbose "Explain the difference between Mamba and Transformer architectures like I'm a DevOps engineer who has never worked with ML."

ollama show nemotron-3-super 
  Model
    architecture        nemotron_h_moe    
    parameters          123.6B            
    context length      262144            
    embedding length    4096              
    quantization        Q4_K_M            
    requires            0.17.1            

  Capabilities
    completion    
    tools         
    thinking      

  Parameters
    temperature    1       
    top_p          0.95    

  License
    NVIDIA Software and Model Evaluation License                                            
    IMPORTANT NOTICE – PLEASE READ AND AGREE BEFORE USING THE NVIDIA LICENSED MATERIALS.    
    ...                                                                                     

Real performance on DGX Spark:

prompt eval rate:  3.51 tokens/s
eval rate: 19.50 tokens/s
eval count:  2504 tokens
total duration:  2m56s

Path 2: llama.cpp from Source (Full Control)

If you want more control over context sizes, quantization, and API serving, you can build llama.cpp from source and run the GGUF model directly. Unsloth has a detailed guide for this: Unsloth Nemotron 3 Super Guide

The key things to know for DGX Spark:

• When building llama.cpp, use -DCMAKE_CUDA_ARCHITECTURES=”121” for the GB10 chip. Without this you’ll fall back to CPU inference.

• The GGUF files are at unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF on Hugging Face.

• NVIDIA recommends temperature 1.0 for general chat, and 0.6 with top_p 0.95 for tool calling.

• Set --ctx-size based on your available memory. On DGX Spark, 16384 to 262144 is practical. Setting it to 1M may trigger CUDA OOM.

• llama-server gives you an OpenAI-compatible API, so VS Code Continue, LangChain, CrewAI, Open WebUI all just work.

The DGX Spark: Quick Hardware Context

For readers unfamiliar with the hardware, the DGX Spark is NVIDIA’s desktop AI computer. The relevant specs:

• Chip: GB10 Grace Blackwell Superchip

• Memory: 128 GB unified LPDDR5x (shared CPU/GPU, 273 GB/s)

• GPU: 6,144 CUDA cores, 5th-gen Tensor Cores, 1 PFLOP FP4 sparse

• CPU: 20-core ARM (10x Cortex-X925 + 10x Cortex-A725)

• Size: 150mm x 150mm x 50mm, 1.2 kg

• Power: 240W

The unified memory is the key. Unlike a discrete GPU where you’re limited by VRAM (24 GB on an RTX 4090, 32 GB on an RTX 5090), the DGX Spark’s 128 GB is coherently shared between CPU and GPU with no PCIe bottleneck. The full 87 GB model lives in one address space.

NVIDIA rates it for models up to 200 billion parameters on a single unit, or 405 billion on two connected Sparks.

Putting It in Context: How Nemotron 3 Super Compares

Here’s the honest comparison against peers, based on published third-party benchmarks:

What I’d Actually Use This For

After spending time with the model, here’s where I see genuine value in the Nemotron + DGX Spark combination:

Openclaw - I am going to put this as the model for openclaw too which is already running on my machine.

Private data analysis. If you work in healthcare, finance, legal, or defense, some data simply cannot leave your building. No cloud provider promise changes the rules. A local model that never touches a network is the only option for these workloads.

Code review and analysis. 73.4% precision on Qodo’s code review benchmark means about three out of four issues it flags are real. That’s useful enough for a local code review helper, especially when you’re working on code you can’t send to an external API.

Long-document reasoning. The million-token context with a tiny KV cache means you can load entire codebases, spec documents, or stacks of research papers and ask questions across everything. No chunking, no RAG pipeline needed. Just load it all and ask.

Where I wouldn’t use it: production serving at scale, real-time latency-critical applications, or model training. The DGX Spark is an inference machine, not a training rig.

The Bigger Picture

Nemotron 3 Super is interesting as a model, but it’s even more interesting as a strategy.

NVIDIA makes the chips (GB10, B200), the inference runtime (NIM, TensorRT-LLM), the training framework (NeMo), and now the models (Nemotron). They’ve released the model, the data, and the recipes. Everything except the hardware to train on.

That’s the play. The more developers build on Nemotron, the more they need NVIDIA hardware. The openness isn’t charity. It’s ecosystem building.

But for us as practitioners, the result is clearly good. We get a top-tier model with full training transparency, running on hardware we can put on a desk. The hybrid Mamba-Transformer architecture with Latent MoE and multi-token prediction isn’t just a research paper. It’s a practical solution for running large models on limited hardware.

NVIDIA has confirmed that Ultra, the bigger sibling at roughly 500 billion parameters, is coming. If Super at 120B is this capable, Ultra will be worth watching closely.

Sources:

• NVIDIA Nemotron 3 Super Technical Report (PDF)

• NVIDIA Technical Blog: Introducing Nemotron 3 Super

• NVIDIA Blog: 5x Higher Throughput for Agentic AI

• Artificial Analysis: Nemotron 3 Super — The New Leader in Open Intelligence

• Qodo: Code Review Analysis

• Ollama DGX Spark Benchmarks

• DGX Spark Performance (LMSYS)

• PinchBench

• My own testing on DGX Spark — 19.5 tok/s (Q4_K_M, Ollama), prompt eval 3.51 tok/s

The core problem isn't moving from one controller to another. It's that Ingress NGINX is held together by annotations. Hundreds of nginx.ingress.kubernetes.io/* annotations that control everything from TLS redirects to rate limiting to sticky sessions to canary deployments. These annotations have no direct equivalent in a new controller. Some map cleanly. Some map partially, with caveats. Some have no equivalent at all. And the existing tooling (ingress2gateway) only handles basic routing — it doesn't tell you what you're losing or how to compensate.

That's why I built ing-switch — an open-source CLI + visual UI that takes you through the full migration lifecycle: scan → analyze → generate → verify → cutover → cleanup.

What ing-switch does

The tool has four commands that map to four stages of migration:

ing-switch scan      # detect your controller + list all ingresses
ing-switch analyze   # map every annotation to the target controller
ing-switch migrate   # generate ready-to-apply manifests
ing-switch ui        # open the visual 4-page migration dashboard at :8080

And it supports two migration targets:

Traefik is the lowest-friction path: you keep kind: Ingress, your team learns almost nothing new, and almost all annotations have a Traefik equivalent. Gateway API is the future-proof path: standardized, implementation-agnostic, and where the ecosystem is heading — but it requires more preparation for partial-support annotations.

Annotation coverage: the hard part

This is the part most migration tools skip. ing-switch maps over 50 nginx.ingress.kubernetes.io/* annotations for both targets. Here's a sample:

The tool shows you exactly which category each annotation falls into: fully supported (just apply the generated YAML), partial (the YAML is generated, but read the note about what's different), or unsupported (manual work required, with guidance on the best alternative).

Installation

# macOS arm64
curl -L https://github.com/saiyam1814/ing-switch/releases/latest/download/ing-switch-darwin-arm64 -o ing-switch
chmod +x ing-switch && sudo mv ing-switch /usr/local/bin/

# Linux amd64
curl -L https://github.com/saiyam1814/ing-switch/releases/latest/download/ing-switch-linux-amd64 -o ing-switch
chmod +x ing-switch && sudo mv ing-switch /usr/local/bin/

Or build from source:

git clone https://github.com/saiyam1814/ing-switch.git
cd ing-switch
make build   # builds React UI then embeds it in the Go binary
./ing-switch --help

The Dashboard

The easiest way to understand what ing-switch does is to open the UI:

ing-switch ui   # opens at localhost:8080

Page 1: Detect

The first page scans your cluster and discovers every Ingress resource across all namespaces, identifies the controller type and version, and flags each ingress by complexity.

The tool shows a countdown banner — Ingress NGINX retires March 2026 — and lets you scope the scan to a specific namespace or scan everything.

After clicking Scan Cluster, the tool connects to your cluster via kubeconfig and enumerates every networking.k8s.io/v1 Ingress object. In my test cluster (a vcluster running 11 production-realistic ingresses), the sidebar updated immediately to show "11 ingresses · retiring · ingress-nginx":

The sidebar shows your cluster name, ingress count, and controller status. "Retiring" badge means the detected controller is Ingress NGINX.

Page 2: Analyze

The Analyze page is where you decide your migration target. You pick between Traefik v3 (labeled "Lowest friction") or Gateway API via Envoy Gateway (labeled "Future-proof standard"), then click Analyze Compatibility.

The engine runs through every annotation on every ingress and produces a per-ingress compatibility matrix. Each annotation gets a status badge:

• Green (supported) — the generated YAML covers this fully; just apply it

• Yellow (partial) — the feature works but with limitations; the note explains exactly what's different

• Red (unsupported) — no direct equivalent exists; the tool explains the best available workaround

This is the output that replaces weeks of reading changelogs and GitHub issues.

Page 3: Migrate

The Migrate page generates the complete output directory of Kubernetes manifests. Click Generate Migration Files and the tool writes every file needed for a zero-downtime cutover.

The checklist walks you through every step in order, so you can't accidentally skip a dependency (like applying GatewayClass before HTTPRoutes).

After generation, each step gains View File, Dry-run, and Apply buttons:

The tool generated 27 migration files for Traefik across 11 ingresses. Each checklist item links to the file it creates, and you can preview the YAML before applying.

The Gaps tab gives you the executive summary — which ingresses are fully compatible, which have auto-generated workarounds, and which need manual review:

"Needs Review" ingresses are listed with the specific annotations that require manual attention, so you know exactly what work remains before cutover.

Page 4: Validate

After applying the generated manifests, the Validate page gives you a structured checklist of post-migration checks and lets you run live validation against the cluster.

The checklist covers TLS verification, auth testing, rate limit validation, canary routing verification, and a 24-hour monitoring window before removing NGINX.

The CLI: same power, no browser

Everything the UI does is also available as CLI commands, which makes it scriptable and CI-friendly:

# Scan your cluster
$ ing-switch scan
Cluster: production-cluster
Controller: ingress-nginx v1.9.4 (namespace: ingress-nginx)

NAMESPACE    NAME              HOSTS                     ANNOTATIONS  TLS
ecommerce    ecommerce-shop    shop.example.com          13           yes
security     payment-api       payments.example.com      6            yes
messaging    realtime-chat     chat.example.com          2            yes
...
11 ingresses found across 6 namespaces

# Analyze annotation compatibility for Gateway API
$ ing-switch analyze --target gateway-api
INGRESS                       SUPPORTED   PARTIAL   UNSUPPORTED
ecommerce/ecommerce-shop      6           5         2
security/payment-api          3           2         1
...

# Generate all migration manifests
$ ing-switch migrate --target traefik --output-dir ./migration
Generated 27 files across 11 ingresses

migration/
├── 00-migration-report.md      # Full annotation analysis
├── 01-install-traefik/         # Helm install script + values.yaml
├── 02-middlewares/             # Traefik Middleware CRDs
├── 03-ingresses/               # Updated Ingress resources
├── 04-verify.sh                # Test script per hostname
├── 05-dns-migration.md         # DNS cutover guide
└── 06-cleanup/                 # Remove NGINX after cutover

How it handles the tricky cases

Three annotation scenarios trip up almost every migration, and ing-switch handles all of them:

HTTP→HTTPS redirect

The naive approach — putting both a RequestRedirect filter and a backend rule in the same HTTPRoute — causes a redirect loop where HTTPS traffic gets redirected back to HTTPS. ing-switch generates two separate HTTPRoutes using Gateway API sectionName to attach each route to the correct listener:

# <name>-redirect: attached to HTTP listener only (sectionName: http)
# Returns 301/302 for all incoming HTTP requests

# <name>: attached to HTTPS listener (sectionName: https-0)
# Routes HTTPS traffic to your backends

This is the correct pattern but requires knowing the Gateway API spec deeply enough to spot the constraint. The tool just does it right.

Regex paths

NGINX annotations like use-regex: "true" and paths like /app(/|$)(.*) are common. Gateway API's PathPrefix type rejects regex characters. ing-switch auto-detects paths containing (, ), |, [, ] and switches them to PathMatch.RegularExpression type — even when the use-regex annotation is absent.

Timeout constraints

proxy-read-timeout: 300 and proxy-connect-timeout: 5 look straightforward to map. But Gateway API enforces backendRequest ≤ request, and mapping read→backendRequest (300s) and connect→request (5s) would violate that constraint. ing-switch maps only proxy-read-timeout → backendRequest and intentionally omits proxy-connect-timeout, with a note explaining why.

Generated output: Gateway API example

For a single ingress with SSL redirect, session affinity, and rate limiting, the Gateway API migration generates:

migration/
├── 00-migration-report.md
├── 01-install-gateway-api-crds/
│   └── install.sh
├── 02-install-envoy-gateway/
│   ├── helm-install.sh
│   └── values.yaml
├── 03-gateway/
│   ├── gatewayclass.yaml      # GatewayClass (Envoy Gateway)
│   └── gateway.yaml           # Gateway: HTTP + HTTPS listeners
├── 04-httproutes/
│   ├── ecommerce-ecommerce-shop-redirect.yaml   # HTTP→HTTPS (sectionName: http)
│   └── ecommerce-ecommerce-shop.yaml            # HTTPS backend (sectionName: https-0)
├── 05-policies/
│   └── ecommerce-ecommerce-shop-btp.yaml        # BackendTrafficPolicy (rate limit)
├── 06-verify.sh
└── 07-cleanup/
    └── remove-nginx.sh

Example ingresses

The repo includes 11 production-realistic NGINX Ingress configurations covering every major annotation category you're likely to have in a real cluster:

You can apply them to a test cluster and run the full migration against them:

kubectl apply -f examples/
ing-switch migrate --target gateway-api --output-dir ./migration-examples
kubectl apply --dry-run=client -f ./migration-examples/03-gateway/
kubectl apply --dry-run=client -f ./migration-examples/04-httproutes/

Zero-downtime migration strategy

The tool is designed around a zero-downtime approach:

1. Install the new controller alongside NGINX — both run simultaneously; DNS still points to NGINX

2. Apply generated manifests — Middlewares, HTTPRoutes, Gateway

3. Verify the new controller — run 06-verify.sh to test each hostname against the new IP

4. Shift DNS — update your DNS records to point to the new controller's LoadBalancer IP

5. Monitor for 24 hours — watch for 5xx errors, auth failures, session issues

6. Remove NGINX — run 06-cleanup/remove-nginx.sh

The Migrate page's step-by-step checklist mirrors this order and locks steps until their dependencies are checked off.

Try it

# Install
curl -L https://github.com/saiyam1814/ing-switch/releases/latest/download/ing-switch-darwin-arm64 -o ing-switch
chmod +x ing-switch && sudo mv ing-switch /usr/local/bin/

# Point at your cluster
export KUBECONFIG=~/.kube/config

# Open the visual UI
ing-switch ui

The source, examples, and annotation mapping database are at github.com/saiyam1814/ing-switch.

The annotation mapping database lives in pkg/analyzer/compatibility.go (status + target resource per annotation) and pkg/server/guides.go (human-readable what/fix/example per annotation). PRs for additional annotation mappings are welcome.

March 2026 is closer than it looks. The tools are ready.

Introduction

msfconsole is the heart of the Metasploit Framework and one of the most powerful tools used by penetration testers to identify, exploit, and validate security vulnerabilities. In real-world security assessments as well as Capture The Flag (CTF) challenges, msfconsole is often used to automate and streamline exploitation workflows.

In this blog, we will explore how to use msfconsole from Kali Linux to exploit an intentionally vulnerable machine, Metasploitable2, in a safe and controlled lab environment.

Both machines are hosted on Oracle VM VirtualBox and configured on the same internal network. This setup allows us to simulate real attack scenarios while maintaining proper ethical boundaries.

The goal of this blog is to:

• Understand what msfconsole is and why it is used

• Learn how attackers interact with vulnerable services using Metasploit

• Gain hands-on experience with a realistic exploitation lab

Note: All demonstrations in this blog are performed on machines owned by us or intentionally designed to be vulnerable. Never use these techniques on unauthorized systems.

In the next section, we will briefly look at the lab architecture before launching msfconsole and beginning the exploitation process.

Press enter or click to view image in full size

Setting Up msfconsole and Metasploitable2 (Step-by-Step Lab Setup)

Before launching any exploitation using msfconsole, we must ensure that both the attacker and the vulnerable target are properly set up and reachable. This section covers the complete setup process for msfconsole on Kali Linux and the Metasploitable2 vulnerable server.

1. Setting Up the Attacker Machine (Kali Linux)

Why Kali Linux?

Kali Linux comes pre-installed with hundreds of penetration testing tools, including the Metasploit Framework.

Verify Metasploit Installation

On Kali, Metasploit is installed by default. To verify:

msfconsole --version

If Metasploit is installed correctly, you will see version details.

— — — — — — — — — — — — — — — — —

Start msfconsole

msfconsole

On first launch, Metasploit may:

• Initialize its database

• Create required configuration files

You should now see the familiar msf6 > prompt.

This confirms that msfconsole is ready to use.

Press enter or click to view image in full size

— — — — — — — — — — — — — — — — —

2. Setting Up the Target Machine (Metasploitable2)

What is Metasploitable2?

Metasploitable2 is a deliberately vulnerable Linux machine created for practicing penetration testing techniques.

Start Metasploitable2 VM

• Launch Metasploitable2 in Oracle VM VirtualBox

• Wait until it boots to the login screen

Default Credentials

Username: msfadmin
Password: msfadmin

Login successfully to access the system.

Check IP Address of Metasploitable2

ifconfig

Example output:

inet addr:192.168.56.101

Press enter or click to view image in full size

Note this IP address, as it will be used as the target (RHOSTS) inside msfconsole.

— — — — — — — — — — — — — — — — —

3. Ensure Both Machines Are on the Same Network

Both VMs must be configured with:

• Network Adapter: Host-only Adapter

• Name: VirtualBox Host -Only Ethernet Adapter

This ensures:

• Kali ↔ Metasploitable2 communication

• No internet exposure (safe lab)

Press enter or click to view image in full size

Press enter or click to view image in full size

— — — — — — — — — — — — — — — — —

4. Test Connectivity (Very Important)

From Kali Linux:

ping 192.168.56.101

If you receive replies, your lab network is working correctly.

Press enter or click to view image in full size

— — — — — — — — — — — — — — — — —

5. Confirm Target Visibility Using Nmap

Before using Metasploit, attackers always enumerate first.

nmap -sV 192.168.56.101

You should see multiple intentionally vulnerable services, such as

• FTP (vsftpd 2.3.4)

• SSH

• Samba

• Tomcat

Press enter or click to view image in full size

This confirms that Metasploitable2 is ready for exploitation.

Setup Checklist

✔ Kali Linux boots successfully
✔ msfconsole launches without errors
✔ Metasploitable2 is accessible
✔ Both machines are on the same subnet
✔ Ping & Nmap scans work

Once all checks pass, your lab is fully prepared.

Basic msfconsole Commands (Getting Comfortable with the Interface)

Before jumping into exploitation, it’s important to understand the basic operating system–style commands and navigation used inside msfconsole. This section helps beginners feel confident while working in the Metasploit environment.

We are using Kali Linux with the Metasploit Framework.

Starting msfconsole

Open a terminal in Kali Linux and run:

msfconsole

Once loaded, you will see:

msf6 >

This prompt indicates that msfconsole is ready to accept commands.

Getting Help in msfconsole

Show All Commands

help

or simply:

?

This lists all available commands, similar to using help in an operating system shell.

Press enter or click to view image in full size

Navigation Commands (OS-Like Basics)

CommandDescriptionpwdShows the current module pathcdChange module directorylsList available modulesclearClear the screen

Example:

pwd
ls

These commands work inside Metasploit, not the Linux filesystem.

Searching for Modules

One of the most used commands:

search <keyword>

Example:

search ftp
search samba
search vsftpd

Press enter or click to view image in full size

This helps you quickly find:

• Exploits

• Auxiliary scanners

• Payloads

Understanding Module Types

Metasploit is organized into modules:

Metasploit is organized into different module types, each designed for a specific purpose in the penetration testing lifecycle.

Press enter or click to view image in full size

You can list them using:

ls exploit
ls auxiliary

Using a Module

To select a module:

use exploit/unix/ftp/vsftpd_234_backdoor

Press enter or click to view image in full size

Once selected, the prompt changes to:

msf6 exploit(unix/ftp/vsftpd_234_backdoor) >

This tells you which module is currently active.

Viewing & Setting Options

Show Required Options

show options

Press enter or click to view image in full size

Set Target IP

set RHOSTS 192.168.56.101

Set Port (if needed)

set RPORT 21

To verify:

show options

Press enter or click to view image in full size

Running a Module

run

or

exploit

Both commands do the same thing.

Press enter or click to view image in full size

Session Management Basics

After successful exploitation:

sessions

Interact with a session:

sessions -i 1

Exit session:

Press enter or click to view image in full size

exit

Exiting Modules & msfconsole

In Metasploit, use back to leave the current module and return to the main console. Use quit or exit to close msfconsole completely.

Key Takeaways

✔ msfconsole feels like a mini operating system
✔ search, use, and show options are core commands
✔ Always understand a module before running it
✔ Enumeration comes before exploitation

Conclusion

In this blog, we explored how msfconsole, the core interface of the Metasploit Framework, can be used to exploit a vulnerable FTP service on Metasploitable2 from an attacker machine running Kali Linux.

Starting from proper lab setup and network configuration, we moved through the essential stages of a penetration test: service enumeration, exploit selection, configuration, and execution. By exploiting the outdated vsftpd 2.3.4 service, we demonstrated how a single vulnerable service can lead to full system compromise when basic security practices are ignored.

This exercise highlights several important lessons:

• Enumeration is more important than exploitation

• Outdated services pose serious security risks

• Automation tools like msfconsole must be used with understanding, not blindly

• Ethical hacking is about learning and improving security, not breaking systems

Practicing in a controlled environment like Metasploitable2 helps build a strong foundation for CTFs, real-world penetration testing, and defensive security awareness.

In future labs, this knowledge can be extended to:

• Exploiting other services such as Samba and Tomcat

• Using Meterpreter for advanced post-exploitation

• Understanding how blue teams detect and prevent such attacks

Always remember: with great power comes great responsibility. Use these skills only where you have explicit permission.

Follow Kubesimplify on Hashnode, Twitter/X and LinkedIn. Join our Discord server to learn with us!