I recently got my hands on an NVIDIA DGX Spark, and the first thing I wanted to figure out was: how do I access this thing from anywhere? Whether I'm at a coffee shop, at a conference, or on a different network entirely — I want to just ssh in and get to work.

The answer? Tailscale. It took me about 10 minutes to set up, and now I can SSH into my Spark from any device, on any network, anywhere in the world. I even set up a friend with access — simultaneously — without giving them my credentials. Here's exactly how I did it.

Why Tailscale?

Tailscale creates a private mesh network (called a "tailnet") between your devices. No port forwarding, no static IPs, no VPN server to maintain. You install it on your devices, log in with the same account, and they can talk to each other. It's built on WireGuard, so it's fast and encrypted.

For the DGX Spark, this means:

• No need to be on the same WiFi network

• No need to mess with your router settings

• Works behind NATs and firewalls

• Encrypted end-to-end

Prerequisites

Before starting, make sure your DGX Spark:

• Is running Ubuntu 24.04 or newer

• Has internet connectivity

• You have sudo access

Here's what my system looked like:

$ lsb_release -a
No LSB modules are available.
Distributor ID:    Ubuntu
Description:    Ubuntu 24.04.3 LTS
Release:    24.04
Codename:    noble

A quick ping to confirm internet:

$ ping -c 3 google.com
64 bytes from tzdela-ba-in-x0e.1e100.net: icmp_seq=1 ttl=118 time=15.3 ms
64 bytes from tzdela-ba-in-x0e.1e100.net: icmp_seq=2 ttl=118 time=13.7 ms
64 bytes from tzdela-ba-in-x0e.1e100.net: icmp_seq=3 ttl=118 time=17.2 ms

--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss

And verify sudo access:

$ sudo whoami
root

Good to go.

Step 1: Install Tailscale on the DGX Spark

SSH into your Spark (or use a directly connected keyboard/monitor) and run:

# Update package list and install prerequisites
sudo apt update
sudo apt install -y curl gnupg

# Add Tailscale signing key
curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/noble.noarmor.gpg | \
  sudo tee /usr/share/keyrings/tailscale-archive-keyring.gpg > /dev/null

# Add Tailscale repository
curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/noble.tailscale-keyring.list | \
  sudo tee /etc/apt/sources.list.d/tailscale.list

# Install Tailscale
sudo apt update
sudo apt install -y tailscale

You'll see the repository being added and the package installing:

# Tailscale packages for ubuntu noble
deb [signed-by=/usr/share/keyrings/tailscale-archive-keyring.gpg] https://pkgs.tailscale.com/stable/ubuntu noble main
...
Setting up tailscale (1.94.2) ...
Created symlink /etc/systemd/system/multi-user.target.wants/tailscaled.service → /usr/lib/systemd/system/tailscaled.service.

Verify the installation:

$ tailscale version
1.94.2
  tailscale commit: 0a29cf18b56e478b9cd33af07755fcae90d5171a
  long version: 1.94.2-t0a29cf18b-g3f044c9f6
  go version: go1.25.5

Check the service is running:

saiyam@spark-5223:~$ sudo systemctl status tailscaled --no-pager
[sudo] password for saiyam: 
● tailscaled.service - Tailscale node agent
     Loaded: loaded (/usr/lib/systemd/system/tailscaled.service; enabled; preset: enabled)
     Active: active (running) since Tue 2026-04-07 11:13:14 UTC; 9min ago
       Docs: https://tailscale.com/docs/
   Main PID: 2410 (tailscaled)
     Status: "Connected; saiyam911@gmail.com; 100.120.233.78 fd7a:115c:a1e0::f83a:e94e"
      Tasks: 22 (limit: 153561)
     Memory: 45.4M (peak: 53.7M)
        CPU: 615ms
     CGroup: /system.slice/tailscaled.service
             └─2410 /usr/sbin/tailscaled --state=/var/lib/tailscale/tailscaled.…

The status says "Needs login" — that's expected. We'll authenticate next.

Step 2: Connect the Spark to Your Tailnet

This is the magic step:

$ sudo tailscale up

To authenticate, visit:

    https://login.tailscale.com/a/1ff5e3e9017787

Open that URL in any browser, log in with your account (Google, GitHub, Microsoft — whatever your org uses), and you'll see:

Login successful. Your device spark-5223 is logged in

Back on the Spark terminal, you'll see:

Success.
Some peers are advertising routes but --accept-routes is false

That's it on the Spark side. Your DGX Spark is now part of your private Tailscale network with the hostname spark-5223.

Note: The --accept-routes message is harmless for SSH access. You can ignore it. If you ever need subnet routing, run sudo tailscale up --accept-routes.

Step 3: Install Tailscale on Your Laptop

macOS

• Option A: Download from the Mac App Store (search "Tailscale")

• Option B: Download the .pkg from tailscale.com/download

Open the app, click Log in, and sign in with the same account you used on the Spark.

Windows

1. Download the installer from tailscale.com/download

2. Run the .msi file

3. Launch Tailscale from the system tray

4. Log in with the same account

Linux

Same commands as the Spark:

sudo apt update
sudo apt install -y curl gnupg

curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/noble.noarmor.gpg | \
  sudo tee /usr/share/keyrings/tailscale-archive-keyring.gpg > /dev/null

curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/noble.tailscale-keyring.list | \
  sudo tee /etc/apt/sources.list.d/tailscale.list

sudo apt update
sudo apt install -y tailscale
sudo tailscale up

Step 4: SSH Into Your Spark From Anywhere

First, confirm both devices see each other:

$ tailscale status
100.104.142.22  spark-5223           saiyamxxx@  linux  -
100.108.115.75  saiyams-macbook-pro  saiyam9xxx@  macOS  -

You should see your Spark listed. Now, simply:

ssh saiyam@spark-5223

That's it. Tailscale's MagicDNS resolves spark-5223 to the right Tailscale IP automatically. No need to remember IP addresses.

If MagicDNS isn't working for some reason, use the Tailscale IP directly:

# Find the IP
tailscale status
# Look for spark-5223 and note the 100.x.x.x address

ssh saiyam@100.104.142.22

Setting Up SSH Key Authentication

For passwordless SSH access, set up key-based authentication. If you already have an SSH key (check ~/.ssh/id_ed25519.pub or ~/.ssh/id_rsa.pub), add it to the Spark:

# Copy your public key to the Spark (will ask for password once)
ssh-copy-id saiyam@spark-5223

Or manually add it on the Spark:

# On the Spark — append the public key
echo "your-public-key-here" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
chmod 700 ~/.ssh

After that, SSH works without a password prompt.

Note: Password authentication still works alongside SSH keys. You don't have to choose one or the other.

What About My Second Laptop?

This is the beauty of Tailscale — just install and log in:

1. Install Tailscale on the second laptop (using the steps above for your OS)

2. Log in with the same account

3. Run ssh saiyam@spark-5223

No extra configuration on the Spark. Every device on your tailnet can reach every other device automatically.

Sharing Your Spark With a Friend

What if a friend also needs SSH access to your Spark — simultaneously, from their own laptop? You don't need to create a new Tailscale account for them. Use a pre-auth key to add their device to your tailnet.

Generate a Pre-Auth Key

1. Go to the Tailscale Admin Console

2. Click "Generate auth key..."

3. Enable Reusable if you want it to work for multiple devices

4. Set an expiration as needed

5. Copy the key (starts with tskey-auth-...)

Your Friend's Setup (macOS)

1. Install Tailscale from the Mac App Store

2. Important: If they're already logged in to their own Tailscale account, they need to leave it first:

   sudo tailscale logout
   

3. Join your tailnet using the pre-auth key:

   sudo tailscale up --auth-key=tskey-auth-xxxxxxxxxxxx
   

4. That's it — their Mac is now on your tailnet. No login, no email needed.

Add Their SSH Key to the Spark

Your friend should generate an SSH key on their Mac (if they don't have one):

ssh-keygen -t ed25519

Then share their public key with you (the contents of ~/.ssh/id_ed25519.pub). On the Spark, add it:

echo "ssh-ed25519 AAAA...their-key-here... friend@hostname" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

Now your friend can SSH in directly:

ssh saiyam@spark-5223

No password prompt — the key handles authentication automatically. SSH automatically tries keys from the default location (~/.ssh/id_ed25519), so your friend does not need to use ssh -i.

Verify it all works:

$ tailscale status
100.104.142.22  spark-5223           saiyamxxx@  linux  -
100.67.209.38   rohits-macbook-pro   saiyamxxx@  macOS  -
100.108.115.75   saiyams-macbook-pro  saiyamxxx@  macOS  -

Three devices, one tailnet, simultaneous SSH access.

Tip: You can manage access from the Tailscale Admin Console. To revoke someone's access, remove their device from the console and delete their key from ~/.ssh/authorized_keys on the Spark.

Troubleshooting

"No Matching Peer" Error

If your friend gets a "no matching peer" error when trying to SSH, it means they're on a different tailnet — not yours.

The 100.x.x.x Tailscale IPs are only reachable between devices on the same tailnet. The fix:

# Friend logs out of their own tailnet
sudo tailscale logout

# Friend joins YOUR tailnet with your pre-auth key
sudo tailscale up --auth-key=tskey-auth-xxxxxxxxxxxx

SSH Connection Timeout

If tailscale ping works but SSH times out:

# On the Spark — check SSH is running
sudo systemctl status ssh

# Check firewall isn't blocking
sudo ufw status

# If SSH isn't running
sudo systemctl start ssh

# If firewall is active and blocking
sudo ufw allow 22/tcp

Also check SSH is listening on all interfaces:

$ ss -tlnp | grep 22
LISTEN  0  4096  0.0.0.0:22  0.0.0.0:*  users:(("sshd",...))
LISTEN  0  4096     [::]:22     [::]:*  users:(("sshd",...))

If SSH is only listening on a specific IP, edit /etc/ssh/sshd_config to ensure ListenAddress is not restricted, then sudo systemctl restart ssh.

Permission Denied (publickey, password)

This means SSH connected but authentication failed. Either:

• Your SSH key isn't in ~/.ssh/authorized_keys on the Spark

• You're using a non-default key path (use ssh -i /path/to/key)

• Password authentication is disabled in sshd_config

Check the authorized keys on the Spark:

cat ~/.ssh/authorized_keys

Make sure your public key is listed there.

Useful Commands Cheat Sheet

Pro Tips

1. Tailscale starts on boot — the tailscaled service is enabled by default, so your Spark will rejoin the tailnet automatically after a reboot.

2. Forward ports for Jupyter — if you run JupyterLab on your Spark:

   ssh -L 8888:localhost:8888 saiyam@spark-5223
   

   Then open http://localhost:8888 in your browser.

3. File transfers work too:

   scp model.bin saiyam@spark-5223:~/models/
   

4. Check who's connected — on the Spark, see active SSH sessions:

   who
   

5. Tailscale admin console — monitor all devices, manage keys, and remove devices at login.tailscale.com/admin.

Cleanup (If Needed)

If you ever want to remove Tailscale from your Spark:

sudo tailscale down
sudo apt remove --purge tailscale
sudo rm /etc/apt/sources.list.d/tailscale.list
sudo rm /usr/share/keyrings/tailscale-archive-keyring.gpg
sudo apt update

To restore: re-run installation steps 1-2.

Wrapping Up

The whole setup took me about 10 minutes. Now I can SSH into my DGX Spark from my MacBook at home, my second laptop on the go, and even my friend can access it simultaneously from his MacBook — all without any port forwarding, static IPs, or VPN servers.

The key takeaways:

• For yourself: Install Tailscale on both devices, log in with the same account, ssh in

• For friends: Generate a pre-auth key, have them join your tailnet, add their SSH public key to the Spark

• Troubleshooting: Make sure all devices are on the same tailnet, SSH is running, and keys are in authorized_keys

Just ssh saiyam@spark-5223 — from anywhere in the world.

I also used this at 30000 feet in the air!

https://x.com/SaiyamPathak/status/2032098978213528037?s=20

Yesterday, Anthropic accidentally published a 59.8 MB source map file inside version 2.1.88 of their @anthropic-ai/claude-code npm package. The build pipeline was configured to generate source maps, and the packaging step — whether it was a missing .npmignore rule or a misconfigured files field in package.json — failed to exclude them. One packaging oversight, and ~512,000 lines of TypeScript source were public. Anthropic's DMCA notice eventually took down over 8,100 GitHub repositories. A clean-room rewrite called claw-code hit 50,000 stars in about two hours and now about 100k+.

Within last 24 hours, the internet was flooded with hot takes. Architecture diagrams. The virtual pet system. Thread after thread of "I read the entire codebase and here's what I found." Sites like ccunpacked.dev did genuinely good visual walkthroughs of the high-level structure.

But let's be real — nobody read 512,000 lines of TypeScript. I certainly didn't, and I'm skeptical of anyone who claims they did. What I did do was feed the source into Claude, systematically analyzed the key modules, cross-referenced what I found against public documentation and other analyses, and verified the claims I'm about to make against the actual code. If you've read other "I analyzed the leak" posts, they probably all used a similar workflow. The difference I'm going for here is honesty about the process.

The Core Loop Is a State Machine, and That's the Whole Point

The agent loop lives in query.ts. It's exactly 1,729 lines (I checked), structured as an async generator function called queryLoop wrapping a while(true) loop. The code itself, in an internal comment, references "7 continue sites" — seven distinct points where the loop yields control and decides what to do next.

The actual function signature:

async function* queryLoop(
  params: QueryParams,
  consumedCommandUuids: string[],
): AsyncGenerator<
  StreamEvent | RequestStartEvent | Message | TombstoneMessage | ToolUseSummaryMessage,
  Terminal
>

Why does this matter? Because most agent frameworks treat the LLM call as the center of gravity. Send a prompt, get a response, run a tool, repeat. That works fine for demos. It falls apart the moment you need to pause a session, resume it later, serialize state, handle errors mid-turn, or compose multiple agents together.

The generator pattern makes every loop iteration an explicit state transition. You can yield control at each of the seven points without losing state. You can test individual stages. You can add compaction, permission checks, or budget tracking as stages rather than side effects bolted onto a callback chain.

If you're building an agent and your core loop is a simple while with await model.chat() in the middle, this is the pattern to study.

Five Compaction Strategies (Not a Neat Stack)

Every long-running agent eventually fills its context window. Most frameworks handle this by truncating old messages. Claude Code has five distinct strategies — though I want to be clear, these aren't a clean "Layer 1 through 5" hierarchy like some other posts have described. They're composable strategies that kick in under different conditions:

Snip prunes older messages for quick headroom. Fast and lossy.

Microcompact targets tool outputs specifically. A 5,000-line file read gets saved to disk; the model sees a summary with a reference. Two implementations handle this: microCompact.ts and apiMicrocompact.ts. This alone is a big deal — a single uncompressed tool output can eat half your context window.

Context Collapse progressively compresses older conversation segments while keeping recent context sharp. It's still behind a CONTEXT_COLLAPSE feature flag, with dedicated persistence types (ContextCollapseCommitEntry, ContextCollapseSnapshotEntry) to survive session restarts. Not yet fully shipped.

Autocompact is full-conversation summarization at configurable token thresholds. Replaces older history with a summary.

Reactive Compact is the emergency brake — behind the REACTIVE_COMPACT feature flag. When the API returns a 413 (payload too large), this aggressively compacts everything so your session doesn't die. Without this, one bad tool output would brick the conversation.

Now, I've seen posts claiming "no other framework has this." That was arguably true in 2025, but it's not true now. Microsoft's Agent Framework has composable multi-strategy compaction pipelines. LangChain Deep Agents (shipped March 15, 2026) does filesystem offloading plus multi-frequency summarization. Google ADK has sliding window with summarization.

What sets Claude Code apart isn't that it has compaction — it's the granularity. Five strategies, two of them still being iterated on behind feature flags. That reflects the kind of edge cases you only discover at scale.

Deferred Tool Loading

This is probably the most practical pattern in the codebase for anyone building agents.

When you connect MCP servers, you might have 200+ tools available. Sending all those schemas on every API call wastes thousands of tokens. Claude Code's solution: mark tools with defer_loading: true. The model doesn't see them. Instead, it has a single meta-tool called ToolSearch (the internal class is ToolSearchTool, but the model-facing name is ToolSearch — defined as TOOL_SEARCH_TOOL_NAME = 'ToolSearch' in the constants). When the model needs a capability, it calls ToolSearch with a query:

User: "Deploy this to my Kubernetes cluster"

Model calls ToolSearch("kubernetes deploy")
  -> System fuzzy-matches deferred tool descriptions
  -> Injects matching schemas into the conversation
Model now has the tools it needs.

The model goes from ~20 core tools to access to hundreds, without the upfront token cost.

This pattern has spread. The OpenAI Agents SDK now has deferLoading: true with tool search (requires GPT-5.4+). ZeroClaw implements nearly identical deferred loading. CrewAI 1.10.2a1 (March 2026) added dynamic tool injection via Anthropic's tool search API.

But there's still no framework-agnostic library for this. The core is straightforward — fuzzy matching over tool descriptions, schema injection on demand, MCP compatibility. If someone built this as a standalone package, it'd be useful immediately.

Default-Deny Permissions With a Graceful Fallback

Claude Code's permission system is built on default-deny. Every tool has two permission-relevant properties defined in Tool.ts:

• isReadOnly — defaults to false (assume the tool writes)

• isDestructive — defaults to false

Tools must explicitly declare their risk profile. The permission system then layers rule-based checks (alwaysAllow/alwaysDeny rules), pre-tool-use hooks (which can modify input, block execution, or log), and an auto-mode safety classifier.

The part I found most interesting is the denial tracking in denialTracking.ts — just 46 lines:

3 consecutive denials → shouldFallbackToPrompting() returns true
20 total denials in a session → same result

If the user keeps saying "no," the system stops running in auto-mode and starts asking for explicit permission on every action. Most agent frameworks either keep retrying or hard-stop. Claude Code's approach gracefully degrades: "You're uncomfortable with what I'm doing, so I'll check before each step."

Small file, big principle.

The Cost Engineering

These are the details that only matter at Anthropic's scale, but they're instructive:

Sticky-on latches. When a feature flag activates during a session, it stays on for the rest of that session. Flipping it back would change the system prompt, which busts the prompt cache. The promptCacheBreakDetection.ts file tracks 14 distinct state fields that can invalidate the cache — system prompt hash, tool schema hashes, model changes, beta headers, effort values, and more. Sticky latches prevent unnecessary cache invalidation from mode toggles.

Tool result persistence. Large outputs get written to disk; the model sees a preview. This isn't just context management — it keeps the cache prefix stable.

Schema stability. Tool schemas assembled once at session start, held stable throughout. MCP tools can come and go, but the core schema block doesn't change.

At scale, these optimizations compound significantly.

What I Actually Took Away From This

I'm not going to pretend the leak is a startup idea list or that I discovered things nobody else saw. But analyzing the code did crystallize a few things:

Context management is harder than it looks. Five strategies, two still behind feature flags, a dedicated promptCacheBreakDetection.ts tracking 14 vectors. This is not a solved problem, even for Anthropic.

Deferred tool loading is becoming table stakes. Claude Code, OpenAI, ZeroClaw, CrewAI — multiple teams independently arrived at the same pattern. If you're building an agent with more than ~20 tools and you're not doing this, you're wasting tokens.

Permission design matters more than permission features. The denial tracking system is 46 lines. The principle it encodes — "degrade gracefully when the user loses trust" — is more important than any specific implementation detail.

The real work is in the orchestration. The model call is one stage out of seven in the main loop. Everything else — state management, compaction, tool loading, permissions, cost optimization — is where the engineering actually lives.

Being Honest About the Process

Every blog you've read about this leak was written with AI assistance. This one included. I used Claude to analyze the source modules, identify patterns, and draft the initial structure. I then fact-checked every claim — cross-referencing the actual source code, public documentation, news coverage, and other technical analyses. Where I found errors in my initial draft (and there were several), I corrected them.

The value isn't in pretending I manually read half a million lines of code. It's in doing the verification work to make sure what I'm telling you is actually true. In a sea of AI-generated analysis of AI-generated code, accuracy is the differentiator.

If you spot something wrong, tell me — I'd rather correct it than let it stand.

I write about cloud-native, AI engineering, and the infrastructure that makes modern software work. Find me on Twitter or LinkedIn.

Since then, ingress-nginx was officially archived (March 24, 2026). March 31 is end of life -- zero security patches after that date.

Based on community feedback from KubeCon, this is the biggest update yet: 119 annotations (up from 50), Gateway API with Traefik as the provider (the #1 request), and impact ratings on every annotation so you know exactly what matters.

This post walks through a complete end-to-end migration on a vind cluster with actual command outputs.

Why You Need to Migrate Now

• Nov 11, 2025: Kubernetes SIG Network announces ingress-nginx retirement

• Jan 29, 2026: Joint statement from Kubernetes Steering + Security Response Committees urging immediate migration

• Mar 24, 2026: GitHub repository archived (read-only)

• Mar 31, 2026: End of life -- zero support from this date

Chainguard maintains a fork for CVE-level fixes only -- no features, no community PRs, no pre-built images. You're on your own.

The Three Migration Paths

The Annotation Problem

The real complexity isn't swapping controllers -- it's the annotations. A typical production Ingress has 10-15 NGINX annotations for SSL, auth, rate limiting, CORS, session affinity, and more.

ing-switch maps 119 annotations with impact ratings:

Every unsupported annotation gets an impact rating: NONE (safe to ignore), LOW (better defaults), MEDIUM (needs workaround), or VARIES (review your snippets). Most teams discover 70%+ of "unsupported" annotations are safe to ignore.

End-to-End Demo: vCluster + ing-switch

Let's walk through a complete migration on a real cluster. We'll use vCluster to spin up a Kubernetes cluster in Docker, deploy 3 services with NGINX annotations, and migrate them to Gateway API with Traefik.

Step 1: Create a Cluster

vcluster create demo --driver docker

Output:

info  Using vCluster driver 'docker' to create your virtual clusters
info  Ensuring environment for vCluster demo...
done  Created network vcluster.demo
info  Starting vCluster standalone demo
done  Successfully created virtual cluster demo
info  Waiting for vCluster to become ready...
done  vCluster is ready
done  Switched active kube context to vcluster-docker_demo

Verify:

kubectl get namespaces

NAME                 STATUS   AGE
default              Active   16s
kube-flannel         Active   6s
kube-node-lease      Active   16s
kube-public          Active   16s
kube-system          Active   16s
local-path-storage   Active   6s

Step 2: Install Ingress NGINX

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --create-namespace \
  --set controller.service.type=ClusterIP \
  --set controller.admissionWebhooks.enabled=false \
  --wait --timeout 120s

NAME: ingress-nginx
LAST DEPLOYED: Sun Mar 29 11:15:57 2026
NAMESPACE: ingress-nginx
STATUS: deployed

kubectl get pods -n ingress-nginx

NAME                                        READY   STATUS    RESTARTS   AGE
ingress-nginx-controller-5486dbd97f-vc9wv   1/1     Running   0          54s

Step 3: Deploy 3 Apps with NGINX Annotations

We deploy three services, each with different annotation patterns:

App 1 -- Basic web app (SSL redirect + timeouts):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-app
  namespace: demo
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "60"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "10"
spec:
  ingressClassName: nginx
  rules:
  - host: web.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web-app
            port:
              number: 80

App 2 -- API with CORS + rate limiting (10 annotations):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-cors
  namespace: demo
  annotations:
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-origin: "https://app.example.com,https://admin.example.com"
    nginx.ingress.kubernetes.io/cors-allow-methods: "GET, POST, PUT, DELETE, OPTIONS"
    nginx.ingress.kubernetes.io/cors-allow-headers: "Content-Type, Authorization, X-API-Key"
    nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
    nginx.ingress.kubernetes.io/cors-max-age: "86400"
    nginx.ingress.kubernetes.io/limit-rps: "50"
    nginx.ingress.kubernetes.io/limit-burst-multiplier: "3"
    nginx.ingress.kubernetes.io/proxy-body-size: "5m"
spec:
  ingressClassName: nginx
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /v1
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 80

App 3 -- Auth-protected dashboard (external auth + IP allowlist + session affinity):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dashboard
  namespace: demo
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/auth-url: "https://auth.example.com/verify"
    nginx.ingress.kubernetes.io/auth-response-headers: "X-User-ID,X-User-Email"
    nginx.ingress.kubernetes.io/whitelist-source-range: "10.0.0.0/8,172.16.0.0/12"
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "dashboard-session"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "3600"
spec:
  ingressClassName: nginx
  rules:
  - host: dashboard.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: dashboard
            port:
              number: 80

After applying all three:

kubectl get ingress -n demo

NAME        CLASS   HOSTS                   ADDRESS   PORTS   AGE
api-cors    nginx   api.example.com                   80      5s
dashboard   nginx   dashboard.example.com             80      5s
web-app     nginx   web.example.com                   80      5s

kubectl get pods -n demo

NAME                           READY   STATUS    RESTARTS   AGE
api-service-5f99b6d99d-x7vmn   1/1     Running   0          24s
dashboard-9ddbf867-7dbgf       1/1     Running   0          24s
web-app-969c76b7c-7wqw5        1/1     Running   0          24s

3 ingresses, 20 NGINX annotations, 3 services running. Now let's see what ing-switch makes of this.

Step 4: Scan the Cluster

ing-switch scan

  ing-switch -- Cluster Scan Results
  Cluster: vcluster-docker_demo

  Ingress Controller Detected
  Type:      ingress-nginx
  Version:   unknown
  Namespace: ingress-nginx

  Found 3 Ingress resource(s)

  NAMESPACE   NAME        HOSTS                   ANNOTATIONS   TLS   COMPLEXITY
  ---------   ----        -----                   -----------   ---   ----------
  demo        api-cors    api.example.com         10            no    unsupported
  demo        dashboard   dashboard.example.com   7             no    complex
  demo        web-app     web.example.com         3             no    complex

ing-switch detected the NGINX controller and found all 3 ingresses with their annotation counts and complexity scores.

Step 5: Analyze Compatibility

Let's compare all three targets:

Traefik v3:

ing-switch analyze --target traefik

  Summary
  -------
  Total ingresses:      3
  Fully compatible:     1
  Needs workarounds:    2
  Has unsupported:      0

Gateway API (Envoy):

ing-switch analyze --target gateway-api

  Summary
  -------
  Total ingresses:      3
  Fully compatible:     0
  Needs workarounds:    3
  Has unsupported:      0

Gateway API (Traefik):

ing-switch analyze --target gateway-api-traefik

  Summary
  -------
  Total ingresses:      3
  Fully compatible:     0
  Needs workarounds:    3
  Has unsupported:      0

Key insight: Traefik is the highest-compatibility target for this workload (1 fully compatible out of 3). The CORS annotations map directly to Traefik's Headers middleware. For Gateway API, CORS is now also fully supported thanks to the native CORS filter in Gateway API v1.5.

Here's the detailed annotation mapping for the API with CORS:

  demo/api-cors
  -------------
  ANNOTATION               STATUS        TARGET RESOURCE                    NOTES
  enable-cors              [supported]   HTTPRoute (CORS filter)            Native CORS filter (GA in Gateway API v1.5)
  cors-allow-origin        [supported]   HTTPRoute (CORS filter)            allowOrigins in CORS filter
  cors-allow-methods       [supported]   HTTPRoute (CORS filter)            allowMethods in CORS filter
  cors-allow-headers       [supported]   HTTPRoute (CORS filter)            allowHeaders in CORS filter
  cors-allow-credentials   [supported]   HTTPRoute (CORS filter)            allowCredentials in CORS filter
  cors-max-age             [supported]   HTTPRoute (CORS filter)            maxAge in CORS filter
  force-ssl-redirect       [supported]   HTTPRoute (RequestRedirect filter) 301 redirect to HTTPS
  limit-rps                [partial]     BackendTrafficPolicy (RateLimit)   Envoy Gateway BackendTrafficPolicy
  limit-burst-multiplier   [partial]     BackendTrafficPolicy (RateLimit)   Burst configurable but uses tokens
  proxy-body-size          [partial]     BackendTrafficPolicy               requestBuffer.limit

7 out of 10 annotations are fully supported. The 3 "partial" ones work -- they just use a slightly different API.

Step 6: Generate Migration Files

ing-switch migrate --target gateway-api-traefik --output-dir ./migration

  ing-switch -- Generating Migration Files
  Target:     gateway-api-traefik
  Output dir: ./migration

  + 00-migration-report.md
  + 01-install-gateway-api-crds/install.sh
  + 02-install-traefik-gateway/helm-install.sh
  + 02-install-traefik-gateway/values.yaml
  + 03-gateway/gatewayclass.yaml
  + 03-gateway/gateway.yaml
  + 04-httproutes/demo-api-cors.yaml
  + 04-httproutes/demo-dashboard.yaml
  + 04-httproutes/demo-web-app.yaml
  + 05-policies/demo-api-cors-ratelimit.yaml
  + 05-policies/demo-dashboard-forwardauth.yaml
  + 05-policies/demo-dashboard-ipallowlist.yaml
  + 06-verify.sh
  + 07-cleanup/remove-nginx.sh
  Generated 13 files in ./migration/

Step 7: Inspect the Generated YAML

GatewayClass -- points to Traefik, not Envoy:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: traefik
spec:
  controllerName: traefik.io/gateway-controller

HTTPRoute with native CORS filter (no more ResponseHeaderModifier hacks):

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-cors
  namespace: demo
spec:
  parentRefs:
  - name: ing-switch-gateway
    namespace: default
  hostnames:
  - "api.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: "/v1"
    filters:
    - type: CORS
      cors:
        allowOrigins:
        - type: Exact
          value: "https://app.example.com"
        - type: Exact
          value: "https://admin.example.com"
        allowMethods:
        - "GET"
        - "POST"
        - "PUT"
        - "DELETE"
        - "OPTIONS"
        allowHeaders:
        - "Content-Type"
        - "Authorization"
        - "X-API-Key"
        allowCredentials: true
        maxAge: "86400s"
    backendRefs:
    - name: api-service
      port: 80

Traefik Middleware CRDs (not Envoy-specific policies):

# Rate Limiting
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: demo-api-cors-ratelimit
  namespace: demo
spec:
  rateLimit:
    average: 50
    burst: 3

# ForwardAuth (external authentication)
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: demo-dashboard-forwardauth
  namespace: demo
spec:
  forwardAuth:
    address: "https://auth.example.com/verify"
  authResponseHeaders:
    - "X-User-ID"
    - "X-User-Email"

# IP AllowList
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: demo-dashboard-ipallowlist
  namespace: demo
spec:
  ipAllowList:
    sourceRange:
    - "10.0.0.0/8"
    - "172.16.0.0/12"

Step 8: Review the Migration Report

The migrate command automatically generates 00-migration-report.md in the output directory. Open it to see the full summary:

cat ./migration/00-migration-report.md

# ing-switch Migration Report
**Target Controller:** gateway-api-traefik

## Summary
| Metric | Count |
|--------|-------|
| Total Ingresses | 3 |
| Fully Compatible | 0 |
| Needs Workarounds | 3 |
| Has Unsupported Annotations | 0 |

## demo/api-cors -- Needs workaround
| Annotation | Status | Target Resource | Notes |
|-----------|--------|-----------------|-------|
| enable-cors | OK | HTTPRoute (CORS filter) | Native CORS filter (GA in v1.5) |
| cors-allow-origin | OK | HTTPRoute (CORS filter) | allowOrigins in CORS filter |
| limit-rps | WARN | BackendTrafficPolicy | Envoy Gateway BackendTrafficPolicy |
...

Step 9: Apply (Dry-Run First)

# Install Gateway API CRDs
bash ./migration/01-install-gateway-api-crds/install.sh

# Install Traefik with Gateway API provider
bash ./migration/02-install-traefik-gateway/helm-install.sh

# Dry-run all resources first
kubectl apply -f ./migration/03-gateway/ --dry-run=server
kubectl apply -f ./migration/04-httproutes/ --dry-run=server

# If dry-run passes, apply for real
kubectl apply -f ./migration/03-gateway/
kubectl apply -f ./migration/04-httproutes/
kubectl apply -f ./migration/05-policies/

At this point, both NGINX and Traefik are running side by side. DNS still points to NGINX. Production traffic is untouched.

Step 10: Verify and Cutover

# Run the generated verification script
bash ./migration/06-verify.sh

# Once verified, update DNS to Traefik's IP
# Then clean up NGINX
bash ./migration/07-cleanup/remove-nginx.sh

Step 11: Use the Web UI

For teams that prefer a visual workflow:

ing-switch ui
# Opens http://localhost:8080

The dashboard provides four pages:

Detect -- Scan your cluster and see all ingresses with annotation counts and complexity:

Analyze -- Choose between 3 targets and see the full annotation compatibility matrix:

Migrate -- One-click generation with step-by-step checklist and dry-run buttons:

View all generated files inline with syntax highlighting:

See migration gaps with impact ratings and fix instructions:

Validate -- Run live cluster checks to confirm your migration phase:

Cleanup

vcluster delete demo --driver docker

done  Successfully deleted virtual cluster demo

What Makes ing-switch Different

The Ecosystem Is Ready

• Gateway API v1.5 -- CORS filter, TLSRoute, BackendTLSPolicy all GA

• ingress2gateway v1.0 -- Official tool with emitter architecture

• Traefik v3.7 -- Native NGINX annotation provider (80+ annotations)

• Envoy Gateway v1.7 -- XListenerSet, enhanced policies

• cert-manager v1.20 -- Gateway API ListenerSet support

• Kubernetes 1.36 -- Ships April 22, first release post-NGINX archival

The tools exist. The standards are stable. The only thing left is to actually run the migration.

Star it, fork it, migrate today: github.com/saiyam1814/ing-switch

ing-switch is open source under the MIT license. PRs welcome.

OpenClaw has 314,000+ GitHub stars. It is the most popular open-source AI agent out there. It connects to WhatsApp and Telegram, does deep research, manages files, writes code, handles voice notes, and genuinely works as a personal assistant. The catch is that setting it up with a local LLM on NVIDIA hardware is a long process with security in place.

I spent some time getting it right on a DGX Spark. Then I automated the entire thing into one command. Along the way I found nine bugs, wrote three source patches, fought Ubuntu's managed Python, debugged WhatsApp's device linking protocol, and integrated a hardware-aware model selection engine. This is the full story.

The Problem

NVIDIA's DGX Spark is a desktop AI supercomputer. GB10 Grace Blackwell chip, 128GB unified memory, 1 PFLOP of AI compute, 20-core ARM Cortex-A725 CPU. It sits on your desk, runs quietly, and has enough memory to load models that actually compete with cloud APIs. The hardware is not the bottleneck anymore.

The bottleneck is setup. If you want to run OpenClaw with a local model on DGX Spark, here is what you need to do manually:

1. Install Node.js 22+

2. Install OpenClaw via npm

3. Install Ollama

4. Figure out which model fits your hardware (there are hundreds)

5. Pull the model (can be 20-80GB)

6. Configure OpenClaw to point at your local Ollama instance

7. Set the correct environment variables (OLLAMA_API_KEY, OLLAMA_BASE_URL)

8. Run onboard, which sets half your config to wrong defaults

9. Fix tools.profile from "messaging" to "full"

10. Start the gateway, then start a separate Node Host process

11. Pair the Node Host to the gateway

12. Link WhatsApp via QR code

13. Patch three different JavaScript files in OpenClaw's dist folder

14. Install skills

15. Harden security

16. Set up voice transcription

Miss one step and things fail silently. The gateway starts, the model loads, but your agent can only send text messages because it has 5 tools instead of 15. Or WhatsApp linking fails because the browser identification string gets rejected. Or group messages never arrive because history sync is disabled.

NVIDIA's own docs at build.nvidia.com recommend gpt-oss-120b for DGX Spark and describe a manual multi-step process using Ollama or LM Studio. Their guide covers the inference setup but not WhatsApp integration, not voice transcription, not security hardening, and not the Node Host that the agent actually needs to do useful work. clawspark automates all of this, including the parts NVIDIA's guide does not cover.

What clawspark Does

One command:

curl -fsSL https://clawspark.dev/install.sh | bash

That is it. The installer runs 14 steps. It detects your hardware, recommends a model, asks a few questions, then installs and configures everything. Here is the full flow:

Step 1-2: Hardware detection. The script probes your GPU via nvidia-smi, reads DMI product name for DGX Spark identification, checks for Tegra signatures on Jetson, and measures total system memory. It classifies your hardware into one of four tiers: DGX Spark (128GB unified), Jetson AGX (64GB), RTX high-end (24GB+ VRAM), or RTX standard (8-24GB).

Step 3: Model selection. This is where it gets interesting. For DGX Spark, I curated a list of 5 models ranked by llmfit score and verified on real hardware:

For non-DGX-Spark hardware (RTX, Jetson, anything else), the installer uses llmfit to analyze your specific hardware, score hundreds of models, map the results to Ollama model IDs, verify each candidate actually exists on the Ollama library, and present the top 5 that fit. No hardcoded lists. Your GPU, your recommendations.

Step 4-5: Deployment and messaging. Choose local-only or hybrid (cloud fallback). Choose WhatsApp, Telegram, both, or skip messaging entirely. The web UI at /__openclaw__/canvas/ always works regardless.

Step 6-14: The actual installation. Ollama install and model pull. Node.js 22 if needed. OpenClaw npm install. Config generation with correct Ollama endpoints. Onboard with overrides for all the wrong defaults. Three source patches (more on these below). Skills installation. Whisper voice setup. WhatsApp QR linking. Optional Tailscale for remote access. ClawMetry dashboard. Security hardening. Final verification.

After installation, you get the clawspark CLI tool for day-to-day management: clawspark status, clawspark benchmark, clawspark restart, clawspark skills sync, clawspark airgap on/off, and more.

The Architecture

Here is how the pieces fit together once everything is running:

WhatsApp / Telegram / Web UI (Canvas)
              |
    OpenClaw Gateway (port 18789)
     |            |            |
   Agent      Node Host     Baileys
   (LLM)      (Tools)      (WhatsApp Web)
     |            |
   Ollama      15 tools:
  (port 11434)  exec, read, write, edit,
     |          web_fetch, message, canvas,
   Model        process, cron, nodes,
   (GPU)        sessions_spawn, vision,
                transcribe, memory_search,
                memory_store

The Gateway is the central process. It manages the agent, routes messages from WhatsApp/Telegram/Web, and coordinates tool calls. The Agent is the LLM reasoning loop that decides what to do. The Node Host is a separate process that provides the actual tool implementations -- reading files, fetching web pages, executing code. Without the Node Host, the agent has only 5 basic messaging tools instead of the full 15.

Baileys is the WhatsApp Web client library that OpenClaw uses under the hood. It connects to WhatsApp's servers using a linked device session, the same way WhatsApp Web works in your browser. Messages flow from WhatsApp through Baileys to the Gateway, which sends them to the Agent, which calls tools on the Node Host, and the response flows back the same way.

The Bugs I Found and Fixed

This section is the reason I wrote this post. These are all real issues I hit on real hardware, and some of them took hours to diagnose. If you are setting up OpenClaw manually, this list might save you a lot of time.

1. tools.profile does not default to "full"

When you run openclaw onboard, it does not set tools.profile to "full". In v2026.3.2 it defaulted to "messaging" (5 tools only). This was partially fixed in v2026.3.7, which changed the default to "coding" -- better, but still missing tools like exec, process, cron, and nodes. The agent looks like it is working, but it cannot do everything it should.

The fix: openclaw config set tools.profile full after onboard completes. clawspark does this automatically.

2. Node Host is required but not documented

The Gateway alone does not provide execution tools. You need a separate "Node Host" process (openclaw node run) that connects to the Gateway and provides filesystem, browser, and execution capabilities. Without it, even with tools.profile full, the agent has no tools to call. The Node Host also needs to be paired with the Gateway (device approval), which is another step that is easy to miss.

clawspark starts the Node Host, detects pending pairing requests, auto-approves them, and restarts the Node Host with the pairing token.

3. Baileys browser string rejected by WhatsApp

OpenClaw's WhatsApp integration uses Baileys, which identifies itself as ["openclaw", "cli", VERSION] to WhatsApp's servers. WhatsApp rejects this during device linking. The QR code scan works, but the connection fails silently.

The fix is a source patch: replace the browser identification with ["Ubuntu", "Chrome", "22.0"], which WhatsApp accepts. This requires patching the compiled JavaScript in OpenClaw's dist folder. clawspark finds the relevant session files and applies the patch automatically.

4. web_search requires a Brave API key

OpenClaw's built-in web_search tool requires a Brave Search API key. For a local setup, requiring an external API key defeats the purpose. clawspark works around this by configuring the agent's TOOLS.md to use DuckDuckGo Lite via web_fetch instead:

web_fetch with url="https://lite.duckduckgo.com/lite/?q=YOUR+QUERY"

This gives the agent web search capabilities without any API keys or external dependencies.

5. Agent narrates tool usage on WhatsApp

When you ask the agent a question on WhatsApp, it sends messages like "Let me search for that..." and "The search returned these results..." before giving you the actual answer. On WhatsApp, this means three or four notification buzzes for one question.

The fix is SOUL.md rules: explicit instructions to never narrate tool usage, use tools silently, and respond with one clean message.

6. syncFullHistory breaks group messages

OpenClaw defaults to syncFullHistory: false in its Baileys configuration. This means after a fresh WhatsApp link, Baileys never receives group sender keys. The result: groups are completely silent. No messages arrive, no errors are logged. It just looks like nobody is talking.

The fix: patch syncFullHistory: false to syncFullHistory: true in the compiled session files. clawspark finds and patches all relevant files automatically.

7. Mention detection has an early return that blocks text @mentions

OpenClaw's mention detection in group chats has a return false early exit when JID (WhatsApp ID) mentions exist but do not match the bot's JID. The problem is that WhatsApp resolves @mentions to JIDs, and sometimes the resolved JID does not match the bot's linked phone JID. The early return prevents the text-pattern fallback from ever running, so typing @botname in a group never triggers the bot.

The fix: remove the return false line so the text-pattern fallback always has a chance to match. Another source patch to the compiled JavaScript.

8. Systemd service missing Ollama environment variables

When OpenClaw's gateway runs as a systemd service, it does not inherit the shell environment. The OLLAMA_API_KEY and OLLAMA_BASE_URL variables are missing, so the gateway cannot reach Ollama. The model appears to load, but every inference call fails.

clawspark writes the environment variables to a gateway.env file, adds them to the user's shell profile (.bashrc or .zshrc), and sources them before starting any OpenClaw process.

9. OpenClaw bindings schema changed between versions

This one cost me an entire evening. Earlier versions of OpenClaw supported a bindings config for routing different message sources to different agents (e.g., full tools in DMs, restricted tools in groups). Starting with v2026.3.2, the bindings schema changed and the old format causes a validation error at startup: Invalid config: bindings.0: Invalid input. This is not a bug per se -- the schema evolved -- but any guide or config from earlier versions will break silently.

The fix: remove bindings entirely and use a single agent with context-aware rules. SOUL.md and TOOLS.md contain explicit sections for DM context (full tools) and group context (Q&A only). The agent enforces the boundary at the prompt level. Groups also use requireMention: true and groupPolicy: open at the config level so the bot only responds when @mentioned.

Security

Running a local AI agent is not automatically secure. clawspark applies multiple layers:

Gateway binding. The OpenClaw gateway binds to localhost only. It is not accessible from other machines on your network unless you explicitly set up Tailscale.

Firewall rules. UFW is configured to deny all incoming connections except SSH. Outgoing traffic is allowed by default, or blocked entirely in air-gap mode.

Token authentication. A random 256-bit token is generated during installation. Only clients with this token can talk to the gateway API.

Context-aware tool restrictions. In direct messages, the owner gets full access to all 15 tools. In group chats, the agent restricts itself to Q&A only (message, web_fetch, memory). This is enforced at the prompt level via SOUL.md, which contains explicit rules for each context. Groups also require @mention to activate.

SOUL.md and TOOLS.md. These workspace files contain the agent's identity, capabilities, and absolute rules. No credential disclosure (applies to all users, including the owner). No system information in groups. No self-modification. Both files are set to chmod 444 (read-only) so the agent cannot edit its own rules.

Air-gap mode. For maximum isolation, clawspark airgap on blocks all outbound internet traffic via UFW. Only local network and loopback traffic is allowed. The model, the agent, and all tools run entirely offline.

One honest caveat: local models do not have the same safety filters that cloud providers build into their APIs. That is both a feature (no arbitrary refusals) and a responsibility. You should think carefully about who has access to message your bot.

Real Performance Numbers

All numbers below are from an actual DGX Spark running Linux 6.14.0-1015-nvidia (arm64), Ollama 0.17.7, Node.js v22.22.1, OpenClaw v2026.3.13, with Qwen 3.5 35B-A3B:

The 59 tok/s generation speed is fast enough to feel responsive. You send a question on WhatsApp and the response arrives in 15-45 seconds depending on complexity. The cold load penalty only hits on the first query after a restart. After that, the model stays in memory.

To put this in perspective: 59 tok/s means the model generates roughly 45-50 words per second. A typical response of 200 words takes about 4 seconds of pure generation time. The rest of the 15-45 second latency comes from the WhatsApp message routing, tool calls (if the agent needs to search the web or read a file), and response formatting.

Is this as fast as GPT-4o or Claude via cloud API? No. Cloud inference on dedicated hardware with massive batching will always be faster for raw token throughput. But it is fast enough for practical use, and your data never leaves your desk. That is the tradeoff.

For the 122B-A10B MoE model (the highest-ranked by llmfit), expect roughly 45 tok/s. Slightly slower but you get access to the full 122B model's knowledge with only 10B active parameters. The DGX Spark's 128GB unified memory can comfortably hold this model (33GB) with plenty of room for the KV cache.

Hardware-Aware Model Selection with llmfit

One of the hardest problems with local AI is knowing which model to use. There are hundreds of models on Ollama, each with different sizes, quantizations, and performance characteristics. Picking the wrong one means either wasting memory (model too small for the hardware) or crashing on load (model too big).

I integrated llmfit to solve this. llmfit is a Rust-based CLI tool that detects your hardware (GPU, VRAM, RAM, CPU), scores every model in its database for fit, speed, and quality, and tells you which ones will actually run well.

For DGX Spark, I ran llmfit on the real hardware and it correctly detected the NVIDIA GB10 with 119.7 GB unified memory and CUDA backend. I then cross-referenced its top recommendations against the Ollama library to verify each model is actually pullable. The result is the curated list of 5 models you see during installation.

For all other hardware, the installer runs llmfit live:

1. Install llmfit (one curl command, Rust binary)

2. Run llmfit recommend --json -n 20 --min-fit good

3. Map each HF-style model name to an Ollama model ID (40+ regex patterns)

4. Verify each candidate exists on Ollama's library (HTTP check against ollama.com)

5. Present the top 5 verified models with score, estimated tok/s, and fit level

If llmfit is not available or fails, the installer falls back to curated lists per hardware tier. No hardcoded guessing. Your GPU, your recommendations.

WhatsApp Integration: The Deep Cut

Getting WhatsApp working reliably was the hardest part of this entire project. Here is why.

OpenClaw uses the Baileys library, which is an unofficial WhatsApp Web client. It works by emulating a linked device session, the same protocol that WhatsApp Web uses in your browser. The connection is end-to-end encrypted and goes through WhatsApp's servers. There is no official API involved.

This creates three categories of problems:

Protocol issues. WhatsApp regularly changes its protocol, and Baileys has to keep up. The browser string rejection (bug #3) is an example. WhatsApp started rejecting non-standard browser identifiers at some point, and OpenClaw's default string got caught.

Group message handling. WhatsApp groups use Signal's sender keys protocol. When you first link a device, it needs to receive sender keys from all group participants before it can decrypt group messages. Setting syncFullHistory: false (bug #6) prevents this initial key exchange, making groups completely silent.

Mention routing. In WhatsApp groups, @mentions get resolved to JIDs (WhatsApp internal user IDs). The bot's JID might not match its linked phone number's JID in all cases. The early return in mention detection (bug #7) means the bot never sees @mentions in groups unless you patch the code.

clawspark applies all three patches automatically and re-applies them after updates (since npm update overwrites the dist files). Groups require @mention to activate, and the agent is restricted to Q&A only in group context. Full tool access is reserved for direct messages with the owner.

Voice notes work through the local-whisper skill, which runs Whisper (OpenAI's open-source speech-to-text model) locally on the GPU. On DGX Spark, it uses the large-v3 model for maximum transcription accuracy. On Jetson, it drops to the small model. On RTX, it scales based on available VRAM. The audio never leaves your machine.

Skills

clawspark installs 10 skills by default, verified against the OpenClaw skill registry:

Web search works without any API keys. The agent uses DuckDuckGo Lite via web_fetch, fetches result URLs, and composes answers from the content. No Brave API key, no Google API key, no external dependencies.

You can add or remove skills with clawspark skills add <name> and clawspark skills remove <name>. The skills.yaml file in your config directory is the source of truth, and clawspark skills sync reads it and installs everything.

Getting Started

Tested and verified on DGX Spark. Should also work on Mac (Apple Silicon M1/M2/M3/M4), RTX desktops, and Jetson. The installer has fallbacks for all platforms -- macOS uses Homebrew, Ollama runs natively on Apple Silicon, and llmfit handles model selection. These platforms have not been end-to-end tested yet. Community testing welcome -- open an issue if you try it on different hardware.

curl -fsSL https://clawspark.dev/install.sh | bash

Or with specific options:

bash install.sh --model=qwen3.5:122b-a10b --messaging=whatsapp

After installation:

clawspark status      # Check all components
clawspark benchmark   # Run a performance benchmark
clawspark logs        # Tail the gateway logs
clawspark restart     # Restart all services
clawspark update      # Update OpenClaw and re-apply patches
clawspark airgap on   # Enable air-gap mode

The web UI is at http://localhost:18789/__openclaw__/canvas/. The metrics dashboard is at http://localhost:8900.

Source code and documentation: clawspark.dev

What is Next

A few things I am working on:

Multi-model routing. Use the fast 35B model for simple queries and automatically route complex reasoning tasks to the 122B model. The hardware can handle both loaded simultaneously since 128GB is enough for both.

Better metrics. ClawMetry currently shows basic gateway stats. I want per-query latency tracking, token usage by model, and cost-equivalent comparisons (how much this query would have cost on cloud APIs).

More hardware testing. The Jetson and RTX paths in clawspark are written and should work (hardware detection, llmfit model selection, Ollama setup), but I have only done full end-to-end testing on DGX Spark. Jetson AGX Orin is next. For RTX desktops, open a terminal and run the install command directly -- no SSH needed. If you try it on your hardware, please open an issue with your results.

Upstream patches. The three source patches I wrote are necessary because of bugs in OpenClaw's compiled code. Ideally these get fixed upstream so the patches become unnecessary. I plan to submit them.

Closing Thoughts

Two years ago, running a capable AI model locally meant a server rack, a cooling system, and deep knowledge of CUDA. Today it means a quiet box on your desk and a bash script.

The gap between local and cloud AI is closing fast. Not because local is getting as fast as cloud (dedicated data center hardware will always win on raw throughput), but because local is getting good enough. 59 tokens per second from a 35B-parameter MoE model on a desktop machine is good enough for a personal assistant. The tradeoff is straightforward: you get complete data privacy and zero ongoing cost in exchange for slightly higher latency.

clawspark is just the glue. It takes hardware that is already capable (DGX Spark), software that is already good (OpenClaw, Ollama), a model that is already smart (Qwen 3.5), and a tool selector that actually knows your hardware (llmfit), and removes the friction between them. The one-click part is not the innovation. The innovation is all the edge cases, patches, and defaults that make it actually work when you run it for the first time on real hardware.

If you have a DGX Spark, an RTX GPU, a Mac with Apple Silicon, or a Jetson, give it a try. One command, a few questions, and you have a private AI assistant that genuinely never phones home.

GitHub: github.com/saiyam1814/claw-spark Website: clawspark.dev

That’s what running NVIDIA’s Nemotron 3 Super on the DGX Spark feels like. After spending time with it, I think this model needs more attention than it’s getting. Not because of one benchmark number, but because of the engineering choices behind it. These choices show you exactly where AI inference is going.

Let me walk you through what I found.

The Headline Numbers (And Why They’re Misleading)

When NVIDIA drops a model, they lead with the big stats: 120 billion parameters, 1 million token context, 5x throughput. These numbers are real, but they hide the real story.

The number that actually matters is 12.7 billion. That’s how many parameters fire per token. Out of 120.6 billion total, only about a tenth light up for any given input. The rest sit there, waiting until the right token needs their skill.

This roughly 10:1 ratio is the whole story. It’s why the model runs on desktop hardware. It’s why it’s fast. It’s why NVIDIA built it this way. Everything else follows from this one design choice.

The second thing that matters is the layer mix. The model has 88 layers total. Most of them are Mamba-2 layers. This is a completely different architecture from transformers and it doesn’t need to store growing key-value caches. Only a small number are traditional transformer attention layers. NVIDIA interleaves them in a repeating pattern: groups of Mamba-2 blocks paired with Latent MoE layers, with attention layers placed at key depths. We’ll come back to why this split is so important.

Three Architectures in a Trenchcoat

Nemotron 3 Super isn’t one architecture. It’s three, stacked together so each one does what it’s best at. Once you get this stack, you get why the model works the way it does.

Mamba-2: The Workhorse

The majority of the 88 layers are Mamba-2 blocks. Mamba is a state-space model. Think of it like a recurrent architecture that keeps a compact, fixed-size state and updates it as each new token comes in.

The key thing: Mamba runs in linear time. Double the sequence length, you roughly double the compute. Compare that with transformer attention, where doubling the sequence quadruples the compute.

This is why Nemotron 3 Super can actually deliver a 1-million-token context window in practice. With most layers being Mamba, the bulk of the model doesn’t care how long your prompt is. Compute grows linearly, and memory doesn’t grow at all. Mamba’s state stays the same size no matter the sequence length.

Transformer Attention: The Precision Tool

A small number of layers in the stack are traditional transformer attention layers, using Grouped Query Attention with 32 query heads, 2 KV heads, and a head dimension of 128. These are confirmed specs from the technical report.

Why keep any attention at all? Because Mamba has a known gap. It struggles with precise associative recall, like connecting a specific detail from position 1,000 with something at position 500,000. The fixed-size state means some info gets compressed away over very long sequences.

NVIDIA’s fix: place attention layers at carefully chosen depths through the 88-layer stack. They act like precision tools, handling the long-range connections that Mamba would miss, while Mamba does everything else fast.

The result: the vast majority of the model’s compute happens in linear time. Quadratic attention is used only where it’s needed.

The KV Cache Payoff

This design has a huge side effect that matters a lot for hardware like the DGX Spark.

In a transformer, the KV cache grows with sequence length. Every attention layer stores key and value tensors for every token it has seen. For a model like Qwen 3.5-122B with 12 full attention layers, head dimension 256, and 2 KV heads, that adds up to about 22.9 GiB at 1 million tokens in BF16. (The math: 12 layers x 2 KV heads x 256 dim x 2 bytes x 2 for K+V x 1M tokens.)

Nemotron 3 Super has far fewer attention layers, each with head dimension 128 and 2 KV heads. Because Mamba layers use a fixed-size state (no KV cache growth), only the attention layers add to the KV cache. The bottom line: the KV cache is roughly 3x smaller than Qwen’s at the same context length. On the DGX Spark’s 128 GB of unified memory, you load the 87 GB model, add a relatively small KV cache even at very long contexts, and you still have plenty of room to spare.

For practical purposes, the KV cache almost doesn’t matter with this model.

Latent MoE: Getting 4x More Experts for Free

The Mixture of Experts layer is where things get really clever.

In a standard MoE, each token is routed to one or two “expert” sub-networks from a larger pool. The idea is simple: different experts specialize in different things, and the router learns which expert to call for each token.

The problem is cost. Routing happens at the model’s full hidden dimension, and each expert operates at that same dimension. If you want more experts (for better specialization), routing gets more expensive. If you want to activate more experts per token (for better accuracy), inference gets slower.

Latent MoE solves this with a compression trick. Before routing, token embeddings are projected from the full hidden dimension down to a smaller latent dimension. The router operates in this compressed space, which is much cheaper. Experts also operate on the compressed representations.

The compute you save on compression doesn’t disappear. It gets reinvested. NVIDIA uses it to increase both the total number of experts and the number of experts active per token by the same factor. The result: 4x more experts consulted per token, at approximately the same inference cost as a standard MoE with fewer experts.

Each token effectively gets a committee of 4 specialists deliberating instead of a single expert making a snap judgment. The accuracy improvement is significant, and you don’t pay for it in latency.

Multi-Token Prediction: Built-In Speculative Decoding

Standard language models predict one token at a time. Generate position N, feed it back in, generate position N+1, repeat. This sequential nature is the fundamental bottleneck in text generation speed.

Nemotron 3 Super predicts multiple future tokens from each position simultaneously. The model has shared-weight prediction heads that project from the same internal representation to predict not just the next token, but several tokens ahead.

This serves two purposes. During training, it forces the model to learn longer-range dependencies. You can’t predict three tokens ahead without understanding the broader context. This makes the model smarter.

During inference, it works like built-in speculative decoding. Instead of generating one token per forward pass, the model proposes multiple tokens, verifies them, and keeps the correct ones. For structured output like code and tool calls, where the next few tokens are often very predictable, NVIDIA reports up to 3x wall-clock speedup. General chat won’t see the full 3x, but code generation benefits a lot.

The nice thing: you don’t need a separate draft model. The speculation is built right into the model.

How It Was Trained: The Three-Phase Pipeline

This is where NVIDIA’s openness really shines. They didn’t just release weights. They published the complete methodology, and the numbers are big.

Phase 1: Pretraining. 25 trillion tokens total (10 trillion unique), plus 10 billion additional tokens focused specifically on reasoning, plus 15 million coding problems. The majority of compute ran in NVFP4, which is NVIDIA’s native 4-bit floating point format. This is unusual and important: most models train in higher precision and quantize down later, losing accuracy. Nemotron 3 Super was born in FP4.

Phase 2: Supervised Fine-Tuning. 7 million carefully selected samples from a corpus of 40 million. Coverage spans reasoning, instruction following, coding, safety, and critically, multi-step agent task completion. This phase is where the model learns to be useful, not just knowledgeable.

Phase 3: Reinforcement Learning. 1.2 million environment rollouts across 21 different configurations, using NeMo Gym and NeMo RL frameworks with 37 datasets. This is where the model learns to reason through complex, multi-step problems. This is the kind of thinking that makes it useful as an autonomous agent.

NVIDIA released around 10 of the pretraining datasets publicly, 15 RL training environments, and about 10 of the 37 RL datasets, along with complete training recipes. The Artificial Analysis Openness Index scored this release at 83 out of 100. Only two research labs (Ai2 and MBZUAI) score higher, and their models aren’t anywhere near this performance level.

This kind of transparency is new for a model this good. With enough compute, you could follow their recipe and reproduce the training run yourself.

Why NVIDIA Built This for Agents

The model wasn’t designed for chatbots. Every architectural decision points toward one use case: long-running autonomous agents. The 1M context, the sparse activation, the MoE efficiency. All of it.

When you run a multi-agent system, token consumption explodes. Each agent interaction requires sending the full conversation history, tool outputs, intermediate reasoning steps, and results from other agents. NVIDIA’s numbers suggest multi-agent workflows generate up to 15x more tokens than standard chat.

This creates two problems that kill most models:

Context overflow. At 128K tokens, even large models run out of context in extended agent sessions. The agent either loses early context (and the original goal with it), or you implement complex summarization/RAG schemes that lose fidelity. Nemotron 3 Super’s million-token window means the agent can hold the entire workflow state. Every tool call, every intermediate result, every reasoning step stays in memory without ever truncating.

The cost of thinking. Agents need to reason at every step. If each reasoning call costs as much as a full 120B forward pass, running thousands of agent subtasks gets very expensive very fast. With only 12B active parameters, each call through Nemotron 3 Super costs a fraction of what a dense 120B model would. You get the intelligence of a large model with the economics of a small one.

The benchmarks back this up. On PinchBench, which tests models as actual coding agents (not just answering coding questions), it scores 85.6%. That’s the best open model out there. On DeepResearch Bench, which tests multi-step research over large document sets, NVIDIA’s AI-Q multi-agent system took the number one position. AI-Q is built on top of a fine-tuned Nemotron 3 Super. Worth noting: AI-Q is a full multi-agent system with orchestrator, planner, and researcher sub-agents. It’s not just the base model running solo. But Nemotron 3 Super is the reasoning engine at its core.

Why Sparse MoE is Perfect for the DGX Spark

Here’s something that sounds wrong: on the DGX Spark, a 120B MoE model runs way faster than a smaller 70B dense model. The bigger model is faster. How?

Running It: Three Paths from Zero to Inference

I tested three ways to get Nemotron 3 Super running on the DGX Spark. Here they are, from simplest to most configurable.

Path 1: Ollama (Two Commands)

The fastest possible path. If Ollama is installed on your Spark (it comes pre-installed on DGX OS):

ollama pull nemotron-3-super

ollama run nemotron-3-super

saiyam@spark-5385:~$   ollama pull nemotron-3-super
pulling manifest 
pulling 0fc53cc990a2: 100% ▕███████████████████████████████████████████████████████████████████████████████████████▏  86 GB                         
pulling d02d998e5ae6: 100% ▕███████████████████████████████████████████████████████████████████████████████████████▏  23 KB                         
pulling 02897ca0d6a3: 100% ▕███████████████████████████████████████████████████████████████████████████████████████▏   31 B                         
pulling 9c35241878aa: 100% ▕███████████████████████████████████████████████████████████████████████████████████████▏  509 B                         
verifying sha256 digest 
writing manifest 
success 


saiyam@spark-5385:~$ ollama list

NAME                       ID              SIZE     MODIFIED               
nemotron-3-super:latest    95acc78b3ffd    86 GB    Less than a second ago    
qwen3.5:35b-a3b            3460ffeede54    23 GB    5 days ago                
saiyam@spark-5385:~$  ollama run nemotron-3-super --verbose "Explain the difference between Mamba and Transformer architectures like I'm a DevOps engineer who has never worked with ML."

ollama show nemotron-3-super 
  Model
    architecture        nemotron_h_moe    
    parameters          123.6B            
    context length      262144            
    embedding length    4096              
    quantization        Q4_K_M            
    requires            0.17.1            

  Capabilities
    completion    
    tools         
    thinking      

  Parameters
    temperature    1       
    top_p          0.95    

  License
    NVIDIA Software and Model Evaluation License                                            
    IMPORTANT NOTICE – PLEASE READ AND AGREE BEFORE USING THE NVIDIA LICENSED MATERIALS.    
    ...                                                                                     

Real performance on DGX Spark:

prompt eval rate:  3.51 tokens/s
eval rate: 19.50 tokens/s
eval count:  2504 tokens
total duration:  2m56s

Path 2: llama.cpp from Source (Full Control)

If you want more control over context sizes, quantization, and API serving, you can build llama.cpp from source and run the GGUF model directly. Unsloth has a detailed guide for this: Unsloth Nemotron 3 Super Guide

The key things to know for DGX Spark:

• When building llama.cpp, use -DCMAKE_CUDA_ARCHITECTURES=”121” for the GB10 chip. Without this you’ll fall back to CPU inference.

• The GGUF files are at unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF on Hugging Face.

• NVIDIA recommends temperature 1.0 for general chat, and 0.6 with top_p 0.95 for tool calling.

• Set --ctx-size based on your available memory. On DGX Spark, 16384 to 262144 is practical. Setting it to 1M may trigger CUDA OOM.

• llama-server gives you an OpenAI-compatible API, so VS Code Continue, LangChain, CrewAI, Open WebUI all just work.

The DGX Spark: Quick Hardware Context

For readers unfamiliar with the hardware, the DGX Spark is NVIDIA’s desktop AI computer. The relevant specs:

• Chip: GB10 Grace Blackwell Superchip

• Memory: 128 GB unified LPDDR5x (shared CPU/GPU, 273 GB/s)

• GPU: 6,144 CUDA cores, 5th-gen Tensor Cores, 1 PFLOP FP4 sparse

• CPU: 20-core ARM (10x Cortex-X925 + 10x Cortex-A725)

• Size: 150mm x 150mm x 50mm, 1.2 kg

• Power: 240W

The unified memory is the key. Unlike a discrete GPU where you’re limited by VRAM (24 GB on an RTX 4090, 32 GB on an RTX 5090), the DGX Spark’s 128 GB is coherently shared between CPU and GPU with no PCIe bottleneck. The full 87 GB model lives in one address space.

NVIDIA rates it for models up to 200 billion parameters on a single unit, or 405 billion on two connected Sparks.

Putting It in Context: How Nemotron 3 Super Compares

Here’s the honest comparison against peers, based on published third-party benchmarks:

What I’d Actually Use This For

After spending time with the model, here’s where I see genuine value in the Nemotron + DGX Spark combination:

Openclaw - I am going to put this as the model for openclaw too which is already running on my machine.

Private data analysis. If you work in healthcare, finance, legal, or defense, some data simply cannot leave your building. No cloud provider promise changes the rules. A local model that never touches a network is the only option for these workloads.

Code review and analysis. 73.4% precision on Qodo’s code review benchmark means about three out of four issues it flags are real. That’s useful enough for a local code review helper, especially when you’re working on code you can’t send to an external API.

Long-document reasoning. The million-token context with a tiny KV cache means you can load entire codebases, spec documents, or stacks of research papers and ask questions across everything. No chunking, no RAG pipeline needed. Just load it all and ask.

Where I wouldn’t use it: production serving at scale, real-time latency-critical applications, or model training. The DGX Spark is an inference machine, not a training rig.

The Bigger Picture

Nemotron 3 Super is interesting as a model, but it’s even more interesting as a strategy.

NVIDIA makes the chips (GB10, B200), the inference runtime (NIM, TensorRT-LLM), the training framework (NeMo), and now the models (Nemotron). They’ve released the model, the data, and the recipes. Everything except the hardware to train on.

That’s the play. The more developers build on Nemotron, the more they need NVIDIA hardware. The openness isn’t charity. It’s ecosystem building.

But for us as practitioners, the result is clearly good. We get a top-tier model with full training transparency, running on hardware we can put on a desk. The hybrid Mamba-Transformer architecture with Latent MoE and multi-token prediction isn’t just a research paper. It’s a practical solution for running large models on limited hardware.

NVIDIA has confirmed that Ultra, the bigger sibling at roughly 500 billion parameters, is coming. If Super at 120B is this capable, Ultra will be worth watching closely.

Sources:

• NVIDIA Nemotron 3 Super Technical Report (PDF)

• NVIDIA Technical Blog: Introducing Nemotron 3 Super

• NVIDIA Blog: 5x Higher Throughput for Agentic AI

• Artificial Analysis: Nemotron 3 Super — The New Leader in Open Intelligence

• Qodo: Code Review Analysis

• Ollama DGX Spark Benchmarks

• DGX Spark Performance (LMSYS)

• PinchBench

• My own testing on DGX Spark — 19.5 tok/s (Q4_K_M, Ollama), prompt eval 3.51 tok/s

The core problem isn't moving from one controller to another. It's that Ingress NGINX is held together by annotations. Hundreds of nginx.ingress.kubernetes.io/* annotations that control everything from TLS redirects to rate limiting to sticky sessions to canary deployments. These annotations have no direct equivalent in a new controller. Some map cleanly. Some map partially, with caveats. Some have no equivalent at all. And the existing tooling (ingress2gateway) only handles basic routing — it doesn't tell you what you're losing or how to compensate.

That's why I built ing-switch — an open-source CLI + visual UI that takes you through the full migration lifecycle: scan → analyze → generate → verify → cutover → cleanup.

What ing-switch does

The tool has four commands that map to four stages of migration:

ing-switch scan      # detect your controller + list all ingresses
ing-switch analyze   # map every annotation to the target controller
ing-switch migrate   # generate ready-to-apply manifests
ing-switch ui        # open the visual 4-page migration dashboard at :8080

And it supports two migration targets:

Traefik is the lowest-friction path: you keep kind: Ingress, your team learns almost nothing new, and almost all annotations have a Traefik equivalent. Gateway API is the future-proof path: standardized, implementation-agnostic, and where the ecosystem is heading — but it requires more preparation for partial-support annotations.

Annotation coverage: the hard part

This is the part most migration tools skip. ing-switch maps over 50 nginx.ingress.kubernetes.io/* annotations for both targets. Here's a sample:

The tool shows you exactly which category each annotation falls into: fully supported (just apply the generated YAML), partial (the YAML is generated, but read the note about what's different), or unsupported (manual work required, with guidance on the best alternative).

Installation

# macOS arm64
curl -L https://github.com/saiyam1814/ing-switch/releases/latest/download/ing-switch-darwin-arm64 -o ing-switch
chmod +x ing-switch && sudo mv ing-switch /usr/local/bin/

# Linux amd64
curl -L https://github.com/saiyam1814/ing-switch/releases/latest/download/ing-switch-linux-amd64 -o ing-switch
chmod +x ing-switch && sudo mv ing-switch /usr/local/bin/

Or build from source:

git clone https://github.com/saiyam1814/ing-switch.git
cd ing-switch
make build   # builds React UI then embeds it in the Go binary
./ing-switch --help

The Dashboard

The easiest way to understand what ing-switch does is to open the UI:

ing-switch ui   # opens at localhost:8080

Page 1: Detect

The first page scans your cluster and discovers every Ingress resource across all namespaces, identifies the controller type and version, and flags each ingress by complexity.

The tool shows a countdown banner — Ingress NGINX retires March 2026 — and lets you scope the scan to a specific namespace or scan everything.

After clicking Scan Cluster, the tool connects to your cluster via kubeconfig and enumerates every networking.k8s.io/v1 Ingress object. In my test cluster (a vcluster running 11 production-realistic ingresses), the sidebar updated immediately to show "11 ingresses · retiring · ingress-nginx":

The sidebar shows your cluster name, ingress count, and controller status. "Retiring" badge means the detected controller is Ingress NGINX.

Page 2: Analyze

The Analyze page is where you decide your migration target. You pick between Traefik v3 (labeled "Lowest friction") or Gateway API via Envoy Gateway (labeled "Future-proof standard"), then click Analyze Compatibility.

The engine runs through every annotation on every ingress and produces a per-ingress compatibility matrix. Each annotation gets a status badge:

• Green (supported) — the generated YAML covers this fully; just apply it

• Yellow (partial) — the feature works but with limitations; the note explains exactly what's different

• Red (unsupported) — no direct equivalent exists; the tool explains the best available workaround

This is the output that replaces weeks of reading changelogs and GitHub issues.

Page 3: Migrate

The Migrate page generates the complete output directory of Kubernetes manifests. Click Generate Migration Files and the tool writes every file needed for a zero-downtime cutover.

The checklist walks you through every step in order, so you can't accidentally skip a dependency (like applying GatewayClass before HTTPRoutes).

After generation, each step gains View File, Dry-run, and Apply buttons:

The tool generated 27 migration files for Traefik across 11 ingresses. Each checklist item links to the file it creates, and you can preview the YAML before applying.

The Gaps tab gives you the executive summary — which ingresses are fully compatible, which have auto-generated workarounds, and which need manual review:

"Needs Review" ingresses are listed with the specific annotations that require manual attention, so you know exactly what work remains before cutover.

Page 4: Validate

After applying the generated manifests, the Validate page gives you a structured checklist of post-migration checks and lets you run live validation against the cluster.

The checklist covers TLS verification, auth testing, rate limit validation, canary routing verification, and a 24-hour monitoring window before removing NGINX.

The CLI: same power, no browser

Everything the UI does is also available as CLI commands, which makes it scriptable and CI-friendly:

# Scan your cluster
$ ing-switch scan
Cluster: production-cluster
Controller: ingress-nginx v1.9.4 (namespace: ingress-nginx)

NAMESPACE    NAME              HOSTS                     ANNOTATIONS  TLS
ecommerce    ecommerce-shop    shop.example.com          13           yes
security     payment-api       payments.example.com      6            yes
messaging    realtime-chat     chat.example.com          2            yes
...
11 ingresses found across 6 namespaces

# Analyze annotation compatibility for Gateway API
$ ing-switch analyze --target gateway-api
INGRESS                       SUPPORTED   PARTIAL   UNSUPPORTED
ecommerce/ecommerce-shop      6           5         2
security/payment-api          3           2         1
...

# Generate all migration manifests
$ ing-switch migrate --target traefik --output-dir ./migration
Generated 27 files across 11 ingresses

migration/
├── 00-migration-report.md      # Full annotation analysis
├── 01-install-traefik/         # Helm install script + values.yaml
├── 02-middlewares/             # Traefik Middleware CRDs
├── 03-ingresses/               # Updated Ingress resources
├── 04-verify.sh                # Test script per hostname
├── 05-dns-migration.md         # DNS cutover guide
└── 06-cleanup/                 # Remove NGINX after cutover

How it handles the tricky cases

Three annotation scenarios trip up almost every migration, and ing-switch handles all of them:

HTTP→HTTPS redirect

The naive approach — putting both a RequestRedirect filter and a backend rule in the same HTTPRoute — causes a redirect loop where HTTPS traffic gets redirected back to HTTPS. ing-switch generates two separate HTTPRoutes using Gateway API sectionName to attach each route to the correct listener:

# <name>-redirect: attached to HTTP listener only (sectionName: http)
# Returns 301/302 for all incoming HTTP requests

# <name>: attached to HTTPS listener (sectionName: https-0)
# Routes HTTPS traffic to your backends

This is the correct pattern but requires knowing the Gateway API spec deeply enough to spot the constraint. The tool just does it right.

Regex paths

NGINX annotations like use-regex: "true" and paths like /app(/|$)(.*) are common. Gateway API's PathPrefix type rejects regex characters. ing-switch auto-detects paths containing (, ), |, [, ] and switches them to PathMatch.RegularExpression type — even when the use-regex annotation is absent.

Timeout constraints

proxy-read-timeout: 300 and proxy-connect-timeout: 5 look straightforward to map. But Gateway API enforces backendRequest ≤ request, and mapping read→backendRequest (300s) and connect→request (5s) would violate that constraint. ing-switch maps only proxy-read-timeout → backendRequest and intentionally omits proxy-connect-timeout, with a note explaining why.

Generated output: Gateway API example

For a single ingress with SSL redirect, session affinity, and rate limiting, the Gateway API migration generates:

migration/
├── 00-migration-report.md
├── 01-install-gateway-api-crds/
│   └── install.sh
├── 02-install-envoy-gateway/
│   ├── helm-install.sh
│   └── values.yaml
├── 03-gateway/
│   ├── gatewayclass.yaml      # GatewayClass (Envoy Gateway)
│   └── gateway.yaml           # Gateway: HTTP + HTTPS listeners
├── 04-httproutes/
│   ├── ecommerce-ecommerce-shop-redirect.yaml   # HTTP→HTTPS (sectionName: http)
│   └── ecommerce-ecommerce-shop.yaml            # HTTPS backend (sectionName: https-0)
├── 05-policies/
│   └── ecommerce-ecommerce-shop-btp.yaml        # BackendTrafficPolicy (rate limit)
├── 06-verify.sh
└── 07-cleanup/
    └── remove-nginx.sh

Example ingresses

The repo includes 11 production-realistic NGINX Ingress configurations covering every major annotation category you're likely to have in a real cluster:

You can apply them to a test cluster and run the full migration against them:

kubectl apply -f examples/
ing-switch migrate --target gateway-api --output-dir ./migration-examples
kubectl apply --dry-run=client -f ./migration-examples/03-gateway/
kubectl apply --dry-run=client -f ./migration-examples/04-httproutes/

Zero-downtime migration strategy

The tool is designed around a zero-downtime approach:

1. Install the new controller alongside NGINX — both run simultaneously; DNS still points to NGINX

2. Apply generated manifests — Middlewares, HTTPRoutes, Gateway

3. Verify the new controller — run 06-verify.sh to test each hostname against the new IP

4. Shift DNS — update your DNS records to point to the new controller's LoadBalancer IP

5. Monitor for 24 hours — watch for 5xx errors, auth failures, session issues

6. Remove NGINX — run 06-cleanup/remove-nginx.sh

The Migrate page's step-by-step checklist mirrors this order and locks steps until their dependencies are checked off.

Try it

# Install
curl -L https://github.com/saiyam1814/ing-switch/releases/latest/download/ing-switch-darwin-arm64 -o ing-switch
chmod +x ing-switch && sudo mv ing-switch /usr/local/bin/

# Point at your cluster
export KUBECONFIG=~/.kube/config

# Open the visual UI
ing-switch ui

The source, examples, and annotation mapping database are at github.com/saiyam1814/ing-switch.

The annotation mapping database lives in pkg/analyzer/compatibility.go (status + target resource per annotation) and pkg/server/guides.go (human-readable what/fix/example per annotation). PRs for additional annotation mappings are welcome.

March 2026 is closer than it looks. The tools are ready.

Introduction

msfconsole is the heart of the Metasploit Framework and one of the most powerful tools used by penetration testers to identify, exploit, and validate security vulnerabilities. In real-world security assessments as well as Capture The Flag (CTF) challenges, msfconsole is often used to automate and streamline exploitation workflows.

In this blog, we will explore how to use msfconsole from Kali Linux to exploit an intentionally vulnerable machine, Metasploitable2, in a safe and controlled lab environment.

Both machines are hosted on Oracle VM VirtualBox and configured on the same internal network. This setup allows us to simulate real attack scenarios while maintaining proper ethical boundaries.

The goal of this blog is to:

• Understand what msfconsole is and why it is used

• Learn how attackers interact with vulnerable services using Metasploit

• Gain hands-on experience with a realistic exploitation lab

Note: All demonstrations in this blog are performed on machines owned by us or intentionally designed to be vulnerable. Never use these techniques on unauthorized systems.

In the next section, we will briefly look at the lab architecture before launching msfconsole and beginning the exploitation process.

Press enter or click to view image in full size

Setting Up msfconsole and Metasploitable2 (Step-by-Step Lab Setup)

Before launching any exploitation using msfconsole, we must ensure that both the attacker and the vulnerable target are properly set up and reachable. This section covers the complete setup process for msfconsole on Kali Linux and the Metasploitable2 vulnerable server.

1. Setting Up the Attacker Machine (Kali Linux)

Why Kali Linux?

Kali Linux comes pre-installed with hundreds of penetration testing tools, including the Metasploit Framework.

Verify Metasploit Installation

On Kali, Metasploit is installed by default. To verify:

msfconsole --version

If Metasploit is installed correctly, you will see version details.

— — — — — — — — — — — — — — — — —

Start msfconsole

msfconsole

On first launch, Metasploit may:

• Initialize its database

• Create required configuration files

You should now see the familiar msf6 > prompt.

This confirms that msfconsole is ready to use.

Press enter or click to view image in full size

— — — — — — — — — — — — — — — — —

2. Setting Up the Target Machine (Metasploitable2)

What is Metasploitable2?

Metasploitable2 is a deliberately vulnerable Linux machine created for practicing penetration testing techniques.

Start Metasploitable2 VM

• Launch Metasploitable2 in Oracle VM VirtualBox

• Wait until it boots to the login screen

Default Credentials

Username: msfadmin
Password: msfadmin

Login successfully to access the system.

Check IP Address of Metasploitable2

ifconfig

Example output:

inet addr:192.168.56.101

Press enter or click to view image in full size

Note this IP address, as it will be used as the target (RHOSTS) inside msfconsole.

— — — — — — — — — — — — — — — — —

3. Ensure Both Machines Are on the Same Network

Both VMs must be configured with:

• Network Adapter: Host-only Adapter

• Name: VirtualBox Host -Only Ethernet Adapter

This ensures:

• Kali ↔ Metasploitable2 communication

• No internet exposure (safe lab)

Press enter or click to view image in full size

Press enter or click to view image in full size

— — — — — — — — — — — — — — — — —

4. Test Connectivity (Very Important)

From Kali Linux:

ping 192.168.56.101

If you receive replies, your lab network is working correctly.

Press enter or click to view image in full size

— — — — — — — — — — — — — — — — —

5. Confirm Target Visibility Using Nmap

Before using Metasploit, attackers always enumerate first.

nmap -sV 192.168.56.101

You should see multiple intentionally vulnerable services, such as

• FTP (vsftpd 2.3.4)

• SSH

• Samba

• Tomcat

Press enter or click to view image in full size

This confirms that Metasploitable2 is ready for exploitation.

Setup Checklist

✔ Kali Linux boots successfully
✔ msfconsole launches without errors
✔ Metasploitable2 is accessible
✔ Both machines are on the same subnet
✔ Ping & Nmap scans work

Once all checks pass, your lab is fully prepared.

Basic msfconsole Commands (Getting Comfortable with the Interface)

Before jumping into exploitation, it’s important to understand the basic operating system–style commands and navigation used inside msfconsole. This section helps beginners feel confident while working in the Metasploit environment.

We are using Kali Linux with the Metasploit Framework.

Starting msfconsole

Open a terminal in Kali Linux and run:

msfconsole

Once loaded, you will see:

msf6 >

This prompt indicates that msfconsole is ready to accept commands.

Getting Help in msfconsole

Show All Commands

help

or simply:

?

This lists all available commands, similar to using help in an operating system shell.

Press enter or click to view image in full size

Navigation Commands (OS-Like Basics)

CommandDescriptionpwdShows the current module pathcdChange module directorylsList available modulesclearClear the screen

Example:

pwd
ls

These commands work inside Metasploit, not the Linux filesystem.

Searching for Modules

One of the most used commands:

search <keyword>

Example:

search ftp
search samba
search vsftpd

Press enter or click to view image in full size

This helps you quickly find:

• Exploits

• Auxiliary scanners

• Payloads

Understanding Module Types

Metasploit is organized into modules:

Metasploit is organized into different module types, each designed for a specific purpose in the penetration testing lifecycle.

Press enter or click to view image in full size

You can list them using:

ls exploit
ls auxiliary

Using a Module

To select a module:

use exploit/unix/ftp/vsftpd_234_backdoor

Press enter or click to view image in full size

Once selected, the prompt changes to:

msf6 exploit(unix/ftp/vsftpd_234_backdoor) >

This tells you which module is currently active.

Viewing & Setting Options

Show Required Options

show options

Press enter or click to view image in full size

Set Target IP

set RHOSTS 192.168.56.101

Set Port (if needed)

set RPORT 21

To verify:

show options

Press enter or click to view image in full size

Running a Module

run

or

exploit

Both commands do the same thing.

Press enter or click to view image in full size

Session Management Basics

After successful exploitation:

sessions

Interact with a session:

sessions -i 1

Exit session:

Press enter or click to view image in full size

exit

Exiting Modules & msfconsole

In Metasploit, use back to leave the current module and return to the main console. Use quit or exit to close msfconsole completely.

Key Takeaways

✔ msfconsole feels like a mini operating system
✔ search, use, and show options are core commands
✔ Always understand a module before running it
✔ Enumeration comes before exploitation

Conclusion

In this blog, we explored how msfconsole, the core interface of the Metasploit Framework, can be used to exploit a vulnerable FTP service on Metasploitable2 from an attacker machine running Kali Linux.

Starting from proper lab setup and network configuration, we moved through the essential stages of a penetration test: service enumeration, exploit selection, configuration, and execution. By exploiting the outdated vsftpd 2.3.4 service, we demonstrated how a single vulnerable service can lead to full system compromise when basic security practices are ignored.

This exercise highlights several important lessons:

• Enumeration is more important than exploitation

• Outdated services pose serious security risks

• Automation tools like msfconsole must be used with understanding, not blindly

• Ethical hacking is about learning and improving security, not breaking systems

Practicing in a controlled environment like Metasploitable2 helps build a strong foundation for CTFs, real-world penetration testing, and defensive security awareness.

In future labs, this knowledge can be extended to:

• Exploiting other services such as Samba and Tomcat

• Using Meterpreter for advanced post-exploitation

• Understanding how blue teams detect and prevent such attacks

Always remember: with great power comes great responsibility. Use these skills only where you have explicit permission.

Follow Kubesimplify on Hashnode, Twitter/X and LinkedIn. Join our Discord server to learn with us!

Kubernetes 1.35 is released and again the velocity for this project never makes you feel that there is no innovation left. The releases is focussed on more stability, AI workloads, introducing concept of workload and so much more.

Huge thanks to all the contributors and the release team for making this happen. Let’s look at some of the cool features for this release.

Native Gang Scheduling is finally here (and it’s a big deal)

For years, Kubernetes scheduled pods one by one. That model works well for stateless services, but it breaks down badly for distributed workloads, think PyTorch training jobs, Spark, Ray, or MPI-style applications.

Kubernetes 1.35 introduces native Gang Scheduling through a new Workload API. This enables all-or-nothing scheduling: either all pods in a group get scheduled together, or none do. No more half-started jobs burning GPUs while waiting for peers. This is foundational work for AI and HPC workloads on Kubernetes.

First, define a Workload that represents the gang policy:

apiVersion: scheduling.k8s.io/v1alpha1
kind: Workload
metadata:
  name: demo-workload
  namespace: gang-demo
spec:
  podGroups:
  - name: workers
    policy:
      gang:
        minCount: 3

Then reference it from your Pod:

apiVersion: v1
kind: Pod
metadata:
  name: worker-0
  namespace: gang-demo
  labels:
    app: gang-demo
spec:
  workloadRef:
    name: demo-workload
    podGroup: workers
  containers:
    - name: demo
      image: nginx:1.27
      resources:
        requests:
          cpu: "200m"
          memory: "128Mi"
---
apiVersion: v1
kind: Pod
metadata:
  name: worker-1
  namespace: gang-demo
  labels:
    app: gang-demo
spec:
  workloadRef:
    name: demo-workload
    podGroup: workers
  containers:
    - name: demo
      image: nginx:1.27
      resources:
        requests:
          cpu: "200m"
          memory: "128Mi"
---
apiVersion: v1
kind: Pod
metadata:
  name: worker-2
  namespace: gang-demo
  labels:
    app: gang-demo
spec:
  workloadRef:
    name: demo-workload
    podGroup: workers
  containers:
    - name: demo
      image: nginx:1.27
      resources:
        requests:
          cpu: "200m"
          memory: "128Mi"

The scheduler now understands that these pods belong together and either they will be scheduled together or won’t get scheduled.

Saiyam Pathak already published a full hands-on video on the Kubesimplify YouTube channel, where he builds Kubernetes 1.35 from source, enables the Workload API, and demonstrates native Gang Scheduling end to end. If you’re working with AI or distributed computing, that walkthrough is worth your time.

In place pod update moves to stable

In Kubernetes v1.35, In-place update of Pod resources graduated to GA (Stable), which means you can now change a running Pod’s CPU and memory requests/limits without recreating the Pod (and often without restarting containers). This is a big deal for stateful and batch workloads where a restart is costly: you can do smoother, less disruptive vertical scaling, and Kubernetes will reflect desired resources in spec while tracking what’s actually applied in status as the kubelet works through the resize. This graduation comes from KEP #1287 (SIG Node).

User namespaces in Pods

Linux user namespaces in Kubernetes allows pods to run with different user and group IDs than the host while preserving strong isolation. With this feature, a process can run as root inside a container but map to an unprivileged user on the node, significantly reducing the blast radius of container breakouts and mitigating several high-severity CVEs. It adds a new pod.spec.hostUsers field to opt into user namespaces, integrates with CRI and idmapped mounts for safe volume handling, and aligns with Pod Security Standards to safely relax certain restrictions when enabled. Overall, this strengthens node-to-pod and pod-to-pod isolation without changing default behaviour, making Kubernetes workloads more secure by design.

Kubernetes is finally letting go of old baggage

Kubernetes 1.35 draws a hard line under several legacy technologies.

If your nodes still rely on cgroup v1, the kubelet will not start by default anymore. Most modern Linux distributions have already moved to cgroup v2, which offers better resource isolation and consistency. Kubernetes waited long enough and now is the time to take action on this.

The same applies to containerd 1.x. Kubernetes 1.35 is the last release that supports it. If you plan to upgrade beyond this version, moving to containerd 2.x is no longer optional.

IPVS mode in kube-proxy is also deprecated. While once popular for performance reasons, it became increasingly complex to maintain. Kubernetes is consolidating around nftables, reducing operational and maintenance burden.

Ingress NGINX retirement: This came as a massive blow where due to maintainer debt the ingress nginx which is uses by thousands of organisations is now getting retired. Although Chainguard has come up to support and maintain the version as per this announcement. IMO you should start your migrations to gatewayAPI and we have a full end to end video about this on Kubesimplify already where you get to see entire demo on how to migrate from ingress to gatewayAPI.

Securing Cached Images in Kubernetes 1.35

KEP-2535 addresses a security gap in Kubernetes image pulling. Historically, when imagePullPolicy was set to IfNotPresent or Never, a pod could start using a private image already cached on a node even if it did not have the credentials to pull that image itself. In multi-tenant clusters, this meant one workload could unintentionally benefit from another workload’s credentials. Kubernetes 1.35 introduces kubelet-level credential verification for cached images, ensuring that a pod is authorized to use an image before it can run, regardless of whether the image is already present. With this beta feature enabled by default and configurable via imagePullCredentialsVerificationPolicy, clusters can now enforce stricter image access without forcing Always pulls, significantly improving tenant isolation and supply-chain security.

Overall I am excited about

• In-place Pod resource updates (now GA): being able to adjust CPU/memory without restarting pods is huge for stateful and long-running workloads.

• Native gang scheduling and the new Workload API direction is a big deal for ML/HPC style workloads and any “all-or-nothing” batch pipelines.

• Pod certificates (beta): a strong step toward native workload identity and simpler mTLS setups without always relying on extra controllers.

• Node declared features (alpha): a practical way to reduce upgrade/version-skew surprises by letting nodes declare supported capabilities before scheduling decisions happen.

Overall this release consists of 60 enhancements, including 17 stable, 19 beta, and 22 alpha features. What is the feature that you are most excited about for this Kubernetes release and is there anything you would like to see a deep dive on Kubesimplify.

We've all been there: you run a resource-intensive Docker build or a compute-heavy container, and your laptop's fans start screaming, the CPU maxes out, and your entire machine slows to a crawl. For developers working with large applications, complex multi-stage builds, or AI/ML workloads, this is a frustratingly common problem. But what if you could have the best of both worlds here? You have the awesome Docker Desktop experience along with the power of a high-performance cloud machine?

Enter Docker Offload, a game-changing service that lets you offload your Docker builds and container runs to a secure, dedicated cloud environment. It's not about learning a new cloud platform; it's about seamlessly extending your existing Docker workflow to where the resources are.

What Exactly Is Docker Offload?

Docker Offload is a fully managed service that allows you to execute Docker commands on powerful, remote cloud infrastructure while maintaining the same user experience on your local machine. Think of it as a bridge that connects your Docker Desktop to a beefy, cloud-based Docker daemon. You still type docker build and docker run In your terminal, but the heavy lifting is done remotely, freeing up your local resources.

This service is particularly useful for tasks that are traditionally a pain on a local machine, such as:

• Building large, complex images: Multi-stage builds with many dependencies or a large build context can take ages on a standard laptop.

• Running compute-intensive workloads: Tasks like machine learning model training, AI inferencing, data processing, and video transcoding often require more CPU, RAM, and most importantly, GPU power than a typical developer machine has.

• Standardizing development environments: It helps teams with a variety of hardware specs work on the same projects without performance bottlenecks, ensuring a consistent and fast experience for everyone.

Key Benefits of Offloading to the Cloud

1. Massive Performance Boost: By leveraging powerful cloud instances with access to NVIDIA L4 GPUs, you can slash build times and run containers that would otherwise overwhelm your local machine.

2. No Changes to Your Workflow: This is the magic of Docker Offload. It's not a new tool or a different syntax. You use the same docker and docker compose commands you already know and love.

3. Resource Optimization and Cost Efficiency: The service uses a pay-as-you-go model. Cloud environments are ephemeral, meaning they're provisioned for your session and then automatically shut down and cleaned up after a period of inactivity. This prevents you from paying for idle resources.

4. Seamless Local-Cloud Integration: Even though your container is running in the cloud, exposed ports are still accessible via localhost. This allows you to interact with your running application as if it were local, making development, debugging, and testing a breeze.

5. Shared Build Cache: Docker Offload uses a shared cache that can be reused by your team, further accelerating build times and ensuring consistency across different machines.

Getting Started: A Quick Guide to Implementation

Implementing Docker Offload is surprisingly simple. You'll need Docker Desktop version 4.43 or later.

1. Start Docker Offload

In your terminal, just run:

docker offload start

This command will prompt you to log in to your Docker account and will ask if you want to enable GPU support. If you're working on AI/ML projects, enabling the GPU is highly recommended. Once it's running, you'll see a cloud icon in your Docker Desktop dashboard, and your terminal's context will be set to the new cloud environment.

2. GPU test with Docker offload

With Docker Offload active, you can now use your regular commands. Let's try running a Minimal GPU smoke test with Compose + Offload.

Create a compose.yaml file

services:
  gpu-smoke:
    image: nvidia/cuda:12.4.1-base-ubuntu22.04
    command: nvidia-smi
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

This will run nvidia-smi In the Offload cloud GPU instance and print the GPU info. You should see an NVIDIA-SMI table in the logs that confirms the GPU is available in your Offload session.

In both cases, you'll see the output as if the container ran locally, but it's all happening on the powerful cloud machine.

3. Stop Docker Offload

When you're finished, you can switch back to local execution by running:

docker offload stop

This will tear down the cloud environment, ensuring you don't incur any additional costs.

4. Checking Docker Offload Run history

   You can also go to your account under your Docker account and see the run history for the offload to track the usage. You can set up the usage limits to avoid any cost surprises.

   

Real-World Use Cases

• AI/ML Development: Easily train or run inference on large models without the need for an expensive local GPU.

• High-Velocity Development: Accelerate CI/CD pipelines and local development feedback loops by offloading builds to a powerful, consistent environment.

• On-Demand Compute: Quickly spin up a resource-heavy container for a short task (like data processing or video rendering) without impacting your local machine's performance.

• Developer Experience: Provide every developer on a team with the same powerful environment, regardless of their local hardware, eliminating "it works on my machine" issues.

Conclusion

Docker Offload is a powerful feature that bridges the gap between local convenience and cloud-scale resources. It's a testament to Docker's ongoing commitment to making containerization accessible and efficient for every developer. So next time your laptop's fans start spinning up, remember you have a new, more powerful option. What do you use for your development workflows when you need these big machines but want the same developer experience?

Enter Docker!

Why is Docker a game-changer for AI, and specifically for MCP tools?

Docker has already proven to be the de facto standard for creating and distributing containerized applications. Its user experience is the key reason why I and millions of other developers use Docker today. Over the years, Docker has evolved to cater to the needs of developers, and it entered the AI game too. With so many MCP servers having a set of configurations living on separate GitHub repositories and different installation methods, Docker has again changed the game on how we think and run these MCP servers and connect to MCP clients like Claude.

Now we already have MCP that solves the problem of how Agents talk with the tools. In simple terms, your LLM models are capable of connecting to your tools and performing a wider set of actions. But how simple it is? There are 100s of MCP servers out there, each having its configurations and its own GitHub repository.

Docker has introduced the Docker MCP Catalog and Toolkit (currently in Beta). This is a comprehensive solution designed to streamline the developer experience for building and using MCP-compatible tools.
We can simply go to the extensions and install the Docker MCP toolkit.

What is the Docker MCP Catalog?

The Docker MCP Catalog is a centralized, trusted registry that offers a curated collection of MCP-compatible tools packaged as Docker images. Integrated with Docker Hub and available directly through Docker Desktop, it simplifies the discovery, sharing, and execution of over 100 verified MCP servers from partners like Stripe, Grafana etc. By running each tool in an isolated container, the catalog addresses common issues such as environment conflicts, inconsistent platform behaviour, and complex setups, ensuring portability, security, and consistency across systems. Developers can instantly pull and run these tools using Docker CLI or Desktop, with built-in support for agent integration via the MCP Toolkit

• Centralized Discovery: A dedicated place (often under the mcp/ on Docker Hub ) to find a growing list of MCP servers. Docker has mentioned collaborations with providers like Stripe, Elastic, and Neo4j, indicating a rich ecosystem of tools.

• Verified and Versioned Tools: The catalog aims to provide access to tools from verified publishers and ensures that tools are versioned, allowing developers to rely on specific, stable releases.

• Easy Distribution : Easy pull based distribution using Docker’s infra

What is the MCP Toolkit?

The Docker MCP Toolkit is a powerful companion to the MCP Catalog that streamlines the setup, management, and execution of containerized MCP servers and their integration with AI agents. It eliminates the need for manual configurations by offering one-click setup, secure defaults, and built-in compatibility with popular LLM-based clients like Docker gordon, Claude Desktop, Cursor, and Continue.dev. Acting as both an aggregator for MCP servers and a gateway for connected clients, the toolkit enables developers to browse, configure, and launch tools directly from Docker Desktop. Security is a core focus, all mcp servers are signed and includes SBOMs for transparency. The MCP Toolkit makes it easy to discover and run tools from the catalog, connect clients like Gordon or Docker AI Agent, and build secure, agent-driven workflows with minimal overhead.

Key functionalities of the MCP Toolkit:

• Simplified Installation: Easily pull and run MCP tools (servers) from the catalog, often with just a few clicks or simple commands.

• Secure Credential Management: One of the major pain points in tool integration is handling authentication securely. The MCP Toolkit often includes features like OAuth-based authentication and secure storage for credentials, preventing the risky practice of hardcoding secrets.

• Isolated and Secure Execution: Leverages Docker's containerization to run tools in isolated environments, enhancing security and stability.

• Client Integration: Facilitates connecting these MCP tools/servers to various AI agent clients (e.g., popular AI models or development environments like Claude, Cursor) without needing to rewrite code for each client.

Together, the Docker MCP Catalog and Toolkit provide a foundational layer for developers, making it easier to find, safer to use, and more scalable to integrate external tools into their AI agent applications.

What You Can Do with the Catalog

Let’s see it in action:

There are four MCP Clients available in the Docker Desktop
1. Gordon
2. Claude Desktop
3. Cursor
4. Continue.dev

You can simply connect to any of these, but we are connecting to the Claude desktop. After installing, click on the Connect button on the Docker Desktop app. It automatically creates a configuration json file and puts it in the right place. When you restart Claude, you will be able to see the MCP_Docker with tools. Here tools will be the MCP servers that you have installed from the catalog.

Now you can install MCP servers, in this case, we have installed docker mcp server.

Once this is enabled, you can go to Claude and simply use human language, which would be able to interact with the tools. Here you can see 1 tool, and that tool will be listed as part of Claude desktop too.

Then let’s perform a simple task of creating an nginx container with the help of the Claude desktop:

As soon as you run the command, it pops up a dialog box to ask permission for using the external integration, that is Docker in this case.

So we can see below that the nginx container has been created with the ID 6d87626dcad5.

Conclusion

Docker has always believed in streamlining the developer workflow; they have been a one-stop solution for packaging applications as Images and then running them as containers with ease. They also let you build and push WASM artifacts, and now with the AI wave, they are simplifying the whole AI ecosystem, keeping the same developer experience and fluid UI. In the future, we will build an MCP server too and then see how that works, till then let me know your favourite MCP server and what you think about Docker MCP catalog and toolkit!

In this blog, we’ll explore the top highlights of Kubernetes v1.33.

How to Try Out Kubernetes 1.33

One of the biggest questions people often have is how they can try out the new Kubernetes version as soon as it is released. Cloud providers take some time to update the Kubernetes version, and until then, you can use vCluster. vCluster allows you to create a virtual cluster running in any version of vanilla Kubernetes, including Kubernetes version 1.33(the latest version at the time of writing this blog) with very simple steps.

Create a vcluster.yaml file:

controlplane:
  distro:
    k8s:
      version:v1.33.0Copy

Then, create the virtual cluster:

vcluster create k8s133 -f vcluster.yamlCopy

Ensure your context is set to the virtual cluster and verify the nodes:

kubectl get nodes
NAME              STATUS   ROLES    AGE   VERSION
live-demo-e0is0   Ready    <none>   13m   v1.33.0
This setup enables you to test new features and plan upgrades accordingly.

  Warning  FailedScheduling        16m                default-scheduler        0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
  Normal   Scheduled               16m                default-scheduler        Successfully assigned vcluster-demo4/demo4-0 to live-demo-e0is1
  Normal   SuccessfulAttachVolume  16m                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-c615f428-70f6-4921-88a9-7ab26349ad00"
  Normal   Pulled                  16m                kubelet                  Container image "ghcr.io/loft-sh/vcluster-pro:0.24.0" already present on machine
  Normal   Created                 16m                kubelet                  Created container vcluster-copy
  Normal   Started                 16m                kubelet                  Started container vcluster-copy
  Normal   Pulling                 16m                kubelet                  Pulling image "registry.k8s.io/kube-controller-manager:v1.33.0"
  Normal   Pulled                  16m                kubelet                  Successfully pulled image "registry.k8s.io/kube-controller-manager:v1.33.0" in 2.406s (2.406s including waiting). Image size: 27635030 bytes.
  Normal   Created                 16m                kubelet                  Created container kube-controller-manager
  Normal   Started                 16m                kubelet                  Started container kube-controller-manager
  Normal   Pulling                 16m                kubelet                  Pulling image "registry.k8s.io/kube-apiserver:v1.33.0"
  Normal   Pulled                  16m                kubelet                  Successfully pulled image "registry.k8s.io/kube-apiserver:v1.33.0" in 2.267s (2.267s including waiting). Image size: 30071307 bytes.
  Normal   Created                 16m                kubelet                  Created container kube-apiserver
  Normal   Started                 16m                kubelet                  Started container kube-apiserver
  Normal   Pulled                  15m                kubelet                  Container image "ghcr.io/loft-sh/vcluster-pro:0.24.0" already present on machine
  Normal   Created                 15m                kubelet                  Created container syncer
  Normal   Started                 15m                kubelet                  Started container syncerCopy

Note: While vCluster allows you to experiment with the latest Kubernetes features, some functionalities, particularly those that interact directly with the underlying host's operating system, kernel, kubelet, or hardware, might have limited or no effect when the host cluster is running an older Kubernetes version. Let’s test out a couple of features from Kubernetes version 1.33 using vCluster.

First let's talk some of the cool stuff from 1.33 and then we will try a couple of features on vCluster 1.33

In-Place Pod Vertical Scaling (Beta)

In Kubernetes v1.33, the long‐awaited in-place Pod resize feature has graduated from alpha to beta and is now enabled by default. Instead of restarting Pods to adjust CPU or memory, you can simply patch the Pod’s resources via the new resize subresource and monitor its progress through two Pod conditions (PodResizePending and PodResizeInProgress). After its alpha debut in v1.27, resizing sidecar containers in place is now supported in beta. By reducing disruption and enabling more efficient resource use, especially for stateful or long-running workloads, this beta release lays the groundwork for future integration with the Vertical Pod Autoscaler and further production hardening based on community feedback.

kubectl edit pod <pod-name> --subresource resizeCopy

Read more about this feature here.

User namespaces on by default

Kubernetes v1.33 now enables user namespaces by default, a significant security feature that enhances isolation between containers and the host system. User namespaces work by mapping user and group IDs (UIDs/GIDs) within a container to different, unprivileged UIDs/GIDs on the host. This is crucial because it prevents lateral movement between containers and increases host isolation, meaning that even if a container is compromised and runs as root internally, it has no elevated privileges on the host. This default enablement in Kubernetes 1.33 allows pods to opt-in to this stronger security posture without needing to enable specific feature flags, provided the underlying stack requirements are met.

To enable user namespaces in a pod you use the hostUsers value and set it to false.

Example:

apiVersion: v1
kind:Pod
metadata:
  name: userns
spec:
  hostUsers: false
  containers:
  - name: shell
    command: ["sleep", "infinity"]
    image: debianCopy

User namespaces allow you to run as root inside the container, but not have privileges in the host.

Job Success Policy

In Kubernetes 1.33, the Job SuccessPolicy feature has reached General Availability, offering more flexible completion criteria for batch workloads. This feature is particularly important for scenarios like leader-follower patterns (e.g., MPI used in scientific simulations, AI/ML, and HPC) where the overall job can be considered successful even if not all individual pods or indexes complete successfully. Instead of requiring every pod to succeed, users can now define specific rules, such as a minimum number of successfully completed indexes or the success of a specific leader index, allowing for early exit and resource cleanup once the defined success criteria are met, thereby optimizing resource usage and accommodating more complex batch processing needs.

HorizontalPodAutoscaler Configurable Tolerance

This Alpha feature lets you control how quickly your applications scale up or down. Before, there was a fixed 10% buffer for all applications before they would scale. Now, you can set this buffer specifically for each application. This means you can make your application scale up very quickly if there's a sudden increase in traffic (by setting a low or zero buffer for scaling up) and scale down more slowly to avoid too many changes if traffic drops a little (by setting a higher buffer for scaling down). This gives you better control, especially for large applications, helping to keep them stable and avoid unnecessary changes when small things fluctuate. You'll need to turn on the "HPAConfigurableTolerance" as a feature gate.

Example - an HPA with a tolerance of 5% on scale-down, and 1% tolerance on scale-up, would look like the following:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: demo
spec:
  ...
  behavior:
    scaleUp:
      tolerance: 0.01
    scaleDown::
      tolerance: 0.05Copy

New configuration option for kubectl with .kuberc for user preferences

In v1.33, kubectl introduces an alpha feature that lets you keep aliases, default flags (e.g. server-side apply), and other preferences in a separate ~/.kube/kuberc file, rather than crowding your kubeconfig.

New features in DRA

Also in Kubernetes version 1.33, Dynamic Resource Allocation (DRA), a flexible way for applications to request specific hardware like GPUs, is getting better even though it's still in beta. A new beta update lets device drivers share more detailed status information. Several new early-test features have also been added:

• the ability to split single devices into smaller usable parts ("Partitionable Devices")

• a way to mark some devices as unusable unless an application specifically allows it ("Device Taints and Tolerations")

• an option for users to list their preferred devices in order ("Prioritized List")

• improved security for administrative access to devices.

These changes aim to make it easier and more efficient to use specialized hardware in Kubernetes, with the goal of making DRA fully available soon. DRA is supposed to go GA in Kubernetes 1.34.

SideCar Container graduates

Launched in 1.28 as an alpha feature, sidecar finally graduated to stable in 1.33. These containers run alongside your primary application container within the same Pod. Kubernetes implements sidecars as a special type of init container configured with restartPolicy: Always. This ensures they start before your main application containers, run for the entire lifecycle of the Pod, and are automatically terminated after the main containers finish. This native support means you can rely on sidecars to use probes (startup, readiness, and liveness) to signal their health, and their memory (OOM) scores are adjusted like primary containers to prevent them from being terminated too early under memory pressure.

Kubernetes 1.33 features on vCluster

Now we’ll try these two features using vCluster.

• Ordered Namespace Deletion

• ClusterTrustBundle

To enable these features we’ll need to set the following feature flags:

controlPlane:
  distro:
    k8s:
      version: v1.33.0
      apiServer:
        extraArgs:
          - "--feature-gates=OrderedNamespaceDeletion=true"
          - "--feature-gates=ClusterTrustBundle=true
ClusterTrustBundleProjection=true"
          - "--runtime-config=certificates.k8s.io/v1beta1/clustertrustbundles=true"Copy

Command

vcluster create k8s-1-33-dev --namespace vcluster-133-ns -f kube133.configCopy

Output

vcluster list
     NAME     |    NAMESPACE    | STATUS  | VERSION | CONNECTED |    AGE      
  ---------------+-----------------+---------+---------+-----------+-------------
    k8s-1-33-dev | vcluster-133-ns | Running | 0.24.0  | True      | 5h47m46s   Copy

Ordered Namespace Deletion

It is an alpha feature that brings in an opinionated deletion process in the Kubernetes namespace deletion to ensure secure deletion of resources within a namespace. The current deletion process is semi-random, which may lead to unintended behavior, such as Pods persisting after the deletion of their associated NetworkPolicies. If this feature is turned on, the Pods will be deleted before other resources with logical and security dependencies. This design enhances the security and reliability of Kubernetes by mitigating risks arising from the non-deterministic deletion order. In order to enable this feature you need to enable the feature flag "--feature-gates=OrderedNamespaceDeletion=true"

Let’s create the pod

apiVersion: v1
kind: Namespace
metadata:
  name: demo-order
---
apiVersion: v1
kind: Pod
metadata:
  name: demo-pod
  namespace: demo-order
spec:
  containers:
  - name: pause
    image: k8s.gcr.io/pause:3.6
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
  namespace: demo-order
spec:
  podSelector: {}
  policyTypes:
  - IngressCopy

This will create a namespace, Pod and a networkPolicy. Now in order to see the ordered deletion you can use any terminal in split view or just two terminal windows so that you can watch networkPolicy and pods.

Once you delete the namespace, you will observe that the pod gets deleted first and then the namespace, and see similar results as in the below image.

ClusterTrustBundle

The ClusterTrustBundle is a beta feature in Kubernetes 1.33, part of the certificates.k8s.io/v1beta1 API group, used to manage cluster-scoped X.509 trust anchors. It allows you to publish CA certificates that in-cluster components (e.g., webhooks, image registries, or workloads) can use for certificate verification.

Let’s try a sample and run it on vCluster:

Download a demo CA

curl -s https://letsencrypt.org/certs/isrgrootx1.pem -o isrgrootx1.pemCopy

Parsing check

openssl x509 -in isrgrootx1.pem -noout -textCopy

Use above and create a manifest as below:

apiVersion: certificates.k8s.io/v1beta1
kind: ClusterTrustBundle
metadata:
  name: letsencrypt-isrg-root-x1
spec:
  # Global (signer-unlinked) bundle: visible to all workloads
  trustBundle: |
    -----BEGIN CERTIFICATE-----
    MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
    TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
    cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4
    WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu
    ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY
    MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc
    h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+
    0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U
    A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW
    T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH
    B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC
    B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv
    KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn
    OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn
    jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw
    qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI
    rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV
    HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq
    hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
    ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ
    3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK
    NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5
    ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur
    TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC
    jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc
    oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq
    4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA
    mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d
    emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
    -----END CERTIFICATE-----Copy

Apply the manifest

Command

kubectl apply -f bundle.yamlCopy

Output:

clustertrustbundle.certificates.k8s.io/letsencrypt-isrg-root-x1 createdCopy

kubectl get clustertrustbundle.certificates.k8s.io
NAME                       SIGNERNAME
letsencrypt-isrg-root-x1   <none>Copy

The above examples allow you to test at the API level some of the new features in Kubernetes 1.33 and how they work on vCluster.

Deprecated and Removed Features

Endpoints API Deprecation: The traditional Endpoints API is deprecated in favor of EndpointSlices, which offer better scalability and support for modern features.

Removal of gitRepo Volume Type: The deprecated gitRepo volume type has been removed. Users should transition to alternatives like init containers with git clone.

Conclusion

Kubernetes releases always come with many new features and this time is no different. Let us know which Kubernetes features you are most excited about. And do read the official announcement post from the release team. A huge thanks to all the members of the release team who helped with this amazing release. If you want to quickly try out Kubernetes 1.33, you can easily use vCluster.

Let’s dive into the step-by-step guide.

What is vCluster?

vCluster is a technology that allows you to create lightweight, isolated Kubernetes clusters within a host cluster. These virtual clusters offer full Kubernetes functionality while being resource-efficient, making them ideal for scenarios like PR testing environments.

Why Ephemeral PR Environments?

Ephemeral environments allow:

• Testing pull request changes in an isolated environment

• Quick validation without interfering with the main cluster

• Automatic cleanup post-testing

By leveraging vCluster and GitHub Actions, you can automate this workflow and ensure every PR gets its own dedicated environment.

Prerequisites:

Kubernetes cluster

You need to have a Kubernetes cluster, in this case I am using a DigitalOcean Kubernetes cluster but any should work. I am creating a realistic production scenario so for that I used a cluster that can create service type: LoadBalancer.

Command:

kubectl get nodes

Output:

kubectl get nodes
NAME              STATUS   ROLES    AGE   VERSION
live-demo-e0is0   Ready    <none>   19d   v1.31.1
live-demo-e0is1   Ready    <none>   19d   v1.31.1
live-demo-e0isz   Ready    <none>   19d   v1.31.1

Deploying Ingress controller

Command

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.9.4/deploy/static/provider/cloud/deploy.yamlCopy

Output

kubectl get po,svc -n ingress-nginx
NAME                                           READY   STATUS      RESTARTS   AGE
pod/ingress-nginx-admission-create-lcb85       0/1     Completed   0          19d
pod/ingress-nginx-admission-patch-xl2fk        0/1     Completed   0          19d
pod/ingress-nginx-controller-79fcc99b4-7f7ls   1/1     Running     0          19d

Getting the LoadBalancer IP for the ingress controller:

Command:

kubectl get svc -n ingress-nginx

Output:

NAME                                         TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
service/ingress-nginx-controller             LoadBalancer   10.109.28.126   209.38.160.229   80:31228/TCP,443:30435/TCP   19d
service/ingress-nginx-controller-admission   ClusterIP      10.109.15.162   <none>           443/TCP                      19dCopy

Domain mapping:

For our application we need dynamic ingress for testing so what we have done here is added the loadbalancer IP of the ingress controller as the A record to the Domain.

Connect the Kubernetes cluster to the platform

We will enable vCluster Pro in order to use templates and create the clusters. For simplicity, I am using my vcluster.cloud account and then creating the access key to login. In this way I don’t have to run any agent on the current cluster. You can either run vcluster platform start or sign up on vCluster cloud and once you login, you should be able to go to access keys and create a short lived access key for the demo (remember to delete the key post demo for security reasons).

Command:

vcluster platform login https://saiyam.vcluster.cloud --access-key <your-access-key>

Output:

Create a template under vCluster templates in the vCluster cloud platform instance.

sync:
  fromHost:
    ingressClasses:
      enabled: true
  toHost:
    ingresses:
      enabled: true
external:
  platform:
    autoSleep:
      afterInactivity: 3600  # Automatically sleep after 1 hour of inactivity

Until now we have a Kubernetes cluster with ingress controller installed and the Public IP of the nginx controller pointed to our domain.

We also have logged into the platform using the access keys created using vcluster.cloud. Now let’s see the demo application that we have.

Demo Application

The scenario we are trying to achieve here involves a sample application deployed onto a Kubernetes cluster. Often, in organizations, new features or bug fixes need to be deployed and tested before being merged into the main branch. In this case, a developer raises a pull request and adds a label to test it. Based on GitHub Actions, the application is built, and then a deployment, service, and ingress Kubernetes object file are generated and pushed to a new branch. A virtual cluster is created, and the new deployment file is applied, allowing the developer to test and verify the new application deployment.

Let’s see how this looks in practice.

GitHub repo — https://github.com/saiyam1814/vcluster-demo

The application for this demo is a simple Go-based HTTP server:

package main
import (
    "fmt"
    "net/http"
)
func handler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintln(w, "Hellooo, World for blog!!")
}
func main() {
    http.HandleFunc("/", handler)
    fmt.Println("Starting server on :8080")
    err := http.ListenAndServe(":8080", nil)
    if err != nil {
        panic(err)
    }
}

Step 1: Setting Up the Deployment Template

The application is packaged as a Kubernetes deployment and exposed via a service and ingress. The deployment uses Jinja2 templating to inject dynamic values like the image tag and ingress host.

tmpl/deploy.j2:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world
  labels:
    app: hello-world
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-world
  template:
    metadata:
      labels:
        app: hello-world
    spec:
      containers:
      - name: hello-world
        image: {{ image_deploy_tag }}
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: hello-world
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: hello-world
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world
spec:
  ingressClassName: nginx
  rules:
  - host: {{ ingress_tag }}
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: hello-world
            port:
              number: 80

Step 2: Automating with GitHub Actions

GitHub Actions handles the workflow from building the application to deploying it on a vCluster.

PR Workflow

File: .github/workflows/build-and-deploy.yml This workflow:

1. Builds the application with the latest changes made by the developer using ko

2. Pushes the container image to docker hub account(credentials for which should be set in the Actions secret as described previously)

3. Creates a deployment manifest using Jinja2 — The action will replace the ingress host and the deployment image variables mentioned in the jinja template and then push to a new feature branch.

4. Creates a vCluster.

5. Deploys the application to the vCluster.

6. Exposes it via ingress for testing.

name: Build and Deploy with vCluster

on:
  pull_request:
    types: [labeled]jobs:
  build-and-deploy:
    if: ${{ github.event.label.name == 'test' }}
    runs-on: ubuntu-latest    steps:
      # Step 1: Checkout PR Code
      - name: Checkout PR Code
        uses: actions/checkout@v3
        with:
          ref: ${{ github.event.pull_request.head.sha }}
      # Step 2: Set up Go
      - name: Set up Go
        uses: actions/setup-go@v4
        with:
          go-version: '1.22.5'
      # Step 3: Set up ko
      - name: Set up ko
        uses: ko-build/setup-ko@v0.6
        with:
          version: v0.14.1
      # Step 4: Log in to Docker Hub
      - name: Log in to Docker Hub
        env:
          KO_DOCKER_REPO: docker.io/saiyam911
        run: |
          echo "${{ secrets.DOCKER_PASSWORD }}" | ko login docker.io --username ${{ secrets.DOCKER_USERNAME }} --password-stdin
      # Step 5: Build and Push Image
      - name: Build and Push Image
        env:
          KO_DOCKER_REPO: docker.io/saiyam911/vcluster-demo
        run: |
          cd app
          export IMAGE_TAG=sha-$(git rev-parse --short HEAD)
          echo "image_deploy_tag=docker.io/saiyam911/vcluster-demo:$IMAGE_TAG" >> $GITHUB_ENV
          ko build --bare -t $IMAGE_TAG
      # Step 6: Generate Deployment Manifest
      - name: Generate Deployment Manifest
        uses: cuchi/jinja2-action@v1.1.0
        with:
          template: tmpl/deploy.j2
          output_file: deploy/deployment.yaml
          strict: true
          variables: |
            image_deploy_tag=${{ env.image_deploy_tag }}
            ingress_tag=pr${{ github.event.pull_request.number }}.vcluster.tech
      # Step 7: Install vCluster CLI
      - name: Install vCluster CLI
        uses: loft-sh/setup-vcluster@main
      # Step 8: Login to vCluster Platform
      - name: Login to vCluster Platform instance
        env:
          LOFT_URL: ${{ secrets.VCLUSTER_PLATFORM_URL }}
          ACCESS_KEY: ${{ secrets.VCLUSTER_ACCESS_KEY }}
        run: |
          vcluster platform login $LOFT_URL --access-key $ACCESS_KEY
      # Step 9: Create vCluster for the PR
      - name: Create A vCluster
        env:
          NAME: pr-${{ github.event.pull_request.number }}
        run: |
          vcluster platform create vcluster $NAME --project default --template my-template --link "Preview=http://pr${{ github.event.pull_request.number }}.vcluster.tech"
      # Step 10: Deploy to vCluster
      - name: Deploy Application to vCluster
        run: |
          kubectl apply -Rf deploy/
      # Step 11: Test Application with curl
      - name: Test Application
        run: |
          sleep 10
          curl --retry 5 --retry-delay 10 http://pr${{ github.event.pull_request.number }}.vcluster.tech

Step 3: Cleanup Workflow

Once the PR is merged or the label is removed, the ephemeral vCluster is deleted.

File: .github/workflows/cleanup.yml

name: Clean Up vCluster

on:
  pull_request:
    types: [closed, unlabeled]jobs:
  cleanup:
    if: (github.event.action == 'closed' && github.event.pull_request.merged == true) || github.event.label.name == 'test'
    runs-on: ubuntu-latest    steps:
      # Step 1: Install vCluster CLI
      - name: Install vCluster CLI
        uses: loft-sh/setup-vcluster@main
      # Step 2: Login to vCluster Platform
      - name: Login to vCluster Platform instance
        env:
          LOFT_URL: ${{ secrets.VCLUSTER_PLATFORM_URL }}
          ACCESS_KEY: ${{ secrets.VCLUSTER_ACCESS_KEY }}
        run: |
          vcluster platform login $LOFT_URL --access-key $ACCESS_KEY
      # Step 3: Delete vCluster
      - name: Delete vCluster
        env:
          NAME: pr-${{ github.event.pull_request.number }}
        run: |
          vcluster platform delete vcluster $NAME --project default

How It Works

A developer creates a PR to do the feature changes.

‎With a small change the developer has raised a PR and now needs to add a test label.

As soon as the label is added the GitHub actions kicks off

In the vCluster platform cloud instance you will be able to see the cluster getting created and the application will be deployed.

The Action is completed and pr14.vcluster.tech is created as part of ingress.

The application is accessible at http://pr<PR_NUMBER>.vcluster.tech.

As you can see the latest changes made by the developer are deployed.

Cleanup:

Upon PR merge or label removal, the ephemeral vCluster is automatically deleted.

After merging, the cleanup action is triggered, which will clear the virtual cluster.

Conclusion

Ephemeral PR environments using vCluster simplify testing, reduce resource usage, and provide a seamless developer experience. By combining vCluster with GitHub Actions, you can achieve an automated and efficient workflow for testing PRs.

Check out the demo repository and give it a try! 🚀

Let me know your thoughts or if you face any challenges while implementing this.

With Kubernetes adoption continuing its upward trend, the real challenge for organizations today is not just adopting Kubernetes but managing it at scale. The widespread cluster sprawl, where companies create separate clusters for each team, environment, or workload, has led to escalating operational complexity and rising costs. According to the CNCF, over 70% of organizations report Kubernetes over-provisioning as a major source of cloud spend. This makes efficient multi-tenancy a necessity rather than a luxury.

Let’s explore how shared platform stacks and internal Kubernetes platforms are shaping the future of multi-tenancy in 2025.

What is Multi-Tenancy in Kubernetes?

Multi-tenancy means dividing a Kubernetes cluster into multiple isolated environments so that different teams or applications can share infrastructure while maintaining security, autonomy, and fair resource usage.

To understand this better, let’s take an analogy where you are looking for an accommodation:

• Renting an entire house gives you full control but comes with high maintenance costs — similar to having a dedicated Kubernetes cluster per team or application.

• Renting an apartment in a shared building gives you personal space but reduces overhead, as maintenance is handled collectively where you have shared access to facilities like elevators, swimming pool, park etc., this is how multi-tenancy works in Kubernetes.

Instead of spinning up an entirely new Kubernetes cluster for every team, organization, or workload, you partition a single cluster into multiple isolated environments.

The three key pillars of true multi-tenancy are:

1. Isolation — Ensuring security boundaries between tenants.

2. Fair Resource Usage — Preventing noisy neighbor issues.

3. Tenant Autonomy — Allowing teams to self-manage workloads independently.

Traditional Multi-Tenancy Approaches & Their Limitations

Natively within Kubernetes, there is a concept of namespaces, which is useful as many resources can be scoped to a namespace to create some level of isolation.

• Workload isolation can be achieved to a certain extent by using pod security standards and preventing privileged access with custom policy engines like Kyverno or jsPolicy. Additionally, you can define a well-structured network policy to restrict traffic to and from pods. When different teams have only namespace-level isolation, you may want to prevent them from communicating with each other while still allowing them to interact with the Kubernetes API. An example of this scenario can be as follows:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-policy
  namespace: tenant-1
spec:
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
            - 100.64.0.0/10
            - 127.0.0.0/8
            - 10.0.0.0/8
            - 172.16.0.0/12
            - 192.168.0.0/16
        - namespaceSelector:
            matchLabels:
              tenant: tenant-1
    - ports:
        - port: 53
          protocol: UDP
        - port: 53
          protocol: TCP
    - ports:
        - port: 443
        - port: 8443
      to:
        - ipBlock:
          cidr: ${KUBE_API}/32

• For managing resource usage, you can use Kubernetes objects like ResourceQuota to define the limit of resources that can be created within a cluster. You can also add LimitRange to set CPU and memory limits.

While these namespace-level resources help create some isolation, achieving true multi-tenancy is still challenging due to several factors:

• It becomes difficult as the number of tenants increases.

• How do you distribute different kubeconfigs per team?

• How is cluster-level resource access, such as CRDs, managed?

• Is resource sharing still an issue?

• How do you handle different cluster versions?

• What about different versions of an application?

• There is still a single control plane and a single state for the cluster.

Yes, multi-tenancy is hard if we rely solely on native Kubernetes constructs. Even with these measures, automating the entire process instead of manually defining everything is a major challenge.

How vCluster Enables True Multi-Tenancy

vCluster is an open-source tool that helps you create virtual Kubernetes clusters, each with its own control plane components and cluster state, in an automated way.

When you create a virtual machine in your cloud account, you gain full access to that virtual machine, but it is actually a slice of physical hardware in a data center. Similarly, a virtual cluster is a slice of a Kubernetes cluster, you have full access to it and complete ownership, but ultimately, it is still a part of a larger Kubernetes cluster.

How does vCluster work?

Instead of managing multiple Kubernetes clusters, you can now have a single Kubernetes cluster and use the vCluster CLI to create virtual clusters. These virtual clusters can reuse the host cluster’s resources, such as Cert Manager, NGINX Ingress Controller, Vault, and more. Each virtual cluster will have its own independent kubeconfig file, allowing teams to deploy their workloads independently. This approach is more secure than namespace-based isolation because each virtual cluster has its own control plane and state (with options like SQLite, embedded etcd, or external etcd)

With vCluster Enterprise, organizations can also gain features like multi-cluster tenancy, enhanced security policies, and automated tenancy provisioning.

The Evolution of Multi-Tenancy: Shared Platform Stacks & Internal Kubernetes Platforms (IKPs)

1. Shared Platform Stack: The Key to Efficiency

Imagine three teams: A, B, and C, each needing their own Kubernetes cluster. As administrators, we create three separate Kubernetes clusters. By default, a newly created cluster only runs the essential components needed for Kubernetes itself, such as the control plane components, the cloud controller manager etc.

Now, if all three teams need to deploy applications with HTTPS support, the typical approach is to install an Ingress Controller and cert-manager. Each team then creates Deployments, Services, Ingress, and Certificate objects. However, since these components need to be installed on every cluster separately, this results in duplicate resources.

This duplication problem also exists in multi-tenancy. One of the biggest challenges in Kubernetes multi-tenancy is the shared platform stack. Ideally, we should be able to reuse resources from the host cluster instead of installing cert-manager and an Ingress Controller in every new cluster.

The easiest way to solve this problem is by using virtual clusters. With vCluster, you can define in the cluster configuration file which resources should be synced from the host cluster, allowing multiple tenants to share platform resources. This optimizes resource utilization and eliminates unnecessary duplication.

This concept of a shared platform stack in a multi-tenant Kubernetes environment using virtual clusters helps organizations efficiently manage resources and is crucial when you are creating an internal Kubernetes platform.

2. Internal Kubernetes Platforms (IKPs)

We believe that an Internal Developer Platform (IDP) is evolving, with Kubernetes becoming the de facto choice for these platforms. Kubernetes is a technology well-suited for building platforms, and if you are developing an IDP in 2025 and beyond, you will or should be leveraging Kubernetes.

This is why we believe the shift is towards an Internal Kubernetes Platform (IKP), where multi-tenancy will play a crucial role, and vCluster will be at the center.

With vCluster integrated alongside your other cloud-native tooling, you can efficiently provision and manage Kubernetes clusters for your teams, making Kubernetes more accessible while maintaining governance and control.
IKPs ensure that tenants don’t need to deal with raw Kubernetes, instead, they receive a pre-configured platform tailored to their needs.

We’d love to hear your thoughts on IKPs as well!

Future of Multi-Tenancy in Kubernetes

Organizations are moving towards:

1️⃣ Standardized Shared Platform Stacks — Providing pre-configured Kubernetes environments.
2️⃣ IKPs for Developer Self-Service — Offering Kubernetes as a managed service within organizations.|
3️⃣ vCluster & Virtualized Control Planes — Reducing cluster sprawl while maintaining autonomy.

Multi-tenancy is no longer just about namespaces or virtual clusters — it’s about creating an internal Kubernetes ecosystem that allows teams to be productive while keeping infrastructure efficient and manageable.

Throughout March, we’re hosting a Multi-Tenancy March series, featuring webinars, deep dives, and hands-on sessions to explore best practices for Kubernetes multi-tenancy. We will be conducting a hands-on workshop on March 6th, where we will demonstrate this in action, and you’ll have the opportunity to try it out alongside us.

Register for the webinar here

Join the vCluster Slack to stay updated!

But does it really work? 🤔 In this blog, let’s put Gordon to the test with real-world scenarios and see if it’s actually useful

🔹 What is "Ask Gordon"?

Think of Gordon as a smart AI assistant built into Docker. Instead of searching the internet or reading documentation, you can ask Gordon questions directly in the Docker CLI or Docker Desktop UI.

For example, if a container crashes, instead of Googling for answers, you can simply ask:

docker ai "Why is my container crashing?"

And Gordon will analyze logs and suggest solutions. Pretty cool, right?

Now, let’s test out some scenarios and see if it actually works.

🔧 Setting Up Gordon

Before we test, let’s quickly enable Gordon:

1. You need to download Docker Desktop version 4.38 or later. You can download it from here.

2. This is how you can enable Ask Gordon:

- After signing in to your Docker Account, enable the Docker AI feature:

- Open the Settings view in Docker Desktop.

- Navigate to Features in Development.

- Check the Enable Docker AI checkbox.
The Docker AI terms of service agreement are displayed. You must agree to the terms before you can enable the feature. Review the terms and select Accept and enable to continue.
- Select Apply & restart

You can now chat with Gordon via:

• CLI → docker ai "<your question>"

• Docker Desktop UI → Click the ✨ icon in various places

Alright, now let’s run some real-world tests.

Let’s say you click “How do I run Redis“, you will get something like below:

🚀 Testing Gordon’s Capabilities

If you are using shell, then you must know these commands.

🛠 Test 1 : Dockerfile issues

I asked Gordon to check issues in my Dockerfile:

#Base image
FROM ubuntu:latest
#Install dependencies
RUN apt-get update && apt-get install -y 
curl 
wget 
python3 
python3-pip 
vim
#Copy application code
COPY . /app
#Set working directory
WORKDIR /app
#Install Python dependencies
RUN pip3 install -r requirements.txt
#Expose application port
EXPOSE 8080
#Start the application
CMD ["python3", "app.py"]

Here comes the output:

saloni@Salonis-MacBook-Air ~ % docker ai

>>> check my dockerfile for issues                                              

    • Sending Dockerfile ✔️

 Your Dockerfile looks functional and should work for most basic use cases. 
However, there are a few potential issues and improvements you can consider:


  ### Issues and Suggestions
    1. Using latest tag for the base image

Using ubuntu:latest is not recommended for production environments because 
the latest tag can change over time, potentially introducing breaking changes. 
It's better to specify a specific version, e.g., ubuntu:20.04.
2. Combining RUN commands

  The RUN command for installing dependencies can be optimized to reduce the number 
of layers in the image. 
For example:


         RUN apt-get update && apt-get install -y \
            curl \
            wget \
            python3 \
            python3-pip \
            vim && \
            apt-get clean && rm -rf /var/lib/apt/lists/*

  This ensures that temporary files created during the installation process are cleaned up, 
  reducing the image size.

  3. No explicit Python version

  The python3 and python3-pip packages will install the default Python version for the 
Ubuntu base image. If your application requires a specific Python version, you should 
explicitly install it.

 4. No non-root user

  The container runs as the root user by default, which can be a security risk. Consider 
adding a non-root user to run the application:
RUN useradd -m appuser
USER appuser

5. No health check

 Adding a HEALTHCHECK instruction can help monitor the health of the container. For example:
HEALTHCHECK --interval=30s CMD curl --fail http://localhost:8080 || exit 1

6. No .dockerignore file
Without a .dockerignore file, unnecessary files (e.g., .git, local environment files) might 
be copied into the image. Create a .dockerignore file to exclude such files:

 .git
  pycache
  *.pyc
   .env

  7. Hardcoded port

  The application exposes port 8080, but this might not be configurable. Consider making the 
   port configurable via an environment variable.

  8. No entrypoint script
  If your application requires environment setup or initialization, consider using an 
  entrypoint script instead of directly running the application in the CMD.

  ### Final Optimized Dockerfile
  Here’s an optimized version of your Dockerfile:

   # Base image
    FROM ubuntu:20.04

  # Install dependencies
    RUN apt-get update && apt-get install -y \
        curl \
        wget \
        python3 \
        python3-pip \
        vim && \
        apt-get clean && rm -rf /var/lib/apt/lists/*

  # Add a non-root user
    RUN useradd -m appuser
    USER appuser

  # Copy application code
    COPY . /app
 # Set working directory
    WORKDIR /app
 # Install Python dependencies
    RUN pip3 install --no-cache-dir -r requirements.txt
 # Expose application port
    EXPOSE 8080
 # Add health check
    HEALTHCHECK --interval=30s CMD curl --fail http://localhost:8080 || exit 1
 # Start the application
    CMD ["python3", "app.py"]

Let me know if you need further clarification!

So it gave us the missing pieces form my Dockerfile and gives the optimised Dockerfile. This is neat as you can directly do this in your cli form where you are running the docker commands.

🛠 Test 2 : Fixing a Broken Container

Scenario:

I have a Node.js container, but when I start, it crashes immediately.

echo 'console.log(require("express"))' > app.js
cat <<EOF > Dockerfile
FROM node:18
WORKDIR /app
COPY app.js .
CMD ["node", "app.js"]
EOF
docker build -t crash-node .

docker run --name crash-node -d crash-node

docker ps -a | grep crash
9ef75103d838   crash-node                            "docker-entrypoint.s…"   13 seconds ago   Exited (1) 12 seconds ago                               crash-node

Now let’s ask Gordon on why my container is failing

docker ai "Why is my container crash-node crashing?"

Response:

To diagnose why your container named crash-node is crashing, you can follow these steps:

    1. Check Container Logs:
  Use the docker logs command to view the logs of the crashing container. This can provide insights into 
  any errors or issues that occur just before the crash.

          docker logs crash-node

    2. Inspect Container Status:
  Use docker ps -a to check the status of the container. This can help determine if the container is 
  exiting with a specific error code.

          docker ps -a

    3. Use Docker Compose Logs (if applicable):
  If you are using Docker Compose, you can use the following command to view logs for all services, which 
  might give you more context:

          docker compose logs crash-node

    4. Check Resource Constraints:
  Ensure that your container is not running out of memory or other resources. You can inspect resource 
  usage with:

          docker stats crash-node

    5. Review Docker Daemon Logs:
  If the container logs do not provide enough information, you can check the Docker daemon logs for any 
  system-level issues. On Linux, you can use:

          journalctl -u docker.service

    6. Inspect Dockerfile and Configuration:
  Review your Dockerfile and any configuration files (like docker-compose.yml) for potential 
  misconfigurations or errors.

    7. Check for Known Issues:
  Refer to the Docker known issues 
  (https://docs.docker.com/desktop/troubleshoot-and-support/troubleshoot/known-issues/) to see if there 
  are any related issues that might be affecting your container.

  By following these steps, you should be able to gather more information about why your crash-node 
  container is crashing. If you find specific error messages or logs, feel free to share them for more 
  targeted assistance.

Let’s check the logs

docker logs crash-node
node:internal/modules/cjs/loader:1143
  throw err;
  ^

Error: Cannot find module 'express'
Require stack:
- /app/app.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1140:15)
    at Module._load (node:internal/modules/cjs/loader:981:27)
    at Module.require (node:internal/modules/cjs/loader:1231:19)
    at require (node:internal/modules/helpers:177:18)
    at Object.<anonymous> (/app/app.js:1:13)
    at Module._compile (node:internal/modules/cjs/loader:1364:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1422:10)
    at Module.load (node:internal/modules/cjs/loader:1203:32)
    at Module._load (node:internal/modules/cjs/loader:1019:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:128:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ '/app/app.js' ]
}

Node.js v18.20.7

Now, let’s ask Gordon again

docker ai "docker logs crash-node
node:internal/modules/cjs/loader:1143
  throw err;
  ^

Error: Cannot find module 'express'
Require stack:
- /app/app.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1140:15)
    at Module._load (node:internal/modules/cjs/loader:981:27)
    at Module.require (node:internal/modules/cjs/loader:1231:19)
    at require (node:internal/modules/helpers:177:18)
    at Object.<anonymous> (/app/app.js:1:13)
    at Module._compile (node:internal/modules/cjs/loader:1364:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1422:10)
    at Module.load (node:internal/modules/cjs/loader:1203:32)
    at Module._load (node:internal/modules/cjs/loader:1019:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:128:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ '/app/app.js' ]
}

The error indicates that the express module is missing in your Node.js application. This happens 
  because the express package is not installed in your Docker container

The main point here is that express is missing!!

so we can just update the Dockerfile as below

FROM node:18
WORKDIR /app
COPY app.js .
RUN npm install express
CMD ["node", "app.js"]

Now when you build, run again and check the logs you will see the logs.

docker logs 154d3d2759a8
[Function: createApplication] {
  application: {
    init: [Function: init],
    defaultConfiguration: [Function: defaultConfiguration],
    lazyrouter: [Function: lazyrouter],
    handle: [Function: handle],
    use: [Function: use],
    route: [Function: route],
    engine: [Function: engine],
    param: [Function: param],
    set: [Function: set],
    path: [Function: path],
    enabled: [Function: enabled],
    disabled: [Function: disabled],
    enable: [Function: enable],
    disable: [Function: disable],
    acl: [Function (anonymous)],
|
|
|
.......

All this is done while staying in the same CLI and not going anywhere else which is the coolest part IMO.

Conclusion: Is Gordon Worth Using?

I have used Gordon for various purposes, in the end its the fine tuned and trained AI agent on Docker documentation which is a good thing as it will have the latest information as compared to other LLM’s out there. But for the simple errors, it should just give smaller outputs with the fix and then explain when the user asks to explain the action. It’s in beta and new so we can excuse for the generic answers it gives at times but if we want people to not use chatgpt or similar LLM’s then it should provide concise answers, to the point issues and also like github copilot it should be able to provide instant feedback when we are actually typing in commands.

What do you think about Docker Gordon? Are you using it already?

If you use Docker Desktop, definitely give Gordon a shot. It won’t replace a seasoned engineer, but it can make life a bit easier, especially for debugging and automation.

💡 Try it out and let me know your thoughts! What’s the weirdest question you asked Gordon? 😆👇

What is Docker Desktop?

Docker Desktop is a cross-platform application available for Windows, macOS, and Linux that provides an easy-to-use interface to manage your Docker containers and images. It combines Docker Engine, Docker CLI, Docker Compose, and Kubernetes (optional) into a unified solution, enabling seamless development and testing of containerized applications on your local machine.

For installation of Docker Desktop, you can refer to my previous post.
https://blog.kubesimplify.com/docker-captain-journey

Key Features of Docker Desktop

1. Cross-Platform Support

Docker Desktop works on Windows, macOS, and Linux, making it a versatile tool for developers across different operating systems. It automatically configures and integrates with the host system, eliminating the need for complex setup processes.

2. Built-in Kubernetes Support

Docker Desktop includes a lightweight, single-node Kubernetes cluster for developers working with Kubernetes. This allows users to deploy, test, and manage Kubernetes workloads locally without needing a separate cluster.

3. Docker Compose

With Docker Compose integrated you can define multi-container applications in a simple YAML file and deploy them using a single command. This is particularly useful for microservices architectures.

4. Resource Controls

Docker Desktop provides an intuitive interface to allocate system resources like CPU, memory, and disk space for Docker containers, ensuring optimal performance without overloading your machine.

5. Image Management

The Image view in the Docker Desktop Dashboard makes it easy to manage container images. You can pull images from Docker Hub, inspect image details, and run images as containers. Additionally, the Dashboard helps you clean up unused images to free up disk space and provides a summary of image vulnerabilities, enabling proactive security management.

6. Volume Management

The Volumes view offers a streamlined way to manage Docker volumes. You can create, delete, and inspect volumes, as well as view which containers are using them, all from a centralized interface.

7. Builds View

Docker Desktop’s Builds view lets you inspect your build history and manage builders. This includes details of ongoing and completed builds, making it easier to track your workflows.

8. Notifications and Learning Center

Stay informed with Docker Desktop’s notification center, which provides updates about new releases, installation progress, and other alerts. The Learning Center offers in-app walkthroughs and resources to help you master Docker quickly.

9. Quick Search

Docker Desktop includes a Quick Search feature in the Dashboard, allowing you to locate containers , compose applications, images, volumes, or extensions with ease. For images, it provides options to pull, run, or view documentation, while for containers, you can perform actions like start, stop, or delete directly from the search results.

10. Docker Scout

Docker Scout is a tool that provides deep insights into container images, helping developers analyze dependencies, identify vulnerabilities, and improve image quality. It integrates seamlessly with Docker workflows, offering actionable recommendations and enabling policy enforcement to ensure secure, efficient, and compliant containerized applications. Perfect for managing your software supply chain!

11. 

    WebAssembly workloads

    Wasm in Docker enables running lightweight, fast WebAssembly (Wasm) workloads alongside Linux containers. To use Wasm, enable the container image store and turn on the Enable Wasm feature in Docker Desktop settings. Docker Desktop installs various Wasm runtimes, like Wasmtime and WasmEdge, to support Wasm workloads

    

    .

12. Docker Extensions

    Docker Desktop Extensions are add-ons that enhance Docker Desktop by integrating tools like security scanners, monitoring, and debugging directly into its UI. They simplify workflows, improve productivity, and allow developers to build and share custom extensions. You can explore and install them via the Docker Extensions Marketplace.

13. Dev Environments

    A Dev Environment in Docker Desktop lets developers quickly set up and share reproducible environments with all tools, dependencies, and configurations. It ensures consistency, simplifies onboarding, and eliminates "it works on my machine" issues, enabling seamless collaboration.

14. Ask Gordon

    Ask Gordon is Docker’s AI-powered assistant, currently in Beta, designed to streamline workflows in Docker Desktop and the CLI. It provides contextual, actionable insights by understanding your local setup, including Dockerfiles, containers, and applications. With features like identifying vulnerabilities and optimizing Dockerfiles, Ask Gordon helps make Docker's ecosystem more intuitive and efficient.

Exploring Docker Desktop

Docker Desktop Dashboard

The Docker Desktop Dashboard serves as your command center. Here are the key views available:

1. Containers View: Provides a runtime view of all your containers and applications, allowing you to manage their lifecycle, inspect logs, and perform other common actions directly from your machine.

2. Images View displays a list of local images, lets you pull images from Docker Hub, run images as containers, and clean up unused images. If you are logged in, you can also view images shared by your organization on Docker Hub.

3. Volumes View: Displays Docker volumes, allowing you to easily manage their lifecycle.

4. Builds View: Shows your build history, ongoing builds, and completed builds.

Docker Menu

Doing Stuff with Docker Desktop

To do everything within the Docker Desktop, you can enable the terminal form within the Docker Desktop to interact with the host machine.

1. Building Custom Images

Let’s say you want to build an image for your python flask application.

Create a simple app.py as below:

from flask import Flask

app = Flask(__name__)

@app.route('/')
def home():
    return "Hello, Dockerized Python App!"

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

Requirements (requirements.txt)

We will have the Dockerfile install the dependencies from requirements.txt, create a requirements.txt file with the following content:

flask

This will ensure that Flask is installed when building the Docker image.

Create a Dockerfile to define a custom image:

FROM python:3.9-slim
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
CMD ["python", "app.py"]

Build and run the image:

docker build -t my-python-app .
docker run -p 5001:5000 my-python-app

Then, open your browser and visit `http://localhost:5001` to see the output.

2. Using Docker Desktop with Kubernetes:

Enable Kubernetes in Docker Desktop settings and see the cluster is running and the context is switched to docker-desktop.

Then, create a Kubernetes ‘deployment.yaml’:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80

Apply the configuration:

kubectl apply -f deployment.yaml

Verify the deployment:

kubectl get deployment
kubectl get pods

You can see that you have a local Kubernetes experience that you can use to test out Kubernetes for your development purposes.

3. Exploring Extensions

Docker Desktop’s Extensions Marketplace allows you to add new functionalities. From CI/CD integrations to security scanners, you can install extensions with a single click and extend the capabilities of Docker Desktop. In the below example, I searched for the Lens extension to visualize Kubernetes.

Conclusion

Docker Desktop is an indispensable tool for developers and DevOps professionals. It simplifies the container lifecycle, bridges the gap between local and production environments, and offers powerful features like Kubernetes integration, resource management, and extensions. Whether building your first containerized app or managing complex microservices, Docker Desktop is your go-to solution for local development.

Start exploring Docker Desktop today and transform how you build, ship, and run applications!

(This blog is based on the latest video on our YouTube channel, "DevOps Tools 2025")

1. Building Secure Base Images

The journey begins with building secure and compliant base images. At Kubesimplify, we recommend using BuildSafe, a tool designed to simplify this process while ensuring compliance and a developer-friendly experience.

BuildSafe, built on Nix, helps you build 0 CVE base image yourself, provide you with higher quality build time SBOM out of the box and also let you achieve higher SLSA levels and even go beyond.

2. CI/CD Pipelines

Automation lies at the heart of DevOps, and effective CI/CD pipelines are key to operational efficiency. We emphasize a combination of:

• GitHub Actions and ArgoCD for modern CI/CD. You can check out this project where we have built and shown everything for a CI\CD pipeline.

  %[https://www.youtube.com/watch?v=kCWAwXFnYic&t=46s]

• Argo CD is a declarative GitOps continuous delivery tool for Kubernetes. It synchronizes Kubernetes resources with a desired state defined in a Git repository, enabling automated deployments and real-time drift detection. Argo CD supports multi-cluster environments and application rollbacks and integrates with Helm, Kustomize, and plain YAML manifests to manage Kubernetes workloads effectively.

Couple more tools highlighted in this video is:

• Dagger is a tool for writing pipelines in programming languages like Go, Python, and TypeScript. It offers flexibility with local and CI-based runs, making it a standout for modern workflows.

• Kargo: Created by the Argo team, it streamlines multi-stage application promotion using GitOps principles, removing the need for custom automation or CI pipelines. Kargo integrates seamlessly with Argo CD, automating progressive rollouts to improve efficiency, safety, and visibility across the application lifecycle.

3. Infrastructure as Code (IaC)

Managing infrastructure has never been easier with tools that provide powerful abstractions and flexibility. Kubesimplify recommends:

• Crossplane: Crossplane is an open-source framework that enables infrastructure and application management using Kubernetes-native declarative APIs. It extends Kubernetes to manage resources like cloud infrastructure, databases, and services through Custom Resource Definitions (CRDs). Crossplane supports multi-cloud environments and infrastructure-as-code practices and integrates seamlessly with existing Kubernetes workflows, providing a unified way to manage both infrastructure and application. The feature I love the most from crossplane is - crossplane composition. You can check out this video that I did with Dan on Crossplane compostion deep dive.

• Pulumi: Pulumi is an open-source infrastructure-as-code (IaC) tool that allows you to define, deploy, and manage cloud infrastructure using familiar programming languages like Python, TypeScript, Go, and C#. It supports a wide range of cloud providers and on-premises systems. Pulumi enables developers and DevOps teams to write infrastructure as real code, fostering better integration, testing, and reusability while simplifying infrastructure management across diverse environment.

• OpenTofu: OpenTofu is an open-source infrastructure such as a code (IaC) framework that helps define, provision, and manage cloud infrastructure using declarative configuration files. It supports multiple cloud providers and on-premises environments, enabling reproducible and automated infrastructure deployments. As a community-driven fork of Terraform, OpenTofu promotes openness and extensibility, empowering users with flexibility for modern infrastructure management.

Along with that some still use Ansible that remains a reliable option.

4. Kubernetes Package Management

Managing additional tooling on Kubernetes clusters requires robust package management. While Helm remains a popular choice, Glasskube is our recommended tool for 2025. It simplifies:

• Dependency management.

• Automatic CRD updates.

• Kubernetes version compatibility testing.

Glasskube’s focus on lifecycle management and CLI-based intuitive updates makes it an excellent choice for managing Kubernetes packages.

5. Observability

Observability is critical for maintaining operational excellence. While tools like Prometheus, Grafana, and Jaeger are well-established, Kubesimplify highlights:

• Signoz: SigNoz is an open-source observability platform for monitoring and troubleshooting applications. It provides metrics, logs, and traces in a single interface, enabling developers to quickly analyze application performance and detect issues. Built to focus on simplicity and scalability, SigNoz integrates with OpenTelemetry and supports popular storage backends like ClickHouse. It serves as a cost-effective alternative to proprietary observability solutions like Datadog or New Relic.

• OpenObserve: OpenObserve is an open-source observability platform that provides a unified solution for logs, metrics, and traces. It is designed for high performance, scalability, and cost-efficiency, making it suitable for modern cloud-native applications. OpenObserve simplifies monitoring and troubleshooting by offering a single interface for observability data, helping teams ensure system reliability and performance.

Both tools are great choices for monitoring, logging, and tracing in a Kubernetes environment.

6. Security and Compliance

Security is a non-negotiable aspect of DevOps. Kubesimplify suggests:

• BuildSafe: BuildSafe is a tool designed to secure the software supply chain by enabling organizations to create tamper-proof, 0-CVE (zero known vulnerabilities) artifacts compliant with government regulations. It focuses on generating high-quality SBOM’s, providing path for secure builds that are developer-friendly and easy to integrate into existing workflows. BuildSafe emphasizes hermetic builds, high-quality SBOMs (Software Bill of Materials), and compliance, helping organizations reduce risks associated with supply chain attacks.

• Trivy: Trivy is a security scanner for identifying vulnerabilities, misconfigurations, and sensitive data in containers, Kubernetes, IaC templates, and repositories. It supports a wide range of formats, integrates seamlessly into CI/CD pipelines, and provides detailed reports to enhance application and infrastructure security. Trivy is widely used for its speed, simplicity, and effectiveness in ensuring secure deployments.

• Kubescape: Kubescape is a Kubernetes security platform by ARMO, providing end-to-end protection across the development and runtime lifecycle. It features shift-left security, runtime threat detection, cluster scanning, YAML/Helm validation, compliance with frameworks like NSA-CISA and MITRE ATT&CK, and multi-cloud support. A CNCF sandbox project, Kubescape ensures a robust security posture for Kubernetes environments.

• Falco: Falco is a runtime security tool for Kubernetes and cloud-native environments. It detects anomalous behavior, potential threats, and policy violations by monitoring system calls in real time. With predefined and customizable rules, Falco provides actionable alerts for suspicious activities, helping secure workloads and maintain compliance in dynamic, containerized environments. It is a CNCF graduated project.

These tools collectively ensure a robust security posture throughout the DevOps lifecycle.

7. Cost Optimization

Kubernetes cost optimization is a growing concern, and tools like Cast.AI offer intelligent auto-scaling to reduce expenses. Alongside cost reduction, vCluster is pivotal in enabling multi-tenancy, sustainability, and platform engineering goals.

8. WebAssembly and AI Workloads

WebAssembly continues to make waves in the industry. Tools like SpinKube and WasmCloud simplify deploying WebAssembly applications on Kubernetes.

For AI workloads, Kubeflow remains a leading choice, offering a complete lifecycle solution from training to inference using components like KServe and pipelines. Kubernetes is rapidly becoming a preferred platform for running AI agents, and tools like Argo Workflows further enhance these capabilities.

Kubesimplify’s Final Thoughts

At Kubesimplify, our mission is to break down the complexities of cloud-native technologies and empower you to make informed decisions. These tools represent the forefront of innovation in 2025, simplifying workflows while addressing real-world challenges.

We’d love to hear from you! What tools are part of your DevOps journey in 2025? Share your thoughts, and let’s grow together as a community.

A big thanks to the CNCF and open-source contributors for driving these innovations. Stay tuned to Kubesimplify for more insights, tutorials, and updates from the cloud-native ecosystem.

Discovering Docker in 2019: A Spark of Interest

It all began back in 2019, when I was working at SAP Labs in Bangalore. As someone passionate about exploring new technologies, I started attending local meetups to connect with like-minded professionals and learn from their experiences. It was during one of these meetups that I first came across the term Docker. The concept of containerization and its potential to transform software development immediately got my attention.

Eager to dive deeper, I started learning Docker through various online platforms. I explored tutorials, blogs, and documentation to understand its fundamentals and practical applications. This newfound knowledge not only enhanced my skills but also ignited a desire to share what I had learned with others.

Hands-On Experience: Working on a Docker Project at SAP Labs

While at SAP Labs, I got the opportunity to work on a Docker-based project in a production environment. This was a game-changer for me, as it allowed me to gain hands-on experience with Docker in real-world scenarios. Working on the project helped me understand the challenges and nuances of using containerization at scale, such as deploying containers in production, managing workloads, and ensuring high availability.

This experience not only strengthened my technical expertise but also gave me a practical perspective on Docker’s impact on enterprise environments.

Taking It to the Next Level: Earning the Docker Certified Associate (DCA)

As my interest in Docker grew, I decided to deepen my understanding by pursuing the Docker Certified Associate (DCA) certification. Preparing for this certification was challenging but rewarding. It provided me with a strong foundation in containerisation concepts, Docker commands, orchestration, and real-world applications. This certification confirmed my expertise and inspired me to explore further possibilities with Docker. I became DCA when the Enterprise business was acquired by Mirantis.

Organising India's Largest Docker Meetup: A Milestone in 2020

One of the most memorable milestones in my journey was organizing India's largest Docker meetup in January 2020. This was the first event of the year, and it was truly extraordinary. With over 550 attendees, it became a platform for professionals and enthusiasts to gather, share knowledge, and discuss the future of containerization.

From coordinating with speakers to managing logistics, the experience of organising such a large-scale event was both challenging and exhilarating. Seeing the community’s enthusiasm and engagement reaffirmed my commitment to contributing to the Docker ecosystem.

Keeping the Community Alive During COVID-19

When the world came to a standstill due to the COVID-19 pandemic, I saw an opportunity to keep the community spirit alive through virtual meetups. I hosted several online sessions on Docker, sharing insights, best practices, and use cases with developers worldwide. These sessions not only helped others learn but also deepened my own understanding as I prepared and answered questions from the community.

During this time, I also created a dedicated Docker playlist on my YouTube channel, Kubesimplify. This playlist became a go-to resource for developers to learn Docker step-by-step, from beginner concepts to advanced techniques. Additionally, I began writing blogs on Medium and Kubesimplify, covering practical Docker topics to reach an even wider audience.

Applying for Docker Captain: The Next Chapter

After years of learning, contributing, and engaging with the Docker community, I felt ready to take the next step: applying for the Docker Captain program. This recognition would not only validate my contributions but also provide me with a platform to reach more people and advocate for Docker in the global developer ecosystem.

Reflections on the Journey

Looking back, what began as curiosity at a meetup has grown into a passion for containerisation and community building. Docker has been more than just a technology for me, it has been a catalyst for personal growth, professional opportunities, and meaningful connections with developers worldwide.

As I continue this journey, I am excited to share more knowledge, build impactful communities, and explore the endless possibilities of containerisation. For anyone looking to start their Docker journey, my advice is simple: stay curious, contribute to the community, and never stop learning.

What’s Happening at Docker Inc.?

Before we talk about Docker Desktop, let me quickly highlight the key areas Docker Inc. is focusing on post the 2019 Mirantis acquisition of their enterprise business. Docker Inc. has been refining its offerings, and here are its major products:

1. Docker Desktop
   A local development environment enabling developers to efficiently build, share, and run containerized applications on their desktops. It integrates seamlessly with multiple developer tools and supports various programming languages. It also has WASM integration and you can build and run wasm OCI images too.

2. Docker Hub
   A cloud-based repository where developers can discover, share, and store container images. It acts as a central hub for managing container images and finding trusted content.

3. Docker Scout
   A tool designed to simplify the software supply chain by providing insights into container images, helping developers identify and address vulnerabilities. It makes finding CVE’s super simple in the container images.

Now that we’ve got a quick overview, let’s zoom in on Docker Desktop and learn how to get started.

Installing Docker Desktop

For this blog, I’ll focus on the installation process for macOS, as that’s what I use. If you’re on Windows or any other you can visit the official website and download as per the OS.

Steps to Install Docker Desktop on macOS:

1. Visit the Docker Desktop installation guide for macOS and click download for MAC

2. Download the installer and click on it to begin the installation process.

3. 

   Follow the on-screen instructions to complete the setup.

   

4. Once installed, open Docker Desktop, and you’re ready to start containerizing!

If you’re planning to use Docker Desktop for work, make sure to select the appropriate license during setup. Docker Desktop offers both personal and professional options, so choose based on your needs.

What’s Next?

In my upcoming blogs, I’ll dive deeper into Docker Desktop’s features, its integration with Kubernetes, and tips to optimise your containerised workflows. For now, I encourage you to explore Docker Desktop, try running your first container, and share your experiences in the comments below.

Kubernetes has revolutionized how we deploy and manage applications, offering unparalleled scalability and flexibility. However, with great power comes great complexity, and this complexity can often lead to excessive cloud costs if not managed effectively.

This blog delves into the journey from basic Kubernetes monitoring to advanced automation, highlighting how intelligent solutions like CAST AI can help organizations unlock significant cost savings without compromising performance or stability.

We’ll start by understanding why monitoring is the foundation of any cost optimization strategy. Next, we’ll explore the common challenges teams face when transitioning to automation for resource management. Finally, we’ll examine how CAST AI addresses these challenges and empowers organizations with its innovative solutions and real-world success stories.

Kubernetes Monitoring: The Foundation of Cost Optimization

Monitoring is the essential first step towards understanding and managing your Kubernetes environment. By providing real-time visibility into performance, security, and resource utilization, monitoring helps teams identify inefficiencies and opportunities for optimization.

Key Areas of Kubernetes Monitoring

1. Performance Monitoring
   Metrics like CPU usage, memory consumption, network traffic, and pod restarts help identify bottlenecks. This information allows teams to:

   • Optimize resource allocation.

   • Adjust application configurations for smoother operation.

   • Ensure that workloads are running efficiently.

2. Security Monitoring
   Monitoring user activity, API calls, and network traffic patterns can reveal suspicious behaviors and potential security breaches. Prompt detection and resolution of security issues prevent costly data leaks and system downtime.

3. Application Health Monitoring
   Metrics such as response times, error rates, and request throughput provide insights into application health. Proactive monitoring allows teams to address performance degradation before it impacts end users, ensuring a seamless experience.

While tools like Prometheus and Grafana are excellent at collecting and visualizing this data, many teams stop short of translating these insights into actionable measures, missing the opportunity to optimize their Kubernetes clusters effectively.

Missed Opportunities: The Case for Automation

Although monitoring provides valuable insights, acting on them manually can be time-consuming and error-prone. Automation is the next logical step where people should be doing resource optimization, cost optimization, yet many teams hesitate to adopt it for several reasons:

1. Fear of Disruption
   Teams worry that automation might introduce instability, especially in complex Kubernetes environments.

2. Lack of Expertise
   The rapidly evolving Kubernetes landscape can make it challenging for teams to implement and manage sophisticated automation solutions.

3. Resistance to Change
   Traditional practices and a reluctance to embrace new technologies often slow down the adoption of automation.

These hesitations lead to significant missed opportunities, such as the adopting of spot instances which is a discounted compute option provided by cloud providers. While spot instances offer substantial cost savings, their ephemeral nature requires robust automation to handle disruptions effectively.

CAST AI: Unlocking the Power of Automation

CAST AI bridges the gap between monitoring and automation, providing a comprehensive platform that optimizes Kubernetes environments for cost, performance, and security.

Key Features of CAST AI

1. Cost Monitoring and Insights
   CAST AI offers detailed dashboards that break down resource usage, trends, and forecasts. Unlike traditional tools, it goes a step further by providing actionable recommendations for cost savings.

2. Rebalancing
   CAST AI’s rebalancing engine minimizes resource fragmentation by intelligently placing pods across nodes. For example, instead of spreading workloads across three underutilized nodes, it consolidates them onto two nodes, reducing costs while maintaining high availability.

3. Workload Right-Sizing
   By analyzing historical data and real-time metrics, CAST AI dynamically adjusts CPU and memory allocations for workloads. This ensures optimal resource utilization without compromising performance.

4. Spot Instance Management
   CAST AI automates the use of spot instances with features like:

   • Fallback Mechanisms: Automatically shifts workloads to on-demand nodes when spot instances are reclaimed.

   • Spot Diversity: Leverages a wide range of spot instance types to minimize disruption risk.

These features empower organizations to achieve cost reductions of up to 50%-80%, depending on their workloads and strategies.

Real-World Success Stories

CAST AI’s solutions have delivered tangible results for organizations across industries:

1. Eliminating Manual Kubernetes Upgrades
   A customer who spent two weeks annually upgrading Kubernetes versions across environments automated the process with CAST AI. This not only saved time but also boosted team morale, allowing engineers to focus on strategic initiatives.

2. Spot Instance Management During Peak Traffic
   Yotpo, a marketing software company, leveraged CAST AI’s spot instance automation during Black Friday. Despite high traffic, CAST AI ensured seamless scaling and uninterrupted operations, resulting in significant cost savings.

Beyond Cost Optimization: CAST AI’s Expanding Capabilities

CAST AI is continuously innovating to address broader Kubernetes challenges. Recent launches include:

• AI Enabler: A solution for leveraging AI to enhance Kubernetes operations.

• Kubernetes Security: A new product designed to improve container and cluster security.

These additions position CAST AI as a comprehensive platform for Kubernetes day-two operations, covering everything from cost optimization to security and AI-driven enhancements.

Embracing Automation for a Cost-Efficient Future

Over-provisioning remains a rampant issue in Kubernetes environments, with studies showing that up to 87% of resources are underutilized. By embracing automation, organizations can unlock significant cost savings, improve operational efficiency, and free up teams to focus on innovation.

As Giri from CAST AI aptly put it: “Don’t do manual work for resource optimization. We’re in 2024 - it’s time to automate.”

Ready to Get Started?

CAST AI offers a free trial to help teams explore its capabilities. Visit their website to set up a demo cluster and see how monitoring and automation can transform your Kubernetes operations.
With intelligent solutions like CAST AI, Kubernetes cost optimization is no longer a daunting task but an achievable goal. Start your journey today and unlock the full potential of automation.

Check out the entire Conversation on Kubesimplify Youtube channel

• Day 1: Artificial Intelligence (AI)

• Day 2: Security

• Day 3: Community

For anyone new to the cloud-native ecosystem, KubeCon is the largest and most influential conference dedicated to cloud-native technologies. This year, the 2024 edition was hosted in Salt Lake City, Utah. Preceding the main event was Rejekts, a two-day conference held on November 10 and 11. Rejekts offers a unique platform for deeper discussions and features talks that didn’t make it into the main KubeCon lineup. I had the opportunity to attend Rejekts in person and was thrilled to see Kubesimplify as a proud community and media partner for the event. The energy, the insightful conversations, and the chance to connect with incredible individuals made the experience unforgettable.

I also had the honor of speaking at KubeCon, delivering a talk titled "Cloud Native Sustainability Speedrun: Tools from Infrastructure to Application Level." It was an incredible experience to share insights into sustainability tools like Kepler, KubeGreen, and Cloud Carbon Footprint, and to demonstrate how cloud-native applications can incorporate sustainability at every level.

Day 1 Highlights: AI in Cloud-Native and Tackling Patent Trolls

The first day of KubeCon was a blend of excitement around Artificial Intelligence and a call to action against legal challenges facing the open-source community.

1. Scaling Kubernetes for Generative AI

Industry experts shared lessons learned while building Kubernetes clusters to support generative AI workloads. They tackled challenges like hardware failures, GPU scheduling optimization, and observability while leveraging CNCF projects to manage platforms for AI.

2. NVIDIA’s Contributions to AI and CNCF Projects
   Chris Lamb from NVIDIA discussed how the company uses and contributes to CNCF projects, showcasing the synergy between open source and AI advancements.

3. Patent Trolls and the Cloud Native Heroes Challenge
CNCF launched the Cloud Native Heroes Challenge, a patent troll bounty program in partnership with Unified Patents. This initiative aims to protect open-source projects from legal threats, allowing the community to contribute while earning rewards. Learn more here.

On Day 1, the blend of technological advancements in AI and the community’s resilience in tackling challenges showcased the collaborative spirit of KubeCon.

Day 2 Highlights: Security in the Cloud-Native Ecosystem
Day 2 focused on the critical topic of Security, reflecting the growing importance of safeguarding open-source ecosystems.

1. Envoy AI Gateway
A significant announcement was the introduction of the Envoy AI Gateway, the first CNCF AI project. Built on Envoy Proxy, this gateway offers scalable solutions for LLM access, unified APIs, and upstream authorization.

2. CNCF End User Awards
Adobe received the CNCF End User Award for contributing to 46 CNCF projects, including Kubernetes, Prometheus, and OpenTelemetry.

3. Announcing the 2024 Community Awards!

This year’s awards include:
- Lifetime Achievement award (new!)
- Top Committer
- Chop Wood Carry Water
- CNCF Lorem Ipsum (previously Documentarian)
- TAGGIE
- Lift and Shift (special)

4. New Certifications
   At KubeCon + CloudNativeCon North America, CNCF is also introducing three new project-specific certifications:

• Certified Backstage Associate (CBA)

• OpenTelemetry Certified Associate (OTCA)

• Kyverno Certified Associate (KCA)

Day 3 Highlights: Community and the Future of Cloud-Native

The final day celebrated the Community, emphasizing collaboration and envisioning the future of cloud-native technologies.

1. Kubernetes’ Ten-Year Journey

Kelsey Hightower reflected on a decade of Kubernetes, sharing stories of its evolution and celebrating the community, which has grown exponentially since the first KubeCon, which had 600 attendees in 2016.

2. Training African Technologists
CNCF announced a partnership with LF Education and Andela to train 20,000 African technologists in cloud-native basics, enabling them to earn certifications like KCNA and CKAD.

3. Graduated Projects
Congratulations to the newest CNCF Graduated projects:

• Cert-Manager

• Dapr

• KubeEdge

• Falco

Co-Located Events

KubeCon + CloudNativeCon started with various co-located events on Monday, November 11, 2024, each providing a deep dive into specific topics within the cloud-native ecosystem. Here's a quick rundown of a few of them:

• WasmCon Day 1

• ArgoCon Hosted by CNCF

• BackstageCon, Hosted by CNCF

• Cilium + eBPF Day Hosted by CNCF

• Cloud Native + Kubernetes AI Day Hosted by CNCF

• Observability Day Hosted by CNCF

• Platform Engineering Day Hosted by CNCF

• Cloud Native University Hosted by CNCF (Half Day)

• Data on Kubernetes Day Hosted by CNCF (Half Day)

• EnvoyCon Hosted by CNCF (Half Day)

• OpenFeature Summit Hosted by CNCF (Half Day)

• AppDeveloperCon Hosted by CNCF

These events set the stage for the main conference by fostering meaningful conversations and offering hands-on workshops for participants to expand their expertise. Save the Dates for Future KubeCons
Here are the upcoming KubeCon events to mark on your calendar:

• KubeCon + CloudNativeCon India 2024 | December 11–12 | Delhi, India

• KubeCon + CloudNativeCon Europe 2025 | April 1–4 | London, England

• KubeCon + CloudNativeCon China 2025 | June 10–11 | Hong Kong

• KubeCon + CloudNativeCon Japan 2025 | June 16–17 | Tokyo, Japan

• KubeCon + CloudNativeCon North America 2025 | November 10–13 | Atlanta, Georgia

• KubeCon + CloudNativeCon India 2025 | August 6–7 | Hyderabad, India

Final Thoughts

This KubeCon was particularly special for me as I recently became a CNCF Ambassador! With this new role, I had the privilege of attending the exclusive CNCF Ambassador breakfast, where I got to connect with other ambassadors and community leaders. It was an enriching experience to exchange ideas and learn more about contributing to the cloud-native ecosystem.

This time, I had my family with me → my husband, Saiyam, and our little one, Rushika. Believe it or not, this was Rushika’s third KubeCon! Thanks to the excellent daycare facility at KubeCon, balancing family and conference activities was seamless. Watching Rushika enjoy herself while I was immersed in the sessions and networking made this trip truly memorable.

Another highlight for me was finally being able to register for a professional headshot session. I’ve always missed this in previous events, but this time, I made it a point to grab the opportunity, and I’m thrilled with how it turned out!
Additionally, Saiyam and I teamed up with Shwetha Vohra to record an exclusive interview on platform engineering for our YouTube channel, Kubesimplify. The discussion was insightful, and I can’t wait to share it with you all soon!

I also love the concept of the job board at KubeCon. It is thoughtful, offering attendees a chance to explore job opportunities at the event. It was great to see companies actively seeking talent and participants being able to connect with potential employers directly.

What were your favorite moments from KubeCon 2024? Let’s discuss this in the comments!

One of the most effective strategies to reduce cloud spending is to use a mix of spot and on-demand instances. Spot instances are significantly cheaper, but they come with the risk of being terminated at any time, while on-demand instances provide the stability needed to keep your applications running smoothly.

On paper, the solution seems simple: combine spot and on-demand instances to get the best of both worlds—cost savings and reliability. However, the reality of managing pod placement across these different instance types is far from straightforward. Let’s explore the problem and how we tackled it.

The Problem: Managing Pod Placement in Mixed Instance Environments

As you begin to implement a mixed instance strategy in Kubernetes, you quickly run into challenges with pod placement. Kubernetes does provide tools for controlling where pods are deployed, but they’re often too simplistic or too rigid for the nuanced control you need.

Node Selectors allow you to direct Kubernetes to place a pod on a specific type of instance, such as a spot instance. But this method is binary—it either places the pod on a spot instance, or it doesn’t. There’s no middle ground, no balancing between instance types.

Affinity and Anti-Affinity Rules provide more control by allowing you to express preferences or requirements for pod placement. For example, you could set a rule that prefers spot instances but allows on-demand instances if no spot instances are available. However, as your cluster grows and your applications become more complex, these rules can become cumbersome to manage. The YAML configurations become lengthy and difficult to maintain, and the rules themselves can become conflicting or lead to unintended consequences.

Additionally, as clusters scale, maintaining even distribution of pods across instance types becomes a challenge. Without careful management, you could end up with too many pods on one type of instance, leading to inefficiencies or increased risk if those instances are interrupted.

This lack of fine-grained control and the complexity of managing pod placement in large, diverse clusters were the core problems we needed to solve.

The Solution: Leveraging Kubernetes Topology Spread Constraints

To address these challenges, we turned to Kubernetes Topology Spread Constraints (TSC), a feature that simplifies and streamlines the process of distributing pods across different topologies within a cluster. TSC allowed us to control pod placement with more nuance and flexibility, reducing the complexity of our configurations and improving our ability to manage mixed-instance environments.

MaxSkew is a critical component of this approach. By setting MaxSkew to 1, we ensured that pods are distributed as evenly as possible across both spot and on-demand instances. This prevents any single instance type from becoming overloaded, thereby improving the overall resilience and performance of our applications.

We also leveraged the Topology Key to distinguish between spot and on-demand instances. By using a node label such as "node.kubernetes.io/instance-type," we were able to clearly differentiate between the two, allowing Kubernetes to make informed decisions about where to place pods.

Here's an example of how this would look in practice using a Kubernetes Deployment:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: mypod
  labels:
    foo: bar
spec:
  replicas: 10
  selector:
    matchLabels:
      foo: bar
  template:
    metadata:
      labels:
        foo: bar
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: node.kubernetes.io/instance-types
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              foo: bar
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.1

In this example:

• maxSkew: 1 ensures even distribution of pods across the different instance types.

• The topologyKey is set to node.kubernetes.io/instance-type to distinguish between spot and on-demand instances.

• We use whenUnsatisfiable: DoNotSchedule to ensure strict adherence to the distribution rule, preventing pods from being scheduled if the constraints can’t be met.

whenUnsatisfiable became our fallback strategy, providing two options:

• DoNotSchedule enforces strict adherence to the constraints, ensuring that if the desired distribution cannot be achieved, the pod won’t be scheduled. This option is crucial for scenarios where balanced distribution is critical for application performance or reliability.

• ScheduleAnyway offers flexibility by allowing the scheduler to proceed with pod placement even if perfect distribution isn’t possible. This approach is particularly useful in situations requiring rapid scaling, such as during sudden traffic spikes, where ensuring availability is more important than maintaining an ideal distribution.

This solution has been around for a few years now, but the adoption hasn't quite caught on due to its lack of awareness.

Real-World Challenges: Limitations and Considerations

While Topology Spread Constraints offer a powerful solution, they are not without limitations. One significant challenge is that Topology Spread Constraints do not rebalance pods at runtime. They only apply when pods are initially scheduled. If the distribution of instances changes—such as when a spot instance is terminated—Kubernetes does not automatically rebalance the pods across the remaining instances. This can lead to uneven distribution over time, potentially undermining the benefits of using TSC.

Another challenge arises during node failures. When a node fails, Kubernetes will reschedule the affected pods, but it may not respect the original Topology Spread Constraints. This could result in pods becoming concentrated on fewer nodes, reducing the effectiveness of your mixed-instance strategy.

Conclusion: Making Kubernetes Work for You

Optimizing costs in Kubernetes is a challenging but essential task for any development team. While the complexities of managing mixed-instance environments can be daunting, tools like Topology Spread Constraints offer a path forward. By embracing these features and understanding their limitations, you can achieve a balance between cost efficiency and reliability, making Kubernetes work for you rather than against you.

In the end, it’s about finding the right tools and strategies to meet your specific needs. Whether you’re managing a small cluster or a large, complex environment, the key is to remain flexible and adaptable, continually refining your approach as your requirements evolve. With the right mindset and the right tools, you can optimize your Kubernetes deployments for both cost and performance, ensuring that your applications are always running at their best.