Running Nemotron 3.5 Lightning on DGX Spark

NVIDIA released Nemotron 3.5 Lightning today (August 11, 2026), and the pitch is simple: long-running agents spend most of their tokens on boring execution work (tool calls, validating outputs, formatting results), and you should not be burning frontier-model money on that. Lightning is the small, fast worker model for that layer.

NVIDIA specifically calls out DGX Spark as a deployment target. I have a Spark on my desk. So instead of quoting their charts, I pulled the model the hour it landed and measured it myself. This is a short post with what I measured, and one gotcha you should know about if you try it right away.

What it actually is #

The specs, verified against the model config on Hugging Face, not the press release:

The family resemblance to Nemotron 3 Super and Ultra is deliberate. Same hybrid Mamba-Transformer MoE recipe, same multi-token prediction training, just shrunk to a size where 3B active parameters means memory bandwidth stops being your enemy. On a bandwidth-bound box like the Spark (273 GB/s), the active parameter count is what decides your decode speed. That is the reason this model exists at this size.

NVIDIA's own positioning worth repeating: on PinchBench it scores 86% while completing 10,000 tasks about 30% faster than Qwen3.6 35B at similar accuracy (their blog says 30%, the launch tweet says 35%, I am going with the blog). They also claim up to 4x the output speed of similar-sized models. Those are NVIDIA's numbers, mine are below.

The gotcha: your Ollama is too old #

Ollama is a day-one launch partner, and the model is already in the library. But:

$ ollama pull nemotron-3.5-lightning:30b-a3b
Error: pull model manifest: 412:
The model you are attempting to pull requires a newer version of Ollama.

Support for the Nemotron 3.5 architecture landed in Ollama v0.32.9, released today, a few hours after the model itself. Anything older fails with that 412, and on launch day "older" included both my Spark's install (0.30.10) and the ollama/ollama:latest Docker image (still 0.32.6 when I tried it). The fix is to upgrade, then pull again.

curl -fsSL https://ollama.com/install.sh | sh
ollama pull nemotron-3.5-lightning:30b-a3b
ollama run nemotron-3.5-lightning:30b-a3b

By the time you read this, a plain upgrade is probably all you need. I mention it because if you searched that 412 error, this is why.

25GB download. The Ollama build is a Q4_K_M GGUF that ollama show reports as 32.9B parameters (the gap vs the marketing 30B is likely the MTP layer and embeddings being counted). Two things worth noticing in the model metadata: it ships with draft_num_predict 2, meaning Ollama is already using the baked-in multi-token prediction for speculative decoding out of the box, and it lists tools and thinking capabilities. Keep in mind this is not the NVFP4 checkpoint. More on that at the end.

The numbers #

All runs on my DGX Spark (GB10, 128GB unified memory, DGX OS), Ollama 0.32.9, temperature 0, measured via the API so the tok/s figures come from Ollama's own eval counters, 3 runs each.

First, the footprint. Cold load took 27.3 seconds, and ollama ps reports 26GB resident at 100% GPU. Interesting detail: Ollama loads this model with the full 262,144 token context window by default, and even at 256K context the whole thing plus KV cache room fits with roughly 86GB of the unified pool still available. On a 24GB card you would be making painful tradeoffs; here it just loads.

NAME                              SIZE     PROCESSOR    CONTEXT
nemotron-3.5-lightning:30b-a3b    26 GB    100% GPU     262144

The throughput numbers:

And the raw runs behind that table, straight from the API counters (repeat runs of the same long prompt hit Ollama's prompt cache, which is why I only count uncached first passes for prefill):

short run 1: prompt 39 tok @ 87.6 tok/s   | decode 500 tok @ 73.01 tok/s
short run 2: prompt 39 tok @ 918.5 tok/s  | decode 500 tok @ 71.90 tok/s
short run 3: prompt 39 tok @ 824.5 tok/s  | decode 500 tok @ 71.73 tok/s
long  run 1: prompt 8194 tok @ 2583.4 tok/s | decode 200 tok @ 38.36 tok/s (first-load outlier)
long  run 2: prompt cached                  | decode 200 tok @ 86.97 tok/s
long  run 3: prompt cached                  | decode 200 tok @ 85.38 tok/s
cache-busted: prompt 15820 tok @ 2654.7 tok/s | decode 200 tok @ 83.93 tok/s
agent run 1: decode 150 tok @ 85.62 tok/s
agent run 2: decode 150 tok @ 86.17 tok/s
agent run 3: decode 150 tok @ 85.94 tok/s

So: roughly 72 to 87 tok/s single-stream decode and about 2,600 tok/s prefill, sustained even with 8K to 16K tokens of context on the clock. An 8K-token prompt is fully ingested in just over 3 seconds. The one 38 tok/s decode reading happened immediately after the model's very first prefill and never reproduced; the cache-busted rerun confirms decode stays in the 80s with a fresh 15K-token prefill.

Two things surprised me:

Decode speed depends on what the model is generating. The short test (YAML plus prose) sat at 72 tok/s while the log-summary and JSON tool-call tests ran 84 to 87 tok/s. My best explanation is the built-in speculative decoding: the Ollama build ships with draft_num_predict 2, so the MTP head drafts ahead and predictable output (JSON, repetitive summaries) gets a higher acceptance rate than free-form prose. So the tok/s you get depends on the workload.

Agent-style calls come back fast. I gave it a tool-calling prompt (find why an nginx deployment is CrashLooping, respond with the next tool call as JSON). It reasoned for a few hundred tokens, then returned exactly this, end to end in 5.9 seconds:

{
  "tool": "run_command",
  "parameters": {
    "cmd": "kubectl get pods -n web"
  }
}

That is the right first step, and it came back as clean JSON with nothing extra around it. This is a reasoning model by default (Ollama reports the thinking capability), so budget for a couple hundred thinking tokens per call, at 86 tok/s that is about 3 seconds of overhead per agent step.

How it compares with other models on the same box #

I first reached for the numbers I measured back in May and the comparison looked flattering. Then I reran everything today, on the same Ollama 0.32.9 server, same prompt, same settings, and the picture changed. (Note: NVIDIA's PinchBench comparison is against Qwen3.6 35B; what I have locally is its predecessor, qwen3.5:35b-a3b, so treat these as class comparisons, not a re-run of NVIDIA's benchmark.)

The first thing this table shows has nothing to do with Nemotron: the runtime itself got faster. Qwen3.5 35B-A3B went from 52.7 to 78.1 tok/s on the same hardware since May, because Ollama now exploits its MTP head for speculative decoding too. If you are still quoting tok/s numbers from a months-old Ollama, they are stale. Mine were.

The second thing: through Ollama, raw decode speed against Qwen3.5 35B-A3B is basically a tie. On the identical prose prompt, Qwen was actually a touch faster (78 vs 72). NVIDIA's "up to 4x the output speed of similar-sized models" comes from the Artificial Analysis leaderboard, measuring hosted NVFP4 endpoints with the full draft-model stack; you do not see that 4x through Ollama today. Against its own big brother Nemotron 3 Super, the model it is meant to take execution work from, Lightning is a real 3.3x to 4x.

Where Lightning does win is token efficiency. I gave qwen3.5:35b-a3b the exact CrashLoop agent prompt from earlier, temperature 0. Both models produced the identical correct kubectl get pods -n web tool call:

Qwen thought four times longer to reach the same place. One prompt is not a benchmark, but it is exactly the behavior NVIDIA claims Lightning was trained for: their PinchBench pitch is 30% faster task completion at similar accuracy, a time-to-done argument rather than a tok/s argument. On this one task the agent step finished 4.4x sooner. For an agent doing thousands of steps a day, tokens per step matters more than tokens per second.

Reality check #

A few things to know before you use these numbers:

1. This is the GGUF, not NVFP4. The whole Lightning story on Blackwell hardware is the NVFP4 checkpoint with the dedicated kernels, plus the DSpark draft model that NVIDIA explicitly recommends for DGX Spark inference.
2. Single stream only. These are one-user interactive numbers. If you want aggregate throughput, that is a vLLM or TensorRT-LLM concurrency story (NVIDIA published deployment guides for both).
3. Launch-day software. Ollama support is hours old. Expect the numbers to move as kernels and the runtime settle. I will update if they move meaningfully.

There is one other public number to compare against: the NVIDIA forum benchmark of Nemotron 3 Nano Omni 30B-A3B, an architecturally similar 30B-A3B, at 56.96 tok/s decode on a Spark via vLLM NVFP4 without speculative decoding. A 55 to 60 tok/s baseline for this size class, lifted into the 70s and 80s by MTP drafting, matches what I measured.

Why I care about this model specifically #

The interesting part is not the benchmark table, it is the division of labor NVIDIA is pushing. Alongside Lightning they released NeMo Switchyard, an open source model router: plans go up to a frontier model, execution comes down to Lightning. LangChain measured a 74% cost reduction routing between Lightning and Claude Opus 4.8 with only 7% of calls escalating to the frontier model, at about a 6 point accuracy tradeoff.

A 3B-active model decoding in the 70s and 80s on a desktop box fits that execution layer well. Two follow-ups I want to do: the NVFP4 checkpoint with the DSpark draft model on this same hardware (the setup NVIDIA actually recommends for Spark), and Lightning as the local execution model with a frontier model in the cloud for planning, Switchyard in between. If that sounds interesting, subscribe.