Nemotron 3.5 Bolt: NVIDIA's 24GB local AI model, explained

Why I read the Bolt technical report

I build features and AI agents at eesel, so most of my week is spent on the boring middle of an agent: what goes into the context, which tool it calls, and what happens when the model is wrong. The question I hear most from technical buyers is not "how smart is it?" It's "where does our data go?" On one sales call this year, an evaluator at a hardware company asked whether the AI would fall back to "some kind of other ChatGPT" when it didn't know an answer, and whether their knowledge stayed closed to their organization.

A model you can run on a box under your own desk is one honest answer to that question. So when NVIDIA shipped a model whose whole pitch is "fit a big model in 24GB," I read the model card, the technical report and every user test I could find. Here's what holds up and what needs a footnote.

What is Nemotron 3.5 Bolt?

Nemotron 3.5 Bolt is NVIDIA's "edge" line inside the Nemotron 3.5 family. NVIDIA announced it on 15 October 2026 alongside Nemotron 3.5 Super VL, and describes the family this way: Lightning handles high-throughput workloads, Ultra handles the hardest reasoning, and "Bolt brings intelligence to the edge."

The name covers two very different models, which trips up a lot of the early coverage:

• Bolt 74B-A9B 24GB is the headline model. It is "designed to run everyday agentic tasks locally on NVIDIA RTX/DGX Spark, using a native 2-bit quantization to fit in 24GB of memory," per NVIDIA.
• Bolt E4B 4GB arrived on 23 October. It's an 8B model with 4B effective parameters, built on Google's Gemma 4 E4B with Nemotron post-training, for "fast local voice assistants, interactive characters and embedded interfaces."

Here are the specs side by side:

Both have reasoning you can switch on or off, and the 74B model supports tool calling and JSON output. If you've been following open releases like Meta Muse Spark Open or Kimi K3, the difference here is the target: Bolt is built for one machine, not a data center.

How does NVIDIA fit a 74B model in 24GB?

The short version: start big, cut it down, then shrink every number. NVIDIA's research page states the goal plainly: "fit into 24GB of memory and achieve a decoding speed of at least 80 TPS, while maximizing intelligence." Four steps get it there.

1. Compress a bigger model. Bolt starts as Super VL, which has 120B parameters and picks 22 experts per token. NVIDIA's Star Elastic method prunes it to 74B and fixes the expert count at 9. On its own that costs 2.85 points of average accuracy, 4.94% relative to the parent, per the technical report.
2. Heal it with training. Bolt then gets 120B tokens of recovery training, re-attached vision, supervised fine-tuning and reinforcement learning. The vision part needs it most: compression dropped the WebArena score from 57.4 to 1.35, and training brought it back to about 50.
3. Quantize the experts to 2 bits. The expert weights are stored at 2.0625 bits each, using quantization-aware distillation so the small model learns to match the big one at that precision. The Mamba layers use 4-bit NVFP4, and attention and embeddings stay at 8-bit. That takes the model from 148GB to 22.1GB, "an 85.1% reduction in size. This quantization is lossy by design," says NVIDIA.
4. Add a drafter for speed. A separate 1.13B-parameter model called LiLiCorr guesses several tokens ahead and the main model checks them, which is how NVIDIA gets 84 tokens per second on RTX Spark.

NVIDIA's bet is that a big model shrunk to 2-bit beats a small model trained at 4-bit. That's the opposite of how most local models are built, and it's the new idea in this release.

One footnote on the "24GB" in the name. The technical report's memory table adds up to 23.83GB for text only. The vision encoder needs another 1.68GB on top.

So on a 24GB card you're running text agents. If you want the image and video skills too, plan for a bit more headroom, and Unsloth's local guide suggests 24 to 32GB of memory anyway.

How good is Nemotron 3.5 Bolt, really?

Read two scoreboards: the independent one, and NVIDIA's own.

The independent score. Artificial Analysis puts Bolt 74B at 21.4 on its Intelligence Index, "placing 4th among models under 100B parameters and first among US models in that class." That's within 2 points of Nemotron 3 Ultra (23), a model with about seven times the parameters. Bolt even beats Ultra on AutomationBench-AA (15.5% vs 3.0%).

Look at the left of that chart, though. Qwen3.8 27B scores 34. Bolt is the best US model in its class, not the best model in its class.

NVIDIA's own table. The model card runs the 2-bit Bolt against three mixture-of-experts peers at full precision:

Against those three, Bolt wins most rows, and doing it at 2-bit is impressive. But the table leaves out dense Qwen3.8 27B. NVIDIA's own technical report charts it at 44.5 on GDPVal v2 against Bolt's 28.5, at roughly a third of the speed. That's a fair trade to offer, but you have to read the PDF to see it.

Two more costs to know:

• 2-bit costs something. Against Bolt's own full-precision weights, GDPVal drops from 31.13 to 28.47 and Omniscience accuracy from 21.80 to 16.90, per the technical report.
• It talks a lot. Artificial Analysis counted 82k output tokens per Index task, 1.8x Super VL and 2.9x Ultra. Locally that costs time rather than money, but it's time.

How fast is it on real hardware?

NVIDIA's own speed table is more modest than the launch posts. On RTX Spark with llama.cpp, Bolt runs at 83.99 tokens per second, against 96.63 for Qwen3.6 35B-A3B, 100.56 for Gemma 4 26B-A4B and 28.26 for Qwen3.8 27B, per the research page. So it's slower than the small mixture-of-experts models it beats on accuracy, and about 3x faster than the dense model that beats it.

Users with bigger cards see much more:

"343 tok/s median with the LiLiCorr drafter (DFlash, n_max 6, about 55% of drafted tokens accepted), 166 tok/s without it. 25.0 GiB at 32K with the drafter and the projector, 22.7 GiB without the drafter."

u/TaraVisionCode, Reddit

That's on an RTX 5090, and note the 25GB figure once vision and the drafter are loaded. On a 16GB card, one user reported about 45 tokens per second by offloading part of the model to system memory. Mac users have it worse for now:

"Nemotron Bolt appears to be really quite good but unfortunately it is quite slow on macOS/Metal in llama.cpp at the moment because of lack of support for a tensor type (or something… not sure I understand)."

dofm, Hacker News

What does Nemotron 3.5 Bolt cost?

The weights are free, and that's the whole price list. As of 27 October 2026, neither Bolt model has a paid hosted API. It isn't listed on OpenRouter or DeepInfra, NVIDIA's build.nvidia.com page for the 74B model returns a 404, and Artificial Analysis lists "No publicly available API."

If you'd rather pay per token, its siblings are hosted. On OpenRouter, Super VL lists at $0.15 per million input tokens and $0.50 output, and Lightning at $0.07 and $0.20. If hosted is the plan, my guide to the best AI model for support tickets compares the closed options too, like Claude Haiku 5.5 and Gemini 3.8 Flash.

The license is the other half of the cost story, and it's a good one. OpenMDW-1.1 grants use "without restriction," puts no limits on outputs, and has no user cap or acceptable-use policy. You keep the notices when you redistribute, and your rights end if you sue claiming the model infringes a patent or copyright. A Reddit commenter called it "an Apache styled license," which is about right.

How do you run it?

You have more options than with most day-one releases. NVIDIA and partners shipped:

• Ollama: ollama run nemotron-3.5-bolt pulls the 25GB build with 256K context and image input. Ollama also lists one-line launchers for Claude Code, OpenCode, Hermes Agent and OpenClaw.
• llama.cpp: NVIDIA's own GGUF plus a separate drafter file. NVIDIA says the build is "optimized for use on RTX Spark with llama.cpp and DGX Spark with vLLM."
• vLLM: needs a nightly build and only supports Blackwell-generation GPUs for now.
• LM Studio, Unsloth and a Canonical snap round it out.

The local agent harness angle is the real use case. NVIDIA pitches handing Bolt projects "from researching the web to writing to debugging code," without sending the work to the cloud. The users who like it most give it a narrow job with a clear finish line:

"I really don't hate this model. I was expecting to, it's a 2 bit quant for fucks sakes, but it seems to do really well when it's got some tools, a defined goal and a success condition such as 'compile works and tests pass'."

u/fantasticsid, Reddit

What are early testers saying?

The reaction on r/LocalLLaMA is warm, with a clear ceiling. Four themes came up again and again.

The 2-bit trick works better than expected. One commenter called "74GB QAT coming in at 24GB on disk" "pretty incredible" and said it "has a decent chance of retiring Qwen3.6-35B for some people." Another runs it for a fax-ingest job at work, where it beat their previous vision model head to head.

It's below dense Qwen 3.8 27B on real work. The most careful test I found ran 33 infrastructure tasks on an RTX 5090:

"26 of 33 passed, with the LiLiCorr drafter on. On the same tasks the same day qwen3.8:27b passed 32, qwen3.6:27b 31, gemma4:31b and glm-4.7-flash 29, qwen3.6:35b and gemma4:12b 27."

u/TaraVisionCode, Reddit

A second tester ran 36 blind tasks against Gemma 31B and found Bolt "about 3.5 times faster, but it made up more problems that weren't there."

It explores a lot. The same 33-task tester found 54% of its tool calls were probes and diagnostics, and summed it up as "Careful, and expensive in turns." Another user saw it skip the Docker instruction in their AGENTS.md file.

Safety is a real question mark. This one matters most for anything customer-facing:

"It is not safe for an agent; it failed 6/21 jailbreak tests. Qwen3.8 Flash and 27 B failed none."

u/Iron-Over, Reddit

It's one person's test, and a reply in the same thread made the fair counterpoint: "If you're relying on model weights to protect you from hostile content, you're doing it wrong." Both are true.

What about Bolt E4B?

Bolt E4B is the one to watch if you build voice or device products. NVIDIA's E4B page claims "~40% higher average quality" than its previous 4B model and "up to ~80% fewer tokens vs peers," and lists "customer service" among its voice-assistant uses. It has a conversational mode that keeps replies short enough to speak, and supports 35 languages.

It's fast. On an RTX 5070 it returns its first token in 548 milliseconds and runs at 133.4 tokens per second, using 3.5GB of video memory. Even a Jetson Orin Nano manages 15.5 tokens per second.

The benchmark table is mostly a win, with one row support teams should notice:

Qwen 3.5 4B beats it by 20 points on Tau2-Bench, the benchmark that simulates an agent handling customer requests under a company policy. If you're building a voice agent for customer support, test both.

Is Nemotron 3.5 Bolt a good fit for customer support?

It's a good fit for one part of support, and a risky fit for the part people usually mean.

The good part: keeping data in-house. A model that runs on one machine with a permissive license answers the "where does our data go?" question cleanly. For internal work, like summarizing tickets, classifying them or drafting replies an agent reviews, a local Bolt is a reasonable engine, and its Tau3 Banking score (17.01) beats every peer in NVIDIA's table.

The risky part: answering customers directly. 17% on a banking customer-service benchmark is still low in absolute terms. Add a reported 6-in-21 jailbreak failure rate and a habit of exploring with extra tool calls, and you have a model that needs a lot of scaffolding before it talks to a customer. Every customer message is untrusted input.

What the scaffolding has to do is the part I spend my days on. eesel learned early that a wrong detail is worse than no answer, which is why every rollout is simulated against historical tickets before an AI replies to a real customer. One of our published customers, an industrial automation company, runs a chat that answers its engineers only from its technical documents, and says where each answer came from. Not finding an answer is an acceptable result there. That's a property of the system around the model, not of the weights. Our guide to preventing AI hallucinations in support covers the guardrails in more depth.

So my verdict: if you have engineers, a 24GB+ NVIDIA GPU and a reason to keep data local, Bolt is worth a weekend. It's not something you point at your Zendesk AI queue on Monday. If you're comparing ready-made tools instead, start with my roundup of AI helpdesk software.

Who should use Nemotron 3.5 Bolt?

Here's how I'd decide:

• You run local coding or research agents on one GPU. This is exactly what Bolt 74B was built for. Give it tools and a clear success condition.
• You need to keep data off the cloud. Regulated data, a contract that bans outside AI processors, or a laptop on a plane. The license and the 24GB footprint make this easy to justify.
• You build voice, game or device products. Bolt E4B is fast, small and short-spoken by design.
• You want the smartest model that fits in 24GB. Look at Qwen3.8 27B first, and accept it's about 3x slower. Or go hosted with Qwen3.8 Flash-Next or DeepSeek V4 Flash.
• You need a hosted API this week. Bolt doesn't have one. Use Nemotron 3.5 Super VL instead, which is on OpenRouter.

Try eesel

Nemotron 3.5 Bolt is infrastructure: free weights, a GPU and a weekend of setup. You'd still have to build the retrieval, the escalation rules, the testing and the audit trail. eesel is the employee. Our AI helpdesk teammate plugs into Zendesk, Freshdesk, Gorgias and the rest, learns from your past tickets and help center, and answers only from your own knowledge. Before it replies to a single customer, you can run it against hundreds of your historical tickets and see what it would have said.

That's how Dokument 24 runs it on Zendesk: in one month the teammate prepared 680 tickets for the team and left 670 draft replies in Danish as internal notes, ready to send or edit.

If you like driving agents from a terminal the way you'd drive a local model, the eesel CLI operates the same teammate. You can read and edit its standing instructions, list actions waiting on a human approval, and check each run's activity log, with JSON output and dry runs on writes, so a coding agent like Claude Code can manage your support teammate alongside your repo. If you write content rather than answer tickets, the same roster has an AI blog writer.

It's free to start. Try eesel on last month's tickets before you buy a 24GB card for it, and see the plans on the pricing page.