Groq 3 LPX: NVIDIA's low-latency inference accelerator, explained

What Groq 3 LPX actually is

NVIDIA describes Groq 3 LPX as "the interactive AI inference accelerator for NVIDIA Vera Rubin." In plainer terms: it is a full rack of specialized chips whose one job is generating tokens fast, and it slots into NVIDIA's newest data-center platform alongside the general-purpose GPUs.

The framing NVIDIA uses is a classic three-way tradeoff. For years, inference hardware gave you two of three properties at once: intelligence plus throughput, or intelligence plus interactivity, but never all three cheaply. Big models are slow to talk to; fast models are small. Groq 3 LPX is pitched as the piece that lets a big model stay responsive, by handing the latency-sensitive part of each token to a chip built for exactly that.

It is worth being precise about what it is not. It is not a standalone product you drop into any server, and it is not a replacement for the GPU. It is a codesigned extension of Vera Rubin NVL72 that runs next to Rubin GPUs, with the two jointly computing every layer of the model for every output token. NVIDIA even calls the LPU "the seventh chip" of the Vera Rubin platform.

Wait, is this Groq the startup?

This is the single most common reaction I've seen, and it's a fair one. Groq spent years as the scrappy inference startup whose LPU was the counterexample to "just use more GPUs." Now "Groq 3 LPX" is an NVIDIA product line. One person on X summed up the mood well:

"so groq the company is now also an nvidia product line, that's going to confuse people for months"

Gabi, X

Here's the accurate version, straight from the primary sources. Groq entered a non-exclusive licensing agreement with NVIDIA for its inference technology, the same tech behind GroqCloud's pricing and its lineup of Groq alternatives. As part of it, founder Jonathan Ross and other Groq team members joined NVIDIA to help scale the licensed design. Groq stays independent under a new CEO, Simon Edwards, and GroqCloud keeps operating. NVIDIA's own press page carries the footer line that "Groq and LPU are used under license from Groq, Inc."

So when you read "NVIDIA Groq 3 LPX," picture a license and a team hire, not a clean acquisition. There are big dollar figures floating around in the press for the deal, but those numbers don't appear in either company's own statements, so I'm leaving them out. What the primary record supports is the structure above, and that's the part that actually matters for understanding the product.

The whole thing, in one rack

The specs are the fun part, because they are genuinely unusual. Here is what NVIDIA lists for a single Groq 3 LPX rack.

The number that jumps out is 40 petabytes per second of on-chip SRAM bandwidth. That is not a typo, and it's the whole design thesis in one figure. Instead of feeding chips from slower external memory, the LPU keeps the working data on-chip and moves it at a bandwidth a conventional memory setup can't touch. The tradeoff is capacity: 128 GB of SRAM across a rack is tiny next to the terabytes of HBM a GPU rack carries, which is exactly why the LPUs need the GPUs beside them.

Why build a whole new chip?

To see why NVIDIA bothered, it helps to put the two chips side by side. NVIDIA's own diagram does this cleanly.

A Rubin GPU brings roughly 288 GB of HBM4, about 22 TB/s of memory bandwidth, and around 50 PFLOPs of compute. A single Groq 3 LPU brings 500 MB of SRAM, but at 150 TB/s of bandwidth, with 1.2 PFLOPs of FP8 compute. The GPU is a workhorse optimized for peak throughput and big memory. The LPU is optimized for one thing: keeping a stream of tokens flowing with predictable, low latency.

The design choices under the hood are what make the LPU different from a GPU, and I find them genuinely interesting because they invert some things GPU programmers take for granted:

• SRAM is the main memory, not a cache. There is 500 MB of on-chip SRAM per LPU, and it holds the active weights, activations, and KV state directly. There is no HBM on the chip at all.
• No hardware-managed caches. The compiler and runtime explicitly place data and move it, rather than relying on cache heuristics. That's what cuts unpredictable stalls and keeps latency stable.
• Deterministic, compiler-scheduled execution. The chip builds on Groq's spatial execution model, where compute, data movement, and synchronization are scheduled ahead of time. NVIDIA describes a hardware protocol that cancels clock drift so hundreds of LPUs act as one coordinated system.
• Explicit, tensor-first data movement. Everything operates on fixed 320-byte vectors through dedicated matrix, vector, and switch units, so scheduling stays predictable.

If you've spent time thinking about why LLM inference costs are so sensitive to latency, this is the hardware answer to it: build a chip where the timing is knowable, then wire hundreds of them together without losing that property.

How it actually works: a two-engine loop

Here is where it clicks. NVIDIA's insight is that inference isn't one uniform workload. A request has two very different phases, and they want different hardware.

The first phase, prefill, ingests your prompt and builds the KV cache. That is dense, parallel, memory-hungry work, and it's a great fit for the GPU. The second phase, decode, generates the answer one token at a time in a loop. NVIDIA splits that loop across both chips, in a technique it calls attention-FFN disaggregation: the Rubin GPUs run attention over the accumulated context, and the Groq 3 LPX handles the feed-forward and mixture-of-experts math that's most latency-sensitive. For every single token, the two exchange intermediate activations.

The traffic cop for all of this is NVIDIA Dynamo, the orchestration layer that routes prefill to GPU workers, moves the per-token activations between engines, and schedules everything against latency targets so an interactive session doesn't get stuck behind a batch job.

LPX plays one more role worth knowing: it acts as the draft engine for speculative decoding. Its deterministic execution and huge SRAM bandwidth let it churn out draft tokens fast, running ahead of the GPU, which verifies them. That's another way the two chips divide labor instead of duplicating it.

Inside the rack

If you like seeing where the parts physically live, NVIDIA's compute-tray diagram is the clearest look. A rack is 32 of these 1U liquid-cooled trays, and each tray holds eight LPUs.

A few things stand out in that layout. Each LPU connects through 96 chip-to-chip links at 112 Gbps, adding up to 2.5 TB/s of scale-up bandwidth per chip, which is how 256 of them behave like one machine. Each tray also carries a BlueField-4 DPU and a host CPU, and the whole thing is cableless and liquid-cooled. It rides NVIDIA's MGX rack architecture, so a data center can plan for one common rack design across its Vera Rubin deployment rather than a bespoke setup for the inference path.

The performance claims, and how to read them

Now for the numbers everyone quotes. NVIDIA's headline is up to 35x higher throughput per megawatt for trillion-parameter models when Vera Rubin NVL72 is paired with LPX, versus its previous GB200 generation, measured at 400 tokens per second per user. It also projects up to 10x more revenue per watt for the most latency-sensitive workloads. The clearest picture is NVIDIA's own Pareto chart.

Read the chart honestly and it says something real: the gap between the green line (Rubin plus LPX) and the older systems opens up dramatically on the right side, where each user is getting tokens fast. At low interactivity everyone looks similar; at high interactivity, the LPX path stays high while the GPU-only lines fall off a cliff. That's the actual claim, and it's a coherent one.

The honest caveat is who measured it. These are NVIDIA's figures against NVIDIA's own baselines, and the modeled "premium" scenario is a specific 2-trillion-parameter model at a 400K context. The one independent-flavored data point I could find is a NVIDIA-cited Artificial Analysis benchmark of 3,400 tokens per second on Gemma 4 31B at 100K context. It's a strong number, but almost all the public conversation so far is still echoing NVIDIA's slides rather than testing the hardware, because Nebius is only the first cloud to deploy it. On Hacker News, one of the few substantive takes read it against Groq's track record rather than the marketing:

"Gemma 31B dense at 3k TPS seems like a pretty big deal. Groq has always been more efficient than Cerebras at running very small models and this release seems to be no exception."

2001zhaozhao, Hacker News

So my read: the architecture is legitimately clever, the benchmark is promising, and the 35x figure is a best-case marketing number you should hold loosely until third parties run it.

Why now: inference became the expensive part

The reason a chip like this exists is that the shape of AI workloads has changed. When most inference was a single chatbot reply, decode latency barely mattered. Agentic systems broke that. They generate tokens across hundreds or thousands of steps, and NVIDIA cites data showing agentic workloads consume roughly 15x more tokens than a simple chat request. Every small per-token delay now multiplies across a long chain.

You can see the same shift in where the market's money is going. NVIDIA's data-center business is riding this wave hard: the company reported $96.2 billion in revenue for its quarter ending in late July 2026, up 106% year over year. Inference, not training, is now the growth engine, and a chip built specifically for low-latency, multi-step generation is a direct bet on that. One analyst take on X put the strategic angle cleanly:

"Nvidia moving its Groq 3 LPX inference accelerator into full production signals a key shift from model training to continuous agentic AI workloads."

Yellow Media, X

The mental model that helps here: the further your workload sits toward "many fast turns," the more a dedicated low-latency engine matters. Batch jobs don't care. Coding assistants, agent swarms, and voice care a lot.

What this means if you build with AI

Here's the part I want to be straight about, because it's easy to get swept up in the spec sheet. Unless you run a cloud or a very large AI platform, you are not going to buy a Groq 3 LPX rack. This is infrastructure that sits several layers below where most teams work. You'll feel it as faster, cheaper tokens showing up in the model APIs and clouds you already use, the same way you never bought a GPU to use an AI agent.

And that's the useful reframe. The chip is the engine; the thing you actually deploy is the worker that rides on top of it. Faster inference makes agents snappier and cheaper to run, which lowers the bar for putting real automation into production and shifts the cost math against a human agent, whether that's a coding agent, a research agent, or an AI helpdesk agent working your support queue. The hardware race is about making that layer viable at scale. The value still lives in the layer.

Where eesel fits

Chips like Groq 3 LPX are the plumbing. eesel is what you hire to do the job on top of it. eesel is an AI teammate platform: you don't provision inference or wire up a pipeline, you bring on a ready-to-work teammate for a specific role and plug it into your apps. The current roster is an AI helpdesk teammate that joins your support queue as more than a basic support chatbot, and an AI blog writer that produces content like this piece.

The concrete difference is where the fast tokens land. A support teammate that answers in one second instead of eight is the whole experience for the person waiting on it, and the years we've spent running AI on live support queues taught us the un-glamorous half: speed only helps if the answer is right, which is why eesel simulates a rollout against your past tickets before it ever replies to a live customer. All the silicon in the world doesn't fix a confident wrong answer. If you want to see the teammate rather than the rack, you can try eesel for free.