DeepInfer is the engine that runs DeepSeek's own API, and as of 20 October 2026 it's free on GitHub under Apache 2.0. The code costs nothing; the ticket to run it efficiently is about 128 Blackwell GPUs, which DeepSeek itself names as the smallest efficient unit. At AWS's cheapest listed Blackwell rate that's roughly $32,500 a day, the same money that buys 54 billion output tokens from the DeepSeek API at off-peak prices.
So the release matters, just not as a "run DeepSeek yourself" kit. It supports one model (V4.1 Flash), two chip families, and ships without the KV cache system and load balancers that make DeepSeek's numbers possible. Its real audience is inference providers and labs, and its real effect is on everyone else's API prices.
If you were reading about DeepInfer because you want AI answering customers, you can skip the GPU math entirely. I build the agents behind eesel, and SE Ranking runs three of them on Zendesk, handling 4,650 conversations in a month with no inference stack of its own.
What is DeepInfer?
DeepInfer is DeepSeek's in-house inference framework: the software that takes a trained model and serves it to millions of users quickly and cheaply. The GitHub README calls it "the high-performance inference framework that powers DeepSeek's large-scale production services." DeepSeek announced it on X on 20 October 2026, and the Hacker News thread hit 2,353 points and 892 comments within days.
If "inference framework" is new to you: training builds the model, inference is every time someone asks it something. Open-source engines like vLLM and SGLang already do this job for most open models. DeepInfer is DeepSeek's own fork of the idea, and the README is upfront that it "was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM."
Here's the short version of what shipped, straight from the README:
| DeepInfer, October 2026 release | |
|---|---|
| License | Apache 2.0 |
| Supported models | DeepSeek-V4.1-Flash only |
| Supported hardware | NVIDIA SM100 GPUs, Huawei Ascend 950 NPUs |
| Software | Linux, GCC 13+, Python 3.12, CUDA 13 or CANN 9.2 |
| Mode | Offline inference (batch), not a ready API server |
| Language | Python 89.1%, C++ 10.6% |
| Companion release | deepseek-recipe, the API-to-token encoder |
The model it serves is DeepSeek V4.1 Flash, a 1M-context model with vision and tool calls that I covered in my V4.1 Flash 1017 review. DeepSeek says "inference code for future DeepSeek models" will land in the same repo.
What did DeepSeek actually release?
The headline says "fully open source." The README is more careful, and the gap is the most useful thing to understand about DeepInfer.
What's in the box, per the README:
- Optimized kernels from DeepSeek's earlier open releases: FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect, plus custom TileLang kernels.
- Parallelism: attention data parallelism, MoE expert parallelism with expert load balancing (EPLB).
- DSpark speculative decoding with confidence-scheduled verification.
- SWA bounded replay, a trick that cuts prefill work (more on that below).
- Asynchronous scheduling, so the GPU never waits on the CPU between steps.
- Batch-invariant inference and deterministic sampling, NVIDIA only. Same input, same output, regardless of what else is in the batch.
- Structured outputs: regex, JSON schema or EBNF grammar constraints.
And what isn't. The README lists three things "required for a complete inference service (not included in this release)":
- A KV cache system.
- Vision encoder / prefill / decode disaggregation ("most required engine functionality is already implemented").
- Load balancing across data-parallel replicas for both prefill and decode.
DeepSeek's deployment Q&A explains why: the KV cache and load balancing pieces "are tightly coupled to our internal infrastructure, so that code is not yet fully open-sourced." The same doc then describes how those missing pieces work in a lot of detail, which is generous. It's a design document for the parts you'd have to build yourself.
Hacker News caught this fast. The most compact version:
"Notably missing their kv cache system"
That matters because the KV cache is where DeepSeek's low prices come from. Its API charges $0.003 per million cached input tokens off-peak against $0.15 uncached, a 50x gap, and the engine that earns that gap is the part you don't get.
How does DeepInfer work?
DeepInfer splits serving into separate stages that run on separate pools of GPUs, then stitches them together with load balancers and a shared cache. This is the architecture DeepSeek published in its Q&A:
Reading it top to bottom:
- DeepSeek Recipe turns an incoming JSON request (Chat Completions, Responses or Messages API format) into tokens, and turns output tokens back into a streaming response. It's written in Rust with Python bindings.
- The entrypoint checks the cache for a matching prefix, then routes the request.
- Prefill reads the whole prompt and builds the attention state. DeepSeek runs it on "ordinary 8-GPU machines," with expert parallelism of 2 to 8.
- Decode generates the answer token by token. This runs wide: expert parallelism of 64 to 128 GPUs.
- The KV cache gets written to a dedicated storage cluster so the next request with the same prefix skips prefill.
The split exists because the two jobs stress hardware differently. Prefill is a big burst of math on a long prompt; decode is a long trickle of tiny steps where memory traffic is the bottleneck. Running them on separate GPU pools lets DeepSeek tune each one, so time-to-first-token and the speed of each following token can be controlled independently.
Wide expert parallelism
V4.1 Flash is a mixture-of-experts model: each token only activates some of the model's "expert" sub-networks. Spreading those experts across 64 GPUs instead of 8 means each GPU holds fewer experts and moves less memory per step. DeepSeek's own chart shows the payoff:
At 64 requests per rank, the EP64 setup does about 15,700 tokens per second per GPU at roughly 246 tokens per second per user. EP8 manages about 12,600 at a slower 198 per user. Going wide is what lets DeepSeek be fast and cheap at the same time, and it's also why the minimum deployment is so big.
DSpark speculative decoding
Speculative decoding has a small draft model guess the next few tokens, then the main model checks them in one pass. DeepSeek's twist is that a "Confidence Head" predicts which guesses are likely right, so it only spends verification compute on the promising ones. Per the Q&A, this cuts verified draft tokens per step by about 30% (from 5.00 to 3.2-4.4) with almost no change in accepted length (3.29 vs 3.20). The result is +4-7% throughput at the production operating point. Modest, but at 155 trillion tokens a day, 5% is a lot of GPUs.
SWA bounded replay
This is the cleverest bit. When a cached prefix is missing part of its sliding-window attention state, DeepInfer recomputes only the last 128 tokens instead of the whole window. The Q&A says the recompute cost becomes "a constant, independent of context length and layer count," and that the decoder-side version is "nearly halving total prefill computation." In the README, you turn it on with DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact.
The KV cache you'd have to build
DeepSeek's cache layer is enormous. The Q&A describes about 100 PiB of SSD storage on its 3FS file system, with cached prompts kept for 3 days for agent traffic and 7 days for chatbot traffic, and about 14.4 TB of logical cache per GPU online. That long retention is why a returning conversation with DeepSeek is so cheap, and it's the piece that's "tightly coupled to our internal infrastructure."
What does it take to run DeepInfer?
Two numbers decide whether DeepInfer is for you: the hardware list and the minimum cluster size.
Hardware. Only NVIDIA SM100 (Blackwell) GPUs and Huawei Ascend 950 NPUs. No Hopper (H100/H200), no consumer cards. A community repo, Deepinfer-SM80, already ports the ideas to older Ampere GPUs, which tells you how much demand there is for a version that runs on hardware people actually own.
Setup. The getting-started path is short if you have the machines:
# Use `ascend` instead for the Ascend platform
export DEEPINFER_ACCELERATOR=nvidia
./build.sh
python examples/offline_inference.py --ep 8
The first run downloads the deepseek-ai/DeepSeek-V4.1-Flash-1017 checkpoint from Hugging Face. That's offline (batch) inference: you hand it prompts, it hands back outputs. There's no OpenAI-compatible server in the box.
Minimum cluster. This is the one most coverage skips. From the Q&A: "the smallest efficient deployment unit is 64 GPUs of prefill plus 64 GPUs of decode." You can technically run one prefill and one decode node, but decode is tuned for EP64 and above, and the efficiency numbers above assume it.
Plus the storage. DeepSeek recommends 20 x 30.73 TB SSDs per KV cache storage server, each delivering at least 40 GB/s reads, sized to hold 24 hours of data. Most teams reading the README won't have that sitting idle.
How much does DeepInfer cost?
DeepInfer itself costs $0. Everything else is hardware, and the honest comparison is against just calling the DeepSeek API.
Here's what DeepSeek charges for the same model DeepInfer serves, from its pricing page:
| Per 1M tokens | V4.1 Flash, off-peak | V4.1 Flash, peak | V4 Pro, off-peak | V4 Pro, peak |
|---|---|---|---|---|
| Input, cache hit | $0.003 | $0.006 | $0.022 | $0.044 |
| Input, cache miss | $0.15 | $0.30 | $0.66 | $1.32 |
| Output | $0.60 | $1.20 | $1.98 | $3.96 |
Peak hours are 01:00-04:00 and 06:00-10:00 UTC on weekdays; everything else, including weekends, is half price. My V4.1 Flash pricing guide walks through those windows in more detail, and the V4 Flash vs V4 Pro comparison covers when the bigger model is worth it.
Now the self-hosting side. AWS EC2 Capacity Blocks list GB200 at $10.582 per accelerator-hour (US East Dallas Local Zone) and B200 at $14.208 (Ohio, Virginia, Oregon and others). Both are SM100 Blackwell.
| 128-GPU cluster | Per hour | Per day | Per 30 days |
|---|---|---|---|
| GB200 at $10.582/GPU-hr | $1,354 | $32,508 | $975,240 |
| B200 at $14.208/GPU-hr | $1,819 | $43,647 | $1,309,402 |
That's before storage servers, networking, and the engineers to build the missing KV cache and load balancers. $32,508 a day buys 54.2 billion output tokens from DeepSeek at off-peak rates, or 27 billion at peak.
Can 128 GPUs even produce that much? Using DeepSeek's own decode benchmark (about 15,700 tokens per second per GPU at 64 requests per rank) across 64 decode GPUs, the ceiling is about 87 billion output tokens a day. DeepSeek says its load balancing reaches "roughly 80% of a perfectly balanced offline benchmark," which brings that to about 69 billion. So self-hosting beats the off-peak API only if you keep the cluster at roughly 80% of DeepSeek's own production efficiency, around the clock, with infrastructure DeepSeek didn't release. That's a rough ceiling, not a quote, but it shows the shape of the decision.
Plug in your own volume:
At the default numbers, a busy product using 500 million output tokens and 3 billion input tokens a day, the API costs about $680 a day against $32,508 for the GPUs. You'd need roughly 48 times that traffic before the cluster wins on hardware alone.
What the $38 per GPU-hour number tells you
The most quoted line from the launch isn't in the README. It's in the Q&A: "A single cluster currently serves 155 trillion tokens per day, with overall revenue of US$38 per GPU-hour (at valley pricing)." Valley pricing means the off-peak rate.
Put that next to the AWS rates above and DeepSeek is earning about 2.7x to 3.6x what it would cost to rent a Blackwell GPU for the same hour, while still charging $0.60 per million output tokens off-peak. DeepSeek doesn't say which card the figure refers to, and it doesn't state its own costs, so treat it as revenue, not margin. HN noticed the gap too:
"For comparison, rental prices seem to be ~$2.75/ per H-100-hour [1]. Meanwhile, Deepseek says "38 USD per hour per card". They haven't specified which card as far as I can tell."
The point isn't that DeepSeek is rich. It's that cheap API prices can be a real business when the serving stack is this efficient. That puts pressure on every provider charging more for the same open weights, and DeepInfer hands them the playbook.
What developers are saying about DeepInfer
The reaction split three ways: excitement about the openness, skepticism about what's missing, and a sharp point about who benefits.
The skeptic case, from the HN thread:
"Deepseek released just enough to stir the pot but not enough to cook your own from scratch. Wake me up when we can clone, point it at a dozen H100s we own, and get a working production service."
That's fair on both counts: H100s aren't supported, and the production pieces aren't there. The other side of it is that third-party hosts were already competing hard on DeepSeek's models before this release:
"Together.ai, Baseten, and others have already achieved higher throughput, lower latency, and lower E2E latency for DeepSeek V4.1 Flash 1017 than DeepSeek's own inference service."
Providers like Together AI, Baseten and Fireworks AI are the ones who can actually use DeepInfer's kernels and the deepseek-recipe encoder, which DeepSeek says is "fully aligned with the official DeepSeek API, so third-party deployments get the same API behavior as the official one." For a developer, that means a DeepSeek model on a third-party host should parse tool calls and reasoning the same way it does on DeepSeek. That's a quiet but real fix.
And a point about pricing models that has nothing to do with engines:
"Even though their API price is cheap compared to OpenAI/Anthropic, I still can't "afford" to use their API because the subscriptions from OAI/Anthropic provide far higher value and peace of mind. I know for sure how much I need to pay each month."
DeepSeek's own announcement framed the goal as helping "the community lower compute costs, reduce energy use, and make AI more affordable." The post pulled 43,339 likes on X.
How does DeepInfer compare to vLLM and SGLang?
DeepInfer isn't a general-purpose rival to vLLM or SGLang yet. It's a specialist built from them. Here's how the three line up based on each project's own README:
| DeepInfer | vLLM | SGLang | |
|---|---|---|---|
| Models | DeepSeek V4.1 Flash only | Many open model families | Many open model families |
| Hardware | Blackwell (SM100), Ascend 950 | Multiple GPU and accelerator vendors | Multiple GPU and accelerator vendors |
| Online API server | Not in this release | Yes, OpenAI-compatible | Yes, OpenAI-compatible |
| Origin | Built primarily on vLLM | Original project | Original project |
| Best for | Studying or reusing DeepSeek's production tricks | Serving most open models | Serving most open models |
If you're picking an engine to serve an open model this week, vLLM or SGLang is still the answer. DeepInfer is where to look for ideas (wide EP, DSpark, bounded replay) that will probably show up in those projects next. Google's TorchTPU launched the same day and has the same flavor: important plumbing for people who run chips, invisible to everyone else.
Who should care about DeepInfer?
Short answer: fewer people than the star count suggests.
- Inference providers and big labs: yes. This is a working reference for serving a frontier-scale MoE model cheaply, with the design for the missing parts written down.
- Researchers and infra engineers: yes, as a study guide. The Q&A doc is one of the most detailed public write-ups of production LLM serving I've read.
- Teams with strict data rules who want DeepSeek on their own hardware: maybe, if you already run Blackwell or Ascend at scale. The deterministic sampling mode is a real plus for audits.
- Startups and app builders: no. Use the API, or a host like Fireworks AI serving open models on your behalf. Compare with Kimi K3.1 Pro or Qwen 3.8 Max before you commit to one model family.
- Support and ops teams: no. You need answers, not an engine.
DeepInfer lowers the floor on what AI costs to serve; it doesn't lower what it costs you to run AI yourself. Those are different things, and most of the excited coverage blurs them.
Where eesel fits: this is infrastructure, eesel is the employee
DeepInfer is infrastructure: it makes tokens cheaper to produce. eesel is the other end of that stack, an AI teammate you hire for a job. Right now that's the AI helpdesk teammate and the AI blog writer, each with the skills, integrations and company context for its role already built in.
For a support team, the cost of the model was never the hard part. At DeepSeek's off-peak rates, a ticket with 6,000 uncached input tokens and 800 output tokens costs about $0.0014, roughly a seventh of a cent. What costs real time is making the agent answer like your best agent would: reading your help center and past tickets, knowing when to hand off, and proving it on real history before customers see it. I've watched confident bots give wrong answers on live queues, which is why every eesel rollout gets simulated against historical tickets first.
SE Ranking is a good picture of what that looks like. It sells three products from one support team, so it runs three eesel agents on three Zendesk connections, each answering for its own product from about 16,800 pages of knowledge. Over a month (8 August to 8 September 2026) they handled 4,650 conversations: 2,554 Zendesk tickets and 2,102 live chats, with about 2,100 chats answered first by eesel. No GPU cluster involved.
If you want to compare what AI costs inside your helpdesk, my Zendesk AI pricing breakdown and the guide to AI customer support are the better starting points than any inference engine. And if you're curious about which models are actually good at support, I keep a running take on the best LLM for support.
For developers who like the terminal, eesel has a CLI that drives the same teammate and workspace as the dashboard. You, a script, or a coding agent like Claude Code can connect a helpdesk, add knowledge and test the agent against past tickets without opening a browser. That's the agent-friendly access most teams actually want from "open infrastructure."
Try eesel
DeepInfer is for people who run GPU clusters. If your goal is AI that answers customers in Zendesk, Freshdesk or Gorgias, eesel is a helpdesk teammate that learns from your past tickets and help center, gets tested on your real history before it goes live, and starts at $299 a month for 500 tickets or chats on the pricing page. Try eesel free with 100 credits, or start from the terminal with npx @eesel/cli.
