DeepSeek open-sources DeepInfer, the framework behind its V4.1 Flash service

DeepInfer is the inference framework that powers DeepSeek’s V4.1 Flash service. On October 20, DeepSeek published it on GitHub under the Apache-2.0 license, and its official Q&A says one cluster now serves 155 trillion tokens a day. The repository provides offline inference. The KV cache system and cross-replica load balancing that a full online service needs are not included.

Key points

• DeepInfer is licensed under Apache-2.0. Its README lists support for NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, and names DeepSeek-V4.1-Flash as the supported model.
• DeepSeek’s headline figure is $38 of revenue per GPU-hour at what it calls valley pricing. The Q&A gives no cost or profit figure and does not say which GPU the number refers to.
• The README lists three things a complete service needs that this release leaves out: a KV cache system; VE/Prefill/Decode disaggregation, although the README says most of the required engine functionality is already implemented; and load balancing across data-parallel replicas for prefill and decode.

Background

DeepSeek had already released some of the building blocks. In late September we reported on six Ascend infrastructure repositories published under the deepseek-ai organization (DeepSeek’s Ascend components). DeepInfer’s README names FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect as sources of its optimized kernels, along with custom TileLang kernels. This release adds the engine that connects them.

DeepInfer has a clear lineage. The README says it was “originally built primarily on vLLM,” with inspiration from SGLang and TensorRT-LLM. Its default checkpoint is DeepSeek-V4.1-Flash-1017, the model card we covered on October 17.

The facts

The table sets each official figure beside its source. The GitHub repository and the Q&A are primary sources. The cache and latency figures come from DeepSeek’s Zhihu article, which Crypto Briefing also reports.

Item	Official figure	Source
Cluster throughput	155 trillion tokens a day	Official Q&A
Revenue basis	$38 per GPU-hour at valley pricing	Official Q&A
KV cache	Kept for more than 72 hours; about 100 PiB of 3FS SSD; TTL of 3 days for agent traffic and 7 days for chatbot traffic	Official Q&A
Parallel layout	Prefill EP2–8; decode EP64–128 (Wide-EP); smallest efficient unit is 64 prefill GPUs plus 64 decode GPUs	Official Q&A
Input cache hit rate	96%	DeepSeek Zhihu article, via Crypto Briefing
Average time to first token	About 400 ms	Same
Decode speed	230 tokens/s; up to 300 tokens/s on coding tasks	Same
Benchmark setup	4 SM100 GPUs for prefill and 64 for decode, each within one scale-up domain	README
Checkpoint size	DeepSeek-V4.1-Flash-1017: 83 files, about 510.3 GB in total	Hugging Face

These figures answer different questions, so read them separately. The $38 is revenue per GPU-hour. The Q&A discloses nothing about hardware depreciation, power or utilization, so it cannot be turned into a margin. The speculative decoding gain is also an estimate. The Q&A computes it from EP64/DP64 benchmark data: at about 60 concurrent sequences per rank, dynamic-length verification adds 4% to 7% throughput over fixed five-token drafts. Across the table’s 16-to-96 concurrency range, the estimate runs from −1.5% to +11%. The Q&A also says a TTL of about three days costs chatbot traffic 0.5% to 1% of its KV cache hit rate.

What others say

Yicai’s headline puts the figure first: a card earns $38 an hour. Its story says savings still depend on hardware configuration, workload and deployment capability, and that a full online service still needs the KV cache system, prefill-decode separation and cross-replica balancing.

On Hacker News, the thread had 795 points and 271 comments (replies included) when we checked through the official API at 12:50 UTC on October 20. One commenter pointed out that DeepSeek cited “38 USD per hour per card” without saying which card. Another argued the release is just enough to stir things up but not enough to build a production service from scratch. Someone wrote that the KV cache system was “notably missing.” Another asked how DeepInfer compares with vLLM and SGLang; neither the README nor the Q&A offers that comparison. A commenter also claimed that Together.ai, Baseten and others already beat the model makers’ own serving on DeepSeek V4 throughput and latency, without citing data.

Crypto Briefing does not mention the $38. It reports the 155 trillion tokens, the 96% cache hit rate, about 400 ms to first token and 230 tokens/s, and frames the release as a way for enterprises to cut compute costs and energy use. Phoenix (凤凰网), carried by Tencent News, stresses compatibility: according to the official description, DeepInfer runs on NVIDIA and Huawei Ascend platforms.

Our take

For a team planning its own DeepSeek V4.1 Flash service, DeepInfer is best read as a reference implementation. It shows how DeepSeek sizes expert parallelism, how it stores KV cache in 3FS and how it splits prefill from decode. It is not a turnkey server. The KV cache system and cross-replica balancing still have to be built, and that work belongs in any cost comparison.

The $38 is the figure most likely to be misread. It is revenue at valley pricing, with no cost data, no named GPU and no independent check, so it says nothing firm about margin. The 155 trillion tokens and the 96% cache hit rate carry the same limit: they are DeepSeek’s own figures, and none of the sources we read includes an independent test.

Ascend users should note one caveat from the README: batch-invariant inference and deterministic sampling run only on NVIDIA. The release also includes deepseek-recipe, an MIT-licensed component that converts between chat API protocols and model token sequences. DeepSeek says its processing matches the official API, so third-party deployments should see the same behavior. We have not verified that claim.

What to watch is whether the KV cache and load-balancing code follows. The Q&A says it will be updated and that future inference code will keep landing in DeepInfer.

How to try it

DeepInfer’s offline example runs on Linux with GCC 13 or newer and Python 3.12. NVIDIA builds also need CUDA 13; Ascend builds need CANN 9.2. Create and activate an empty Python 3.12 environment, then run:

export DEEPINFER_ACCELERATOR=nvidia   # use "ascend" on Ascend hardware
./build.sh
python examples/offline_inference.py

The first run downloads the default checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017, into the Hugging Face cache. That repository holds 83 files and about 510 GB. If you already have a copy, set DEEPINFER_HF_MODEL_DIR to its path. The README’s benchmark charts come from 4 SM100 GPUs for prefill and 64 for decode, so a smaller test setup will not reproduce them.