DeepSeek open-sources DeepInfer inference engine behind DeepSeek-V4.1-Flash

DeepInfer is a high-performance inference engine for DeepSeek’s own models, according to its GitHub README. This release covers offline inference with core features needed at scale. It was originally built primarily on vLLM, with ideas from SGLang and TensorRT-LLM. The release supports one model, DeepSeek-V4.1-Flash, and a 1M-token context window per the model card. It runs on NVIDIA SM100 Blackwell GPUs and Huawei Ascend 950 NPUs. Benchmarks used four GPUs for prefill and 64 for decode. DeepSeek says one cluster serves 155 trillion tokens per day.

The engine stacks DeepSeek kernels including FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels. Attention uses data parallelism plus Wide-EP; MoE layers use expert parallelism with an expert-parallel load balancer. Decode runs EP64 to EP128 inside a supernode, while prefill uses the smallest EP that memory allows, typically EP2 to EP8. DSpark, a confidence-scheduled speculative decoder, has a draft model propose five tokens per step; a Confidence Head predicts each token’s acceptance rate and the scheduler verifies only the most valuable prefix. DeepSeek says this cuts verified draft tokens by about 30%, from 5.00 to 3.2-4.4, while accepted length barely moves, 3.29 versus 3.20.

On a cache hit, DeepInfer’s sliding-window-attention bounded replay recomputes only the last 128 tokens of attention state, making recomputation cost constant regardless of context length. DeepSeek says this nearly halves total prefill compute in decoder layers. CPU batch scheduling overlaps with GPU execution, so the GPU does not idle between steps. Batch-invariant inference and deterministic sampling are available on NVIDIA only. Structured outputs support regex, JSON schema and EBNF grammars.

At 131,072 context, DeepSeek published Pareto curves for V4.1-Flash. Prefill ran on four SM100 GPUs and decode on 64. In decode with EP64DP64, nextn=3 and 75% acceptance, throughput is above 25,000 output tokens/s/GPU at 256 requests per rank, near 100 tokens/s per user; at eight requests, per-user speed passes 300 tokens/s. In prefill with EP4DP4 and 140-token bounded replay, throughput is about 60,000 non-cached tokens/s/GPU at 4,096 tokens per rank and nears 90,000 at 65,536 tokens, with roughly 730 ms per step. An EP8 versus EP64 chart shows EP64 ahead at every batch size. DSpark’s dynamic verification adds 4% to 7% throughput at about 60 sequences per rank and is 1.5% slower than fixed five-token verification at 16 sequences per rank.

The release is offline inference only. A full service still needs a KV cache system, VE/Prefill/Decode disaggregation and load balancing across data-parallel replicas; DeepSeek says those components are tightly coupled to internal infrastructure. Hardware is narrow: builds require Linux, GCC 13 or later, Python 3.12, and CUDA 13 or CANN 9.2. The smallest efficient production unit is 64 prefill GPUs plus 64 decode GPUs. DeepSeek says its internal FlashMLA decoding kernel is 5% to 10% faster than the public one, and the end-to-end gap is 5% or less.

Compared with vLLM, SGLang and TensorRT-LLM, DeepInfer has one supported model and a narrower hardware matrix, while all four carry Apache 2.0 licenses. DeepInfer includes expert parallelism with EPLB and Wide-EP, DSpark speculative decoding, no P/D disaggregation in this release, and offline inference only; the others list broader model coverage, hardware support and online serving, according to the comparison published by MarkTechPost.