DeepSeek open-sources DeepInfer inference framework

DeepInfer currently supports one model, DeepSeek-V4.1-Flash, and by default loads the updated 1017 version DeepSeek released over the weekend. The framework can run on Nvidia Corp.'s SM100 Blackwell GPUs and Huawei Technologies Co. Ltd.'s Ascend 950 neural processing units, but Hopper and other older Nvidia architectures are not on the list. Two features that keep outputs identical from run to run work only on Nvidia hardware. All published performance benchmarks ran on Nvidia chips, and the figures are DeepSeek's own.

DeepSeek described reaching a massive user base at a reasonable price as the core problem an inference framework has to solve, in a deployment Q&A posted with the code. One cluster now serves 155 trillion tokens a day, according to the same document, and revenue at off-peak pricing works out to $38 per graphics processing unit per hour.

Speculative decoding in DeepInfer runs through DSpark, a system in which a draft model proposes five tokens at a time and a confidence head predicts which of them the main model will accept. Verification compute goes only to the likeliest candidates. That trims the number of draft tokens checked per step by about 30%, and DeepSeek estimated a 4% to 7% throughput gain at its production load. A second technique, which the company says nearly halves total prefill computation, runs only the last 128 tokens of each prompt back through the model's decoder layers.

DeepSeek has not yet released its key-value cache management and load-balancing code, saying the two are tightly coupled to internal infrastructure. A blog post the company published on Zhihu lays out how both work. Splitting prefill and decode across separate machines is also left to operators, with the README noting that most of the engine work for it is already in place. The smallest deployment DeepSeek describes as efficient uses 64 GPUs for prefill and another 64 for decoding.

The same coupling kept DeepSeek from releasing its full engine in April 2025, when it described the software as an early fork of vLLM built around its cluster management tools. Maintaining a large open-source project was also more than a small research team focused on developing better models had bandwidth for, the company wrote at the time. DeepSeek had already released components such as the FlashMLA, DeepGEMM and DeepEP kernel libraries, which DeepInfer now uses. The new README names vLLM as its original foundation.

DeepInfer's Ascend support follows a Sept. 30 release in which DeepSeek brought the TileLang kernel programming language to Huawei's chips with the chipmaker's help. The startup plans to deploy at least 160,000 Ascend 950DT accelerators at a new data center in Inner Mongolia, Bloomberg reported in September.

The framework arrived a day after the U.S. Department of Homeland Security ordered federal civilian agencies to remove DeepSeek products from their systems, including third-party services built on its technology.

DeepInfer is available on GitHub under the Apache 2.0 license, and the companion deepseek-recipe library released with it converts requests into token sequences using the same logic as DeepSeek's official application programming interface. DeepSeek said inference code for future models will go into the same repository.

Editor's Summary

DeepSeek released DeepInfer under Apache 2.0, exposing the framework behind its production service while withholding key-value cache and load-balancing code. The release supports V4.1-Flash on Nvidia Blackwell and Huawei Ascend 950 hardware and follows a U.S. Department of Homeland Security order to remove DeepSeek products from federal civilian systems.