DeepSeek open-sources its production inference framework

DeepSeek has released DeepInfer, the inference framework powering its own production services, under the Apache 2.0 license on GitHub. The framework currently supports one model, DeepSeek-V4.1-Flash, and runs on Nvidia SM100 Blackwell GPUs and Huawei Ascend 950 NPUs. One cluster using the system serves 155 trillion tokens per day. Key optimizations include a speculative decoding system called DSpark, which cuts draft tokens checked per step by 30%, and a prefill technique that nearly halves total prefill computation. DeepSeek has not yet released its key-value cache management and load-balancing code due to tight coupling with internal infrastructure.