Zhipu AI's GLM-5.3-Flash Matches Kimi K3's Key Parameter, Highlighting Deeper Tech Convergence

According to a Leiphone report, GLM-5.3-Flash uses a hybrid architecture that combines DeepSeek's sparse attention mechanism (NSA) with Moonshot AI's linear attention, and also incorporates DeepSeek's manifold-constrained hyper-connection (mHC). The design is intended to cut inference costs while maintaining long-context performance. Chen Guangyu, known for his paper on attention residuals, examined the model's configuration file and found that both GLM-5.3-Flash and Kimi K3 adopt KDA linear attention with nearly identical layout logic. The "gate_lower_bound" parameter is set to -5 in both applications. The public technical report for Kimi K3 was released less than a month before Zhipu's new model appeared, prompting some observers to marvel at the pace of iteration.

The -5 value is not arbitrary. Linear attention replaces full KV caching with a fixed-size state matrix, using a forget gate to control how quickly old information decays. This makes memory use nearly constant regardless of context length, enabling low-cost million-token APIs. But the recurrence in linear attention can cause numerical overflow or underflow, leading to NaN crashes during training. Previous work on gated linear attention (GLA) proposed a lower bound for state decay. Moonshot AI later adapted this into KDA and, during large-scale pre-training on half-precision hardware, discovered that -5 is the optimal lower bound after extensive engineering trials. After activation, -5 corresponds to about 0.67% retention, meaning the model is forced to keep a tiny fraction of old information even when it decides to forget. A higher bound like -3 causes excessive accumulation and gradient instability; a lower bound like -8 leads to severe memory loss. -5 sits at the sweet spot.

For Kimi K3, the gate_lower_bound=-5 was part of the original design from the first day of pre-training, as documented in the public report. Chen Guangyu suggested that in GLM-5.3-Flash, the parameter may have been injected midway through training, a common strategy where a gating constraint is added in the later stages of pre-training and the model is fine-tuned with a reduced learning rate. This approach avoids restarting from scratch and can align the model with industry-validated practices, saving substantial computation costs. A developer identified as Oliver Sieberling commented that this is a "non-invasive change" because the forget gate learns adaptively and adding a lower bound only clips extreme values. However, Chen noted a potential risk of hidden distribution shift: after tens of millions of training steps, locking the lower bound could disturb the model's internal memory distribution, potentially causing subtle logical errors in extreme long-tail scenarios such as ultra-long code and documents.

Not all Chinese labs have taken this path. DeepSeek has pursued independent research with its MLA and NSA mechanisms, achieving numerical stability through low-rank compression and hardware-level operator optimization. The broader shift to operator-level tuning is driven by the economics of reasoning models. With models generating long chains of thought, traditional attention's quadratic scaling would make inference costs prohibitive. Linear attention offers a path to keep long-context performance affordable, and as teams converge on similar model scales and training frameworks, the mathematically optimal range for the gate parameter naturally narrows. The convergence on -5 is therefore seen as a sign of industry maturity rather than a simple copy.

The Leiphone report concludes that in the industrial world, what matters is not whether Zhipu referenced Kimi's parameters, but that the search for the best engineering practices has pushed China's top AI teams to the same narrow solution space. The competition has moved to specifics like operator implementation, numerical boundaries, and training engineering details. This is reflected in how quickly Chinese open models have caught up with international closed source models in long-context processing and low-cost inference.