Alibaba Papers Tackle Recurrent Transformer Redundancy Ahead of GPT-6 Hype

The report said GPT-6 Astra uses a technique called recurrent depth, in which the same set of Transformer layers is run multiple times to deepen computation without adding a comparable number of parameters. The Information first reported the move. The technique has drawn criticism from AI safety experts because computations occur in the hidden state, which is not necessarily written as a human-readable chain of thought and may therefore be harder to monitor. OpenAI's chief scientist, Jakub Pachocki, responded that he is preparing a detailed explanation of how it works.

Recurrent Transformers have been discussed as a possible way to scale models without continuously growing their size. But research has shown that under equal compute, recurrent models that save parameters often lag behind ordinary Transformers. The problem is what the Alibaba-led team calls "computational redundancy": later passes through the same layers can produce much smaller incremental changes. The team examined this phenomenon and published two solutions.

The first paper, MeSH, proposes managing information flows across cycles. The model adds a Memory Buffer composed of multiple slots to store the original input and partial results, together with two types of learned routers. A Write Router decides how each cycle's new output is spread across the slots, and a Read Router decides how much historical information each slot contributes to the next cycle's hidden state. This setup allows different cycles to specialize instead of recomputing similar patterns. In experiments at the Pythia-1.4B scale, weight sharing reduced non-embedding parameters by roughly 33%. With MeSH, the zero-shot average downstream accuracy rose to 50.56%, compared with 49.50% for a standard Transformer. The added router parameters equaled only about 0.005% of a standard Transformer's non-embedding parameters, and extra computational overhead was about 0.014%.

MeSH emerged from three observed symptoms of redundancy: the first core loop contributed most of the update while later loops changed the hidden state less; representations across consecutive cycles were highly similar; and the representations gradually collapsed into a low-dimensional space. The paper attributes these to a lack of computational differentiation and an overload of information in the hidden state.

The second paper, SpiralFormer, focuses on the resolution, or granularity, of computation. Instead of running every cycle on the full token sequence, it compresses adjacent tokens into shorter sequences and expands them cycle by cycle—for instance, starting at one-eighth the original length and moving through one-quarter, one-half, and finally the full length. With roughly the same parameter count as a standard Transformer at the Pythia-1.4B level, SpiralFormer-L used 13.13T FLOPs versus 14.08T for the baseline, while improving the 5-shot average downstream accuracy from 51.93% to 54.37%. Probing experiments showed that attention tended to be broader in early low-resolution cycles and gradually became more local, matching the intended global-to-local progression.

The two papers complement each other by introducing division of labor to recurrent computation: MeSH manages what information is carried between cycles and SpiralFormer decides the scale at which each cycle operates. Together, they suggest that recurrent Transformers can become more efficient and accurate when redundancy is systematically removed.