Stanford and UC Berkeley Researchers Release TokenGate for Faster Long-Context LLM Prefill

TokenGate is a drop-in component that adds trainable token pruning to pretrained LLMs. The implementation adds a small MLP gate and LoRA with rank 16 to Qwen3-1.7B or Qwen3.5-2B and was tested on contexts up to 128k tokens. Paper timings use one A100 at batch size 1, while training used four A100s.

The design targets a core cost of transformer inference: every token passes through every layer, and attention cost grows quadratically with sequence length. Training-free pruning is cheap but loses accuracy at high pruning rates, the researchers say, while architectural changes often require pretraining from scratch.

A forward pass splits into three phases. Layers below a gate layer L process all N tokens as usual. At layer L, an MLP scores each token from its post-attention residual, and a top-K selector keeps the highest-scoring tokens. Layers above L then process only those K tokens and never create KV entries for pruned ones. Only the document part of the prompt is prunable; the question is never pruned.

Top-K selection is not differentiable, which blocks end-to-end training. TokenGate borrows a Mixture-of-Experts router trick: each kept token’s score becomes a bias on attention logits above the gate, giving gradients a path back to the scorer. A learnable per-head gain and a learnable temperature refine the mechanism, with the temperature correcting for each query seeing fewer keys. Training uses plain next-token loss and averages a full-context loss and a random-budget loss each step, so one trained gate works across budgets.

The team tested Qwen3-1.7B on RepoBench-P, 2WikiMultihopQA, MultiNews, BioDEX and needle-in-a-haystack (NIAH). Baselines were FastKV, SlimInfer, DASH, GemFilter, FlexPrefill and MInference on the same LoRA backbone. Against full prefill, TokenGate delivered 1.4x to 11.3x faster TTFT and 1.5x to 3.4x lower memory; against the next-best baseline, it was 1.2x to 6.4x faster on TTFT and used up to 2.1x less memory. The team describes the results as Pareto-optimal on five tasks against six baselines.

The best reported case was NIAH at 16k tokens, with 11.3x faster TTFT and 3.42x lower memory. TTFT gain reached 13.0x at 128k, while the worst speedup dropped to 1.4x on some tasks. On 2WikiMultihopQA, at 95% of full-context accuracy, prefill ran 3.0x faster with 2.7x less memory. A gate trained at 16k reached 12.8x at 32k and 13.0x at 64k and 128k, with memory savings staying near 3.43x. The learned selector beat FastKV’s heuristic at the same layer, and even random-K selection scored higher on the TokenGate backbone, which the team credits to backbone co-adaptation.

On hybrid-attention models, Qwen3.5-2B mixes Gated DeltaNet and full-attention layers. On NIAH 32k, TokenGate at layer 3 matched full-prefill accuracy and ran 3.8x faster than full prefill and 2.1x faster than FastKV. Gating at layers 1 or 2 reached 7.8x while staying within 6% accuracy. FastKV can only prune attention layers, the researchers note.

The GitHub repository requires Python 3.10+ and CUDA PyTorch 2.9+. An optional FlashAttention-2 key-bias kernel speeds up dense Qwen3 at 1.0x to 1.1x the cost of a plain call; without it, inference falls back to FlexAttention. A {state} placeholder marks the prunable document, and generate takes a max_kept_prefill_tokens budget. Checkpoints cover all five tasks on Qwen3-1.7B plus a Qwen3.5-2B NIAH gate, with an example at liana313/tokengate-niah-qwen3-1.7b. Training mirrors a TRL/PEFT recipe: each step is 1.3x slower than plain LoRA, runs converged faster on four of five tasks, and mean training time was 1.8 versus 2.2 A100 hours.

The reported speedups are self-reported and use different setups, so they are not directly comparable across systems, according to MarkTechPost.