Google Research's R4T Compiles Query Fan-Out Into a 53.9M Diffusion Retriever

Search and recommendation systems increasingly need to return a set of results rather than one best match. A query such as “camping gear” should return a tent, a sleeping bag, a stove, and a headlamp, not ten near-identical tents. Query fan-out splits a broad prompt into sub-queries, but the research team identifies two problems when a generic large language model performs this at inference time. The first is paraphrastic collapse. For “Bohemian festival style,” zero-shot Qwen3-4B wrote “bohemian festival fashion” and “festival bohemian clothes,” near-synonyms that retrieve a homogeneous slate. The second is latency. Autoregressive generation plus repeated retrieval calls is slow, and Best-of-N sampling improves quality while multiplying inference cost.

R4T addresses this with a three-stage pipeline. In fan-out language model training, a FOLM generates k sub-queries, a frozen dense retriever executes them, and a set-level reward scores the whole retrieved set rather than each item alone. In supervision synthesis, the trained FOLM samples 128 fan-outs per query at temperature 0.9, producing (query, target set) training pairs without human labels. For open-ended tasks, targets are retrieved-content embeddings; for compositional tasks, targets are sub-query embeddings. In diffusive retriever training, a 53.9-million-parameter diffusion transformer learns to map a query embedding to a full set of target embeddings using a variance-exploding formulation within the EDM framework. At inference, it generates all embeddings in a single non-autoregressive pass, and nearest-neighbor search maps each embedding to database items.

For open-ended abstract retrieval, or OAR, the reward combines three weighted terms. Groundedness, with λg = 0.6, penalizes distance between each sub-query embedding and its nearest database item. Diversity, with λd = 0.2, uses the Vendi Score over representative retrieved items such as the top-1 item per sub-query. Alignment, with λa = 0.2, measures mean cosine similarity between each sub-query and the original query. For weakly supervised compositional retrieval, or WSCR, the reward is the fraction of reference-set items the fan-out retrieves. An ablation explains why all three OAR terms matter. With groundedness alone, Gemma3-4B converged to strings such as “line ending line ending line ending.” Adding alignment made collapse even faster, as the policy repeated paraphrases of the query. Adding diversity closed both shortcuts. Training uses GRPO with soft PPO regularization, which adds forward and reverse KL penalties. Key settings include group size 8, learning rate 1×10⁻⁷, and global batch size 512.

Experiments used the Polyvore fashion outfit dataset with a CLIP-based matryoshka encoder at 128 dimensions, and a proprietary expert-playlist music dataset with MuLan embeddings. Every fan-out method produced k = 10 sub-queries, and Best-of-N used N = 5. OAR quality was scored by an LLM judge on 5-point Likert scales. On Polyvore, Gemma3-4B R4T-FOLM averaged 49.1, versus 40.9 for Best-of-N and 38.5 zero-shot. Diversity rose from 56.0 zero-shot to 76.8, and R4T-Diffusion retained most of it at 74.3. On Music, Gemma3-4B R4T-FOLM averaged 58.1 versus 49.2 for Best-of-N. Groundedness is not reported for R4T-Diffusion because it produces no text sub-queries.

WSCR results on Polyvore show a coverage and diversity trade-off. R4T-FOLM with Qwen reached 20.9 Recall@5K and 64.6 Hit@5K, versus 15.7 and 52.1 for Gemini-2.5-Flash. Its Vendi Score, however, dropped to 27.5. The authors link this to reduced output entropy under strong RL optimization. R4T-Diffusion with Qwen kept a higher Vendi Score of 34.7 with 16.5 Recall@5K.

On efficiency, at batch size 8, autoregressive fan-out took about 1.46 seconds, while the diffusion model took 0.07 seconds. At batch size 1024, autoregressive fan-out reached nearly 50 seconds, versus 4.21 seconds for diffusion. The authors report a consistent 12x to 20x speedup. R4T runs RL once offline, then deploys the 53.9-million-parameter diffusion retriever; the three rewards of groundedness, diversity, and alignment are designed to block reward hacking and paraphrastic collapse.

Editor's Summary

Google Research's R4T framework uses offline reinforcement learning to train a fan-out language model and then distills its behavior into a compact diffusion retriever that generates all query fan-out embeddings in one pass. Reported benchmarks show improved set-level retrieval quality on Polyvore and a music dataset, alongside a 12x to 20x speedup over autoregressive fan-out. The work points to a more efficient way for search and recommendation systems to diversify results without repeated autoregressive inference.