LEANN Cuts 60M-Chunk RAG Index From 201 GB to 6 GB by Recomputing Embeddings at Query Time

LEANN is a low-storage approximate nearest neighbor index built on top of FAISS, aimed at a common problem: vector indexes are often larger than the data they index. The paper says a 76 GB corpus needs 173 GB of embeddings plus a 15 GB HNSW graph, more than doubling the raw data. LEANN keeps a pruned graph, drops the full vectors and rebuilds them on demand. The code is MIT licensed and installs with uv pip install leann. Its default embedder is Contriever, a 110-million-parameter model, and LEANN is an index rather than a large language model, so no context length applies.

To avoid stalling search, LEANN stores the graph plus a compact product quantization table and re-embeds only nodes on the search path. Two-level search ranks neighbors with cheap PQ distances first and recomputes exact vectors only for top candidates. Dynamic batching groups candidates from several hops, for example 64 at once, for GPU efficiency, while high-degree-preserving pruning keeps few edges for most nodes and more edges for the top 2% hub nodes. In ablations, two-level search gives an average 1.4x speedup and adding batching raises it to 1.8x; embedding recomputation still accounts for about 76% of query latency.

The research team tested NQ, TriviaQA, GPQA and HotpotQA against a 60-million-passage RPJ-Wiki datastore, measuring latency at 90% Recall@3. The figures are self-reported in the paper. On NQ with Qwen3-4B on an RTX 4090, LEANN and HNSW both reach 25.5% downstream accuracy; LEANN uses 4 GB of storage while HNSW uses 188 GB. BM25 scores 18.3% and PQ compression scores 17.9%. The tradeoff is search time: LEANN retrieval takes 2.48 seconds on NQ versus 0.05 seconds for HNSW, and with generation taking about 20 seconds, end-to-end latency rises from 20.95 seconds to 23.34 seconds. Across datasets on the RTX 4090, LEANN retrieval ranges from 1.12 seconds on GPQA to 7.12 seconds on HotpotQA, while an IVF-based recomputation baseline inspired by EdgeRAG is up to 200 times slower. LEANN improves exact match by up to 11.8% over BM25.

The paper tested an RTX 4090 machine with 32 GB of RAM and an M1 Ultra Mac. A CPU-only Linux install exists, but no CPU latency has been published. On the Mac, HotpotQA retrieval takes 44.80 seconds and accounts for 50.1% of total pipeline time. The paper's comparison table lists HNSW as out of memory on a 32 GB workstation while LEANN fits. The repository reports similar storage savings on personal data: 400,000 chat messages shrink from 1.8 GB to 64 MB, and 38,000 browser entries shrink from 130 MB to 6.4 MB.

The repository ships ready apps for documents, Apple Mail, Chrome history, WeChat, iMessage, and ChatGPT or Claude exports. MCP readers pull live data from Slack and Twitter bookmarks, and LEANN also runs as a semantic search MCP server for Claude Code. On 30 SWE-Bench Pro tasks from ContextBench, the team reports 24.2% initial relevant-code recall versus 11.4% for BM25, with the agent using 8.4% fewer tokens. The project says HNSW is the default backend, with DiskANN for faster search; generation works with Ollama, Hugging Face, Anthropic or any OpenAI-compatible API. It supports Ubuntu, Arch, WSL, macOS and Windows, while some apps such as Apple Mail search are macOS only.

Editor's Summary

LEANN is an MIT-licensed vector index from UC Berkeley's Sky Computing Lab that trades query-time embedding recomputation for much lower storage, cutting a 60-million-chunk index from 201 GB to 6 GB in the project's benchmark while matching HNSW's 25.5% NQ accuracy at 90% recall. The paper reports retrieval times of seconds, with generation dominating end-to-end latency in tested setups, and the repository extends the approach to local personal-data search and coding-agent tools. It won the Best Research Paper award at MLSys 2026.