Linkup Research Releases SPARSEUP, a 149M-Parameter Open-Source Sparse Embedding Model
Linkup Research has released SPARSEUP, an Apache 2.0 sparse embedding model built on a 149M-parameter ModernBERT backbone. Linkup reports 56.4 nDCG@10 on BEIR-13, placing it first among public sparse encoders under 150M parameters by its own account.
The weights are available on Hugging Face under Apache 2.0. The model loads through Transformers or Sentence Transformers with trust_remote_code=True, according to the report. Sparse models output weights over a vocabulary rather than a single dense vector, so each dimension maps to a real token; the vectors fit inverted indexes and can be read by humans, and they tend to match rare words well.
The release follows LightOn's DenseOn and LateOn models. LightOn published open data, a training recipe, a dense model and a late-interaction model, and SPARSEUP fills the sparse slot. Because it uses the same backbone family and fine-tuning data, the three retrieval styles can be compared side by side, Linkup said.
Training starts from the LateOn-unsupervised checkpoint. That checkpoint had no masked-language-modeling head, so the team grafted back ModernBERT's original one. Fine-tuning used LightOn's fine-tuning mixture with contrastive learning only. Each query receives seven hard negatives sampled from a pool of 50, plus in-batch negatives. There is no cross-encoder distillation, and Linkup said training fits on a single H100.
A vanilla SPLADE on this backbone produced large bags full of stopwords, according to Linkup. SPARSEUP addresses that with three changes. It shifts logits by computing log(1 + ReLU(x - 15)), because ModernBERT's MLM logits sat too high and saturated the log, making bags dense at initialization. It applies per-position top-k, keeping only each input token's 12 strongest vocabulary dimensions before max pooling, which caps expansion per token rather than total vector size. It also folds case variants: byte-level BPE stores heat, Heat, Ġheat and ĠHeat as separate ids, and SPARSEUP maps them to one id while keeping the largest weight. Output dimensions fall from about 50,000 to about 34,000. Queries and documents take [Q] and [D] prefixes and are scored by dot product, with evaluation maximum lengths of 128 tokens for queries and 512 for documents.
On BEIR-13 nDCG@10 without MS MARCO, SPARSEUP scores 56.4, ahead of opensearch-neural-sparse-encoding-doc-v3-gte at 54.6, opensearch-neural-sparse-encoding-v1 at 52.44, ModernBERT-VT at 52.4, SPLADE-v3 at 51.7 and granite-embedding-30m-sparse at 50.6, according to the model card. LACONIC-1B scores 58.7 but is a 1B-parameter model in a different size class. The controlled comparison is less favorable: with backbone and data fixed, LateOn scores 58.9, DenseOn 57.9 and SPARSEUP 56.4. SPARSEUP uses approximate Seismic search, while LightOn reports exact search, Linkup said. SPARSEUP wins ArguAna and Touché and beats DenseOn on HotpotQA, but lags on more semantic sets, with FiQA showing the largest gap; DBPedia is another weak spot. On decontaminated BEIR, the gap to DenseOn shrinks to 0.17 points. Linkup warned that decontaminated NQ and MS MARCO have only 21 and 46 queries, respectively, so those results are noisy.
On MS MARCO, SPARSEUP averages 47 non-zero terms per query and 190 per document, compared with 25 and 170 for SPLADE-v3. With the Seismic inverted index, it reaches more than 97% recall against exact search in about 380 microseconds per query, single-threaded, according to the report. Linkup said inflating vector size could add 1 to 2 BEIR points, but it chose to stay sparse.