H Company Releases NeoMME Single-Tower Multimodal Encoders for Efficient Document Retrieval

The report says all checkpoints ship under Apache 2.0 with day-zero support in Hugging Face Transformers. Most production visual document retrievers are repurposed generative vision-language models whose causal decoder never generates a token; NeoMME instead processes multilingual text tokens and raw 32×32 RGB image patches through the same Transformer layers, trained from random initialization. Exact parameter counts are 262,937,906 and 793,715,032. Text enters through an ALBERT-style factorized embedding, while images are split into non-overlapping 32×32 patches and projected by a two-layer MLP trained from scratch; there is no patch-merging module or SigLIP2 tower. Both models support a 16,384-token context, described as enough for two standard 3,840×2,160 4K UHD images after patching.

According to the same report, most layers use symmetric sliding-window attention, with every sixth layer and the final layer attending globally. The stack also uses grouped-query attention, query-key normalization, gated attention, 2D rotary position embeddings, and squared-ReLU MLPs. The tokenizer is a whitespace-unconstrained BPE with a 131,072-entry vocabulary trained from scratch; across 14 target languages in the FLORES-200 devtest set, it emits 44.4% fewer tokens than ModernBERT.

Pretraining is discrete masked diffusion over text, optionally conditioned on visible image patches. Text-only segments draw a corruption rate uniformly from 0 to 1; multimodal segments draw from 0.30 to 1 to remove the language-only shortcut and force the model to read the page. At 90% masking, visible page patches raise masked-token accuracy by 38.4 points for the 260M model and 40.5 points for the 800M model, according to a cross-modal ablation probe. Each run processes about 524 billion packed input tokens, roughly 290 billion text-only, on 16 and 32 H100 accelerators for the two sizes, respectively.

For retrieval, NeoMME-Retriever adds a mean-pooled dense head with Matryoshka widths and a late-interaction head that projects every token and patch to 128 dimensions, returning both from one forward pass. The report says the 260M result on ViDoRe v3 is within 0.002 of the 3.75B-parameter ColQwen2.5-v0.2 and 26.1 points above the best other sub-300M model, while the 800M model trails the similarly sized Vultron Retriever Flash by 0.9 points. On ViDoRe v1 and v2, the 260M model scores 0.860 and 0.522 nDCG@5, and the 800M model scores 0.874 and 0.559. Text retrieval remains weaker: on BEIR-15, late interaction reaches 0.4881 and 0.5126, against LateOn’s 0.5722 at 149M parameters. The report attributes this difference partly to supervision scale, noting NeoMME saw roughly 430K text query examples versus roughly 660M contrastive examples for mLateOn.

Late-interaction indexes can still be expensive: a 2048×2048 page yields 4,162 vectors, or about 1.5 MB per ViDoRe v3 document in float32. Hierarchical token pooling at factor 10 with int8 queries and documents brings that to 39.0 kB per page (a 39.4× reduction retaining 99.16% of baseline nDCG@10), and pool factor 8 with int8 queries and binary documents gives 6.0 kB per page (a 255.5× reduction retaining 95.19%). At a matched 2048×2048 input on one NVIDIA L40S, NeoMME-260M encodes 51.3 pages per second versus 26.0 for ColModernVBERT, a 1.97× gap, and it encodes a query in 78.3 ms on a CPU-only host, according to the report. The report also lists text-only retrieval and frozen natural-image transfer as clear weak spots. A paper describing the models is posted on arXiv.