Hugging Face Launches NeoMME Multimodal Encoders Without Vision Tower or Causal Decoder
Hugging Face introduced NeoMME, a 260M/800M-parameter multimodal encoder using a single bidirectional Transformer over text and images, enabling efficient visual document retrieval with compact indexes.
NeoMME was fine-tuned for visual document retrieval using the ColPali-style page-image approach. The resulting NeoMME-Retriever returns dense and late-interaction embeddings in one forward pass. According to the blog post, both model sizes lie on the ViDoRe v3 Pareto frontier for nDCG@10 and model size. On an NVIDIA L40S GPU with a matched 2048×2048 image input, the 260M model encodes about 51 pages per second, roughly twice ColModernVBERT's throughput. Hierarchical token pooling and asymmetric quantization reduce late-interaction index storage from around 1.5 MB to 6 kB per page, a 255× reduction while retaining more than 95% of the baseline nDCG@10.
The architecture provides a native multimodal path: text uses factorized token embeddings, and images are split into 32×32 patches projected by a small MLP. Dynamic image resolution preserves aspect ratio and size, and the context length of 16,384 tokens can hold up to two 4K UHD images. The model also uses grouped-query attention, query-key normalization, 2D rotary position embeddings, and a multilingual BPE tokenizer with a 131k-token vocabulary.
Pretraining was done from scratch as a discrete masked-diffusion denoiser on multilingual text, code, mathematics, natural images, and document images. Each model processed about 524 billion packed input tokens, including 290 billion tokens from text-only examples. The authors chose the NorMuon optimizer to improve data efficiency given the relatively small text budget.
NeoMME is available in Hugging Face Transformers, and all model checkpoints are released under the Apache 2.0 license.