Contrastive-LM Releases CLM-8B, an Open Model That Scores Agent Actions Instead of Generating Text

Jev entered limited early access on 15 September 2026 and returns typed values with probabilities rather than text. CLM targets the same interface. The project's GitHub repository serves CLM-8B behind a TypeSafe-compatible API and exposes three question types: Noul, which returns the probability that a statement is true; Choice, which selects one option from a declared set with probabilities; and Score, which returns an expected level on an ordered rubric. A request written for TypeSafe's API can be replayed through CLM's Python client.

Deployment is lightweight. The Apache-2.0 head weighs 75 MB and runs on a single NVIDIA GPU under Linux, with vLLM serving the Qwen3-8B encoder.

CLM trains a state encoder and an action encoder with a bidirectional InfoNCE loss. Each encoder is a frozen Qwen3-8B backbone plus a 20-million-parameter trainable projection head. Training pulls each state toward the action actually taken and pushes it away from the others. At inference, the model scores each candidate by the dot product of the state and action embeddings, and a softmax over those scores becomes the answer distribution. The same primitive ranks best-of-N solutions, routes tools and answers typed decisions.

The design disaggregates states and actions. In an agent loop the state changes every step while the action set stays mostly fixed, and clm-serve reserves a slab of GPU memory similar to vLLM's KV cache and reuses cached vectors. On one RTX 4090 with three actions, revisited states drop from 1.7 ms to 0.6 ms. The model card reports CLM running 13 times faster than Jev with about 1,000 candidates.

Training proceeds in three stages: pre-training on roughly 60 million Nemotron DQA question-answer pairs, mid-training on about 30 million synthetic hard negatives generated by Gemini 2.5 Flash-Lite, and post-training on roughly 1 million agent trajectories from Agent Data Protocol, Endless-Terminals and LiteCoder-Terminal-SFT. On about 100,000 held-out questions, pre-training alone reaches 52.1 percent top-1 accuracy and mid-training lifts it to 69.2 percent. Training on hard negatives from the start peaks at 62.4 percent and then overfits.

In zero-shot comparisons, CLM-8B completes the T-Rex game in 16.5 ms against Jev's 149.8 ms, with both scoring 5 out of 5. On BFCL v4 tool calling it takes 76.8 ms against 125.5 ms but trails on success, 95.2 percent to 99.2 percent. On WikiRacing it runs in 79.8 ms against 225 ms and reaches 26 of 30 against Jev's 30 of 30. On Super Mario it records 33.5 ms against 132.6 ms, with both at 5 out of 5. The nine-fold figure comes from the T-Rex game, where actions repeat across states.

As a verifier for coding agents, a generator samples several candidate solutions and the verifier selects one. Opus 5 produced DeepSWE candidates in a best-of-4 setup and Fable 5 produced Terminal-Bench 2.1 candidates in a best-of-5 setup. The team evaluated 38 held-out DeepSWE tasks and 30 held-out Terminal-Bench 2.1 tasks, measuring latency on an H100. DeepSWE's pass@1 was 73.7 percent, with CLM at 81.6 percent and Jev at 71.1 percent; CLM ran in 79 ms against Jev's 449 ms. Terminal-Bench 2.1's pass@1 was 84.0 percent, with CLM at 87.6 percent and Jev at 83.1 percent; CLM ran in 32 ms against 131 ms. Jev scores below pass@1 on both benchmarks, so selecting with Jev is worse than taking a single sample, while CLM runs 4.1 to 5.7 times faster. The research team reports these as new state-of-the-art verifier results, but notes that the numbers use lightweight fine-tuned heads rather than the zero-shot checkpoint and come from held-out subsets, not full leaderboard submissions.