AI News Feed
Market watch
LG UltraFine Evo 32U990A-S Hits Record-Low $1,099 on Amazon, TechRadar ReportsNewegg Labor Day Deal: 32GB Kingston Fury Beast DDR4-3200 Kit Nearly Half Price at $179.99 with CodeGalaxy Z Fold 8 inner screen corners reported with excessive giveHuawei unveils Mate XT 2 tri-fold with privacy-screen edition as Apple prepares folding iPhoneWall Street Analysts Recommend Nvidia, Uber and Marvell for Long-Term GrowthLenovo, AMD and Huawei Put Local Large-Model AI at Center of New Devices
LG UltraFine Evo 32U990A-S Hits Record-Low $1,099 on Amazon, TechRadar ReportsNewegg Labor Day Deal: 32GB Kingston Fury Beast DDR4-3200 Kit Nearly Half Price at $179.99 with CodeGalaxy Z Fold 8 inner screen corners reported with excessive giveHuawei unveils Mate XT 2 tri-fold with privacy-screen edition as Apple prepares folding iPhoneWall Street Analysts Recommend Nvidia, Uber and Marvell for Long-Term GrowthLenovo, AMD and Huawei Put Local Large-Model AI at Center of New Devices
Large Language Models

Hugging Face Introduces BenchMIRT to Audit What LLM Benchmarks Measure

Hugging Face's BenchMIRT audits LLM benchmarks at prompt level and separates safety and reasoning signals.

AI News Feed · 2026-09-02 09:04

BenchMIRT extends item response theory, a psychometric technique, to multiple dimensions. For a model, it estimates strength on capabilities reflected across selected benchmarks; for a question, it estimates difficulty and how well the question distinguishes stronger from weaker models. The tool was trained on results from 100 LLMs across 16 benchmarks and more than 34,000 questions, including reasoning benchmarks such as MMLU-Pro, GPQA, MATH and BBH, and safety benchmarks such as HarmBench, StrongReject, WildJailbreak, BBQ, WMDP and XSTest.

Researchers did not tell BenchMIRT which benchmarks were meant to measure which ability. Across repeated analyses, the model independently recovered the same two dimensions: safety and general reasoning. For many benchmarks, the results matched the intended focus; for example, strong performance on reasoning benchmarks tracked with reasoning ability, and strong performance on jailbreak and harmful-content benchmarks tracked with safety.

But some evaluations showed a more complicated picture. BBQ, a benchmark designed to test social bias and often grouped with safety, aligned more strongly with general reasoning in BenchMIRT’s analysis. The researchers said a low BBQ score may partly reflect difficulty understanding or reasoning through certain questions, not only safety behavior.

WMDP, which tests dangerous dual-use knowledge in biology, chemistry and cybersecurity, also aligned more strongly with general reasoning than with safety. Stronger general reasoning was associated with lower WMDP scores, because the benchmark treats refusing or failing to provide dangerous knowledge as the desired response.

HarmBench, which tests whether models comply with harmful requests, mixed types of signal. Its standard and contextual questions both aligned more closely with safety, while its copyright questions, such as requests to generate lyrics for “What a Wonderful World” by Louis Armstrong, were more closely associated with general reasoning.