Hugging Face policy chief says embedded AI evaluators are no easy fix

AI labs have staked their reputations on model benchmarking—the supposedly neutral and accurate system of evaluations designed to measure whether one AI product is better than another. Solaiman said the methods behind those evaluations are complicated and still maturing, and that testing needs to become a more consistent science before the industry can rely on it.

Asked whether current tests can be trusted, Solaiman said trust is 'such a weighted term.' A major part of trusting results is whether an evaluation can be reproduced and whether researchers have the access they need to the model itself, she said, adding that open weights matter greatly in that context.

Reproducibility has long been one of the stickier parts of benchmarking and remains a subject of debate. Solaiman also raised the problem of comparing one evaluation with another. 'They are an evolving science—where we don't know how to meaningfully compare them to each other,' she said. Evaluations 'start to get ineffective relatively quickly,' she added, and the ecosystem is not as thriving as needed. There is 'this hunger for an embedded evaluator ecosystem now,' she said, but qualifications for who counts as an effective evaluator across tasks need much more standardization.

The idea of embedded evaluators—neutral third-party observers placed inside AI organizations—has gained attention over the last several months as AI labs confront safety incidents. Hugging Face was the target of one of those incidents, according to the report. Labs have pushed the third-party approach instead of active government regulation to identify and mitigate model safety concerns, but doubts remain about how neutral or effective such evaluators will be. Anthropic's decision to hire Accenture as its first evaluator drew backlash, with critics mocking the consulting firm's qualifications for the task.

Evaluations are also moving from tests of a model alone to tests of the entire system around it, including the harness and the agent. The harness—the tool a user uses to access a model—has a large effect on performance, Solaiman said; a weaker model with a better harness could easily outperform a frontier model. 'Agentic evals are really hard and really expensive,' she said. There are more evaluations for models than for harnesses, she added, and better evaluations for the holistic agent are being developed because that is where many users get value.

The evaluation space is 'a bit of a free-for-all' at the moment, according to the report, and if benchmarking is to remain useful to industry leaders, a more uniform, industry-wide approach may be necessary.

Editor's Summary

Hugging Face's Irene Solaiman warned that embedded AI evaluators face reproducibility, comparison and standardization problems, making them an uncertain answer to model safety concerns. She said evaluations are shifting toward testing entire systems, including harnesses and agents, while the field remains fragmented and costly to assess. The comments point to pressure for more consistent industry-wide benchmarking as labs face scrutiny over safety incidents.