Hugging Face's Solaiman Says Embedded Evaluators Need Standards

Irene Solaiman, chief policy officer at Hugging Face, used a TechCrunch Disrupt panel to pour cold water on the idea that embedded third-party auditors will fix AI oversight, TechCrunch reported. The qualifications for the job, she argued, barely exist yet.

"There's this hunger for an embedded evaluator ecosystem now, but the qualifications that determine who is an effective evaluator across what tasks need a lot more standardization," she said. Her reference point was Anthropic's hire of Accenture as its first embedded evaluator, a move the piece says "spurred a good amount of backlash, with some critics mocking the idea that the consulting firm was qualified to do the task."

Solaiman's broader claim is that evaluation itself is still unformed. Evals "are an evolving science — where we don't know how to meaningfully compare them to each other," she said, and "they start to get ineffective relatively quickly." Reproducibility and model access, in her telling, are still gating problems: "can you reproduce the results of an evaluation?" and "do you have the level of access that you need to the model itself — this is where open weights come in in a big way."

The agentic case is harder. "Agentic evals are really hard and really expensive," Solaiman said. "We have just more evaluations for models. … We don't have as many evaluations for harnesses."