AI Benchmarking Lacks Standardization, Hugging Face Policy Chief Says

Hugging Face chief policy officer Irene Solaiman warned at TechCrunch Disrupt that AI model benchmarking remains an immature and inconsistent science. Key issues include poor reproducibility, difficulty comparing evaluations across tests, and a lack of standardization for embedded third-party evaluators. Solaiman also noted that evaluations are expanding beyond models themselves to include harnesses and agentic systems, with agentic evaluations described as especially difficult and costly. She cautioned that a weaker model with a better harness can outperform stronger models, underscoring the need for a more uniform, industry-wide evaluation approach.