AI evals are becoming the new compute bottleneck

AI evaluation benchmarks have become prohibitively expensive, with costs ranging from thousands to tens of thousands of dollars per run. The Holistic Agent Leaderboard spent $40,000 evaluating 21,730 agent rollouts, while individual GAIA frontier-model runs cost $2,829; agent configuration choices alone drive 33× cost variation. This cost barrier is gatekeeping evaluation access and forcing reproducibility trade-offs.