A joint paper from AI lab Cohere, Stanford University, MIT, and Ai2 accuses LM Arena — the organization behind the popular crowdsourced Chatbot Arena benchmark — of helping a select group of AI companies achieve better leaderboard scores at rivals’ expense.
According to the authors, LM Arena allowed certain industry-leading companies to privately test multiple AI model variants and decline to publish the scores of the worst performers. That made it easier for those companies to top the leaderboard, the authors say — an opportunity not every company enjoyed. “Only a handful of companies were told they could do this kind of private testing, and some got far more private tests than others. That’s gaming it,” Cohere AI research VP and co-author Sara Hooker told TechCrunch.
Chatbot Arena was created in 2023 by UC Berkeley as an academic research project and has since become the industry’s benchmark of record — the ranking every launch press release cites: two models’ answers shown side by side in “battles,” users pick the better one, and unreleased models often compete under pseudonyms. Votes accumulate into scores and rankings, and LM Arena has long insisted its benchmark is impartial and fair. The paper’s authors say that is not what they found. In the January-to-March window before Meta launched Llama 4, they report, Meta privately tested 27 model variants on Chatbot Arena over those three months — and at launch disclosed the score of exactly one model, which happened to top the leaderboard.
LM Arena co-founder and Berkeley professor Ion Stoica emailed TechCrunch that the study is full of “inaccuracies” and “questionable analysis.” In a statement, LM Arena said: “We are committed to fair, community-driven evaluation and invite all model providers to submit more models for testing to improve their human-preference performance. If one model provider chooses to submit more tests than another, that does not mean the second provider was treated unfairly.”
The Allegedly Favored Labs
The authors began the research in November 2024 after hearing that some AI companies might receive preferential access to Chatbot Arena, and over five months measured more than 2.8 million battles. They found evidence that LM Arena let certain companies — including Meta, OpenAI, and Google — collect more data from Chatbot Arena by having their models appear in more battles, and claim this elevated sampling rate conferred an unfair advantage. Extra data from the arena can lift a model’s performance on Arena Hard, another LM Arena-maintained benchmark, by 112 percent — though LM Arena posted on X that Arena Hard performance does not directly correlate with Chatbot Arena performance. Hooker says it remains unclear how certain companies got preferential access, but that LM Arena owes the community transparency either way.
LM Arena pushed back that some claims do not reflect reality, pointing to a blog post (published earlier that week) showing models from non-major labs appeared in more battles than the study claims. A significant limitation of the research: it relied on “self-identification” to determine which models were being privately tested — the authors repeatedly prompted models about their parent company and classified them by the answers, a method that is not foolproof. Still, Hooker says, when the authors contacted LM Arena to share their preliminary findings, the organization raised no objections. TechCrunch contacted Meta, Google, OpenAI, and Amazon for comment; none responded immediately.
LM Arena in Hot Water
The paper calls for changes to make Chatbot Arena fairer: clear, transparent limits on how many private tests a lab may run, and public disclosure of those tests’ scores. LM Arena rebutted the recommendations on X, claiming it has published pre-release testing information since March 2024 — and that “showing scores of unreleased pre-release models is meaningless” since the community cannot test them anyway. The researchers also proposed equalizing sampling rates so every model in the arena fights the same number of battles; LM Arena publicly adopted that suggestion and says it will create new sampling algorithms.
The paper landed weeks after Meta was caught benchmark-gaming around its Llama 4 launch: one Llama 4 model was tuned specifically for “conversational” performance, which produced an impressive Chatbot Arena ranking — but the tuned model was never released, and the original version scored far worse on the leaderboard. At the time, LM Arena said Meta should have been more transparent about its benchmarking methods. And earlier this month, LM Arena announced it was forming a company to raise funding from investors — a move that dramatically heightens the scrutiny on private benchmark organizations generally, and on whether they can be trusted to evaluate AI models free of corporate influence. An update to the coverage noted a Google DeepMind engineer disputing parts of the research as inaccurate — without denying, as Cohere claimed, that Google sent ten models for pre-release testing between January and March; he noted only that the open-source team responsible for Gemma sent one.
The deeper takeaway for anyone using leaderboards to choose models: evaluation infrastructure is itself a power structure. Whoever controls sampling rates, private-testing policies, and disclosure timing shapes which models look best — and the Chatbot Arena saga proves that even community-driven, academically-born benchmarks are subject to those pressures. The pragmatic defense remains triangulation: arena votes, objective suites, and real-usage trials, weighted toward the last — because a leaderboard can be gamed with private tests, but a workflow that runs on your own data cannot.
