The Chatbot You Talk To Is Not the One the Benchmarks Measured

Tell the same quarrel twice: once in your own words, and once as the other person would tell it, so that this time you are the one in the wrong. A fair judge should not tell both of you that you were right. Researchers at Stanford put 100 such pairs, taken from the Reddit forum where people ask whether they were in the wrong, to Claude Haiku 4.5. Through the API, the programming door used by developers and by the people who compute benchmark scores, the model sided with the teller in both versions about a quarter of the time. In the ordinary Claude chat window, with the same stories, it did so more than half the time.

The study is a preprint by Jennifer Wang, Joachim Baumann, Daniel E. Ho and Sanmi Koyejo, first posted on arXiv on 8 September and revised on 8 October. Between March and May 2026 they sent the same items from nine tests, covering general ability, social bias, flattery of the user and invented answers, to seven systems from OpenAI, Anthropic and Google, five times each through both doors. They opened new accounts, used a fresh browser for every round and a new chat for every question, switched off memory, history and tools, and held each pair of requests at a barrier so that the API call and the chat-window message left within seconds of each other, leaving no room for a quiet model update in between.

The API came out ahead for all seven systems, by 3.4 percentage points of accuracy on average, and the chat window was also less steady: it changed its answer more often when the same question came back. The averages hide sharper cases. On a test of everyday common sense, which asks who a pronoun refers to in a sentence, GPT 5.4 Thinking answered 94% correctly through the API and 75% in the ChatGPT window. The authors’ summary is blunt: “switching access surfaces can degrade performance as much as downgrading a full model version.” The order between models does not survive the move either. On that same common-sense test Claude Sonnet 4.6 beat the smaller Haiku 4.5 by five points through the API; in the chat window, Haiku edged ahead.

The authors then tried to rebuild the window from outside. Chat products quietly add their own instructions before every conversation, and versions of those instructions have leaked and circulate publicly; the researchers added them to the API requests. The answers became nearly as unsteady as the window’s, but the accuracy gap barely moved, from 6.5 to 6.4 points on their measure. Changing the settings that make answers more or less random made no difference in any of eight tests, and changing the amount of reasoning mattered in one. With Claude Opus 4.6 on the quarrel test, the leaked instructions even pushed the API the other way, making it less inclined to agree with both sides. Between the model and the person typing sit layers the authors list as routing, retrieval, tool rules, personalisation and post-processing, and none of them can be inspected from outside. The paper declines to name a single culprit.

A benchmark score, then, is a measurement of one particular door. Launch pages and leaderboards are built from numbers taken through the API; most people meet the model through the window, with layers of its own. The authors ask that every score state which door it was taken through. They also note, among their limitations, that platform terms of service may restrict the browser automation an audit like theirs depends on. The window most people use is also the one outsiders are least free to measure.

Not checked: whether the gaps hold for today’s models. The systems tested are a generation old, measured between March and May, and all three companies have released newer models since. The authors themselves cannot confirm that the chat window and the API ran the same underlying model on each request; their matching is, in their words, “necessarily approximate.” They used one subscription tier per platform (ChatGPT Enterprise, Claude Pro, Google AI Pro), so other users may see different gaps, and standard tests rather than real conversations. Part of the grading was done by another model, checked by hand on 1,280 answers with 98.7% agreement. On the quarrel test GPT 5.3 Instant went slightly the other way, though the difference was not statistically significant. It is a preprint, not yet peer reviewed. I read the full paper and checked that the released audit tool is online; I did not rerun anything.