相似模型，失真的赔率：Luce 公理在 Chatbot Arena 中的失效之处

View PDF HTML (experimental)

Abstract:Chatbot Arena ranks language models with the Bradley-Terry model, which gives each model one score and so assumes Luce's choice axiom. Choice theory predicts where this fails: near-copies are told apart more decisively than their scores imply. On 1.8 million Arena votes, votes between versions of one model are 1.37 times as decisive as Bradley-Terry predicts, between sizes of one generation 1.26 times and between models of one lab 1.12 times; a release a year later gives consistent multipliers, and the effect survives alternative tie treatments, prompt subsets and topics, and timing checks. How alike two models' answers are worded and formatted tracks the effect. A correction with four to six parameters predicts held-out votes better than Bradley-Terry and than models with hundreds of parameters, mostly by modelling ties; frozen and carried to new releases in a test fixed in advance, its gains are concentrated in ties, and better head-to-head odds are not established. The leaderboard order barely moves. With language-model judges and three-answer menus, a near-copy takes votes mostly from its twin when choices are pooled over prompts (log-odds shift +0.10, 95% interval 0.05 to 0.16); within a prompt the shift is smaller and clear for two of four judges.

Comments:	18 pages, 5 figures, 9 tables
Subjects:	Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as:	arXiv:2610.19095 [cs.CL]
	(or arXiv:2610.19095v1 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2610.19095 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Soovadeep Bakshi [view email]
[v1] Tue, 13 Oct 2026 19:33:25 UTC (170 KB)