arXiv cs.AI· Katy Blumer, Jon Kleinberg, Ravi Kumar, Andrew Tomkins·· 14 小时前AI 评分38
研究发现 LLM 对短内容存在稳定主观偏好，提问方式变化即可改变选择结果
Bouba Beats Kiki, and Other Regularities in the Subjective Preferences of LLMs
AI 导读
一项将发表于 NeurIPS 2026 的研究考察 LLM 在无明确最优答案时如何在多个可行回答间做选择，聚焦数字、婴儿名和虚构城市三类短内容，识别出长度、字母顺序、流行度、元音构成、发音等 29 个语法与语义维度。
正文 · 原文
Abstract:Users engage with large language models on tasks that are purely objective ("When was Netflix founded?"), purely subjective ("What should I name my child?") or anywhere in-between. We have rich tools for evaluating model accuracy with respect to objective criteria. However, when it comes to subjective aspects of a task, we do not have similarly developed methodology at hand. In this paper, we take some early steps towards framing the question of subjective evaluation. Rather than measure how often a model is right or wrong on a question, we instead ask, how do models make choices among multiple viable responses without a clear winner?
To study this question, we restrict our focus to short-form content, and consider comparison tasks among three such content types: numbers, baby names, and fictional cities. We identify 29 aspects of these short-form examples covering many areas of syntax and semantics (length, alphabetic order, popularity, vowel mix, spoken sound, etc.). We show first that models tend to have strong and reproducible preferences for or against some of these aspects. Second, we show that variations in how the model is asked to express a preference may cause some of these aspects to activate or deactivate in the model's selection. Third, we show that overall preference distributions among pairs may vary radically based on seemingly-unimportant variations in the nature of the question posed to the model, raising new questions about manipulation. And finally, we show that, if models are used both to create lists of alternatives and to rank them, there are unexpected interactions between the two stages that may also drive the models' stated preferences.
| Comments: | 10 pages, 5 figures in main text; 27 pages, 13 figures in supplement. To appear at NeurIPS 2026 |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.37692 [cs.CL] |
| (or arXiv:2610.37692v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.37692 arXiv-issued DOI via DataCite |
Submission history
From: Katy Blumer [view email]
[v1]
Wed, 21 Oct 2026 01:37:54 UTC (4,386 KB)
来源：arXiv cs.AI · arxiv.org