arXiv cs.CL· Dmitrij \.Zatuchin·· 12 小时前AI 评分31
研究：查询语言显著影响 LLM 回复的自动情感评分
Query Language Sensitivity in Automated Sentiment Scores of LLM Responses
AI 导读
一项基于 12,933 条 LLM 回复的研究发现，查询语言可解释自动情感评分拟合方差中的 26.49%，品牌仅占 1.46%。剔除英语后重新拟合，语言占比降至 5.07%、品牌升至 2.23%，总拟合方差下降 65.08%。英语 1,620 条回答中有 591 条被分类器判为负面，其余七种语言 11,313 条中仅 77 条，作者强调这不能证明英语回答对品牌更负面。
正文 · 原文
Abstract:Automated scores are often used to summarize how large language models describe brands across prompts, providers, and languages. Such scores can vary because the answers differ, because the scoring instrument behaves differently across texts or languages, or for both reasons. We examine this measurement problem in 12,933 responses from a design crossing 20 stored brand labels, eight query languages, three provider aliases, and 15 prompt identifiers. The outcome is the signed winning-class confidence from a multilingual sentiment classifier: neutral winners are coded zero, while positive and negative winners receive signed confidence. A crossed linear mixed model fitted to all responses assigns 26.49% of fitted variance to query language and 1.46% to brand. An exploratory refit after excluding English assigns 5.07% to language and 2.23% to brand. These percentages have different denominators: total fitted variance falls by 65.08%, and absolute brand variance falls by 46.65%. English contains 591 negative classifier scores among 1,620 answers, compared with 77 among 11,313 answers in the other seven languages. The result shows that the original variance decomposition is highly sensitive to English inclusion. It does not identify a causal English effect or establish that English answers are more negative toward the brands. The classifier was trained for tweet sentiment, receives no separate target-brand condition, and English is the only study language represented in its documented sentiment fine-tuning set. Automated score differences should therefore be checked as properties of both the answers and the measurement instrument before they guide substantive conclusions.
| Comments: | 11 pages, 2 tables, 1 figure. Corrected analysis of stored automated sentiment scores, with exploratory language-exclusion sensitivity. Removes unsupported brand-ranking and query-budget prescriptions. No new human validation |
| Subjects: | Information Retrieval (cs.IR); Computation and Language (cs.CL) |
| ACM classes: | H.3.3; G.3 |
| Cite as: | arXiv:2607.13304 [cs.IR] |
| (or arXiv:2607.13304v2 [cs.IR] for this version) | |
| https://doi.org/10.48550/arXiv.2607.13304 arXiv-issued DOI via DataCite |
Submission history
From: Dmitrij Żatuchin [view email]
[v1]
Tue, 14 Jul 2026 22:12:52 UTC (46 KB)
[v2]
Sun, 1 Nov 2026 09:19:00 UTC (43 KB)
来源：arXiv cs.CL · arxiv.org