arXiv cs.CL· Asal Mehrabi, Mohammad Aliannejadi, Clemencia Siro·· 13 小时前AI 评分34
研究揭示 LLM-as-a-Judge 提示策略效果取决于评审模型：覆盖 27 种语言的多语言研究
Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness
AI 导读
一项覆盖 9 种提示策略、4 个开源评审模型、27 种语言、4 类任务和 13 个数据集的多语言研究发现，LLM-as-a-Judge 的提示策略不可迁移：评审模型的选择主导提示策略的效果，策略效果在不同评审模型间甚至会方向反转。
正文 · 原文
Abstract:LLM-as-a-Judge evaluation is increasingly applied to multilingual tasks, yet prompting strategies such as Chain-of-Thought and Role Conditioning are recommended based solely on English benchmarks. Whether their effectiveness transfers across judge models and languages remains unknown: prompt-sensitivity studies are largely English-centric, while existing multilingual judge studies fix the prompt template. In this work, we address this gap by measuring this interaction through a benchmark of 9 prompting strategies, 4 open-weight judges, 27 languages, and 4 tasks across 13 datasets, yielding over 1M pointwise scores on gold-reference outputs, validated against human judgments. We find that prompting strategies are not transferable: the choice of the judge model dominates the choice of prompting strategy, strategy effects reverse direction across judges, and stability degrades in low-resource languages, where these effects are concealed by aggregate reporting. These findings show that prompting effects in multilingual evaluation are conditioned on the judge, task, and language. Consequently, prompting strategies should be validated for the intended evaluation setting, and cross-study comparisons should account for both judge and prompt configuration.
| Comments: | 23 pages, |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.31172 [cs.CL] |
| (or arXiv:2610.31172v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.31172 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Clemencia Siro [view email]
[v1]
Thu, 22 Oct 2026 12:27:05 UTC (3,176 KB)
来源：arXiv cs.CL · arxiv.org