arXiv cs.CL· Lujun Li, Lama Sleem, Yangjie Xu, Yewei Song, Aolin Jia, Jerome Francois, Radu State·· 2 天前AI 评分31
LLM-as-a-Judge 中温度设置的必要性研究
The Necessity of Setting Temperature in LLM-as-a-Judge
AI 导读
一项超百万次大规模评测研究发现，LLM-as-a-Judge 中低温度能带来一致且错误少的输出，尤其在小模型上，但一致性不等于判断可靠，低温度会收窄推理探索范围。研究通过四种推理人格刻画评判行为，发现高温重复采样结合多数投票，或高温推理加低温决策，可在不损害清晰样本的前提下提升模糊样本的一致性，代价是额外推理开销。
正文 · 原文
Abstract:Large Language Models (LLMs) are increasingly used as automated judges, yet decoding temperature is often set by intuition rather than evidence. We run over one million large scale evaluations, multiple open-source models, prompting strategies, and judging paradigms to examine how temperature affects judge reliability. Low temperatures yield consistent outputs with few errors, especially in smaller models, while higher temperatures reduce consistency and increase errors. Yet consistency alone does not guarantee sound judgments: low temperatures can narrow the range of reasoning explored. We characterize judge behavior through four reasoning personas and find that temperature shifts their distributions in model-dependent ways, most visibly in ambiguous cases where competing answers are hard to distinguish. Repeated high-temperature sampling reveals this hidden uncertainty, and combining it with majority voting or using high temperature for reasoning and low temperature for the final decision, improves agreement on ambiguous cases without hurting clear ones, at the cost of extra inference. Our findings suggest a simple rule: keep temperature low for stable outputs, but explore more when the judgment is hard.
| Comments: | 17 pages |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2603.28304 [cs.CL] |
| (or arXiv:2603.28304v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2603.28304 arXiv-issued DOI via DataCite |
Submission history
From: Lujun Li [view email]
[v1]
Mon, 30 Mar 2026 11:31:29 UTC (9,073 KB)
[v2]
Fri, 5 Jun 2026 10:06:49 UTC (7,243 KB)
[v3]
Thu, 15 Oct 2026 12:31:43 UTC (3,610 KB)
来源：arXiv cs.CL · arxiv.org