arXiv cs.CL· Ilija Subasic, Andrew Rabinovich, Zhao Chen·· 11 小时前AI 评分29
用分位数而非点值：面向序数感知 LLM-as-a-Judge 校准的流式分位数匹配
Quantiles, Not Points: Streaming Quantile Matching for Ordinal-Aware LLM-as-a-Judge Calibration
AI 导读
研究提出流式分位数匹配（SQM），通过计算 LLM 与人类评分分布间的最优传输映射来校准 LLM-as-a-Judge 分数，而非逐点映射。在自建评测框架中，SQM 尊重人类锚点集的序数性，在跨模型与数据上取得最佳分布保真度和最高综合排名。研究还发现事后校准普遍缩小了小型与大型评审模型之间的原始误差差距，为更高效的 LLM 评审模型提供了可能。
正文 · 原文
Abstract:LLM-as-a-judge scores are often distributionally shifted from human ratings on any given evaluation task, and post-hoc calibration (mapping LLM to human scores in a small labeled anchor set) has become a popular method to correct for this shift. In this paper we derive the calibration from two positions: that LLMaJ scores are ordinal ratings drawn from a distribution rather than isolated points. We present streaming quantile matching (SQM), a version of a classical method not previously bench marked for LLMaJ calibration, which is built on the two positions by computing the optimal-transport map between the LLM and human score distributions. We develop an evaluation framework and within it we show that unlike point-wise calibrations, SQM respects the ordinality of the human anchor set and achieves the best distributional fidelity and the top overall rank across models and data. We also show that in general, post-hoc calibration narrows the raw error gap between smaller and larger judge models, which opens the possibility of more efficient LLM judges.
| Comments: | 17 pages, 6 figures |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| ACM classes: | I.2.7 |
| Cite as: | arXiv:2611.00447 [cs.AI] |
| (or arXiv:2611.00447v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.00447 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ilija Subasic [view email]
[v1]
Fri, 30 Oct 2026 23:02:26 UTC (550 KB)
来源：arXiv cs.CL · arxiv.org