arXiv cs.CL· Haocheng Yang, Licheng Pan, Yiming Dong, Xiaoxi Li, Yuan Lu, Zhenhao Chen, Jiajun Fan, Shuai Shao, Chunyuan Zheng, Hao Wang·· 3 天前AI 评分22
并非所有标准都同等重要：面向评分标准式 LLM 评估的重要性感知标准加权
Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation
AI 导读
针对评分标准式 LLM 评估中统一加权会高估次要、重叠标准的问题，研究者提出重要性感知标准加权（ICW），将 LLM 对全部标准的整体判断拆解为一系列两两重要性比较，检查并修正不一致比较，再从比较矩阵的主特征向量推导权重。实验显示 ICW 持续优于统一加权及竞争性加权方法，并在下游强化学习中转化为更好的策略模型。
正文 · 原文
Abstract:Rubric-based evaluation assesses the responses of large language models (LLMs) by aggregating the verification results of explicit criteria, yet not all criteria matter equally. However, existing methods commonly aggregate the verification results with uniform weighting, which assigns minor and overlapping criteria the same weight as essential ones and thus produces misleading evaluation scores. Although some methods prompt an LLM to directly assign weights, direct weighting requires a holistic judgment over all criteria that LLMs struggle to make reliably. To address these limitations, we propose Importance-Aware Criteria Weighting (ICW), which derives the weights of criteria from pairwise importance comparisons made by an LLM. Specifically, ICW transforms the holistic judgment over all criteria into a series of pairwise importance comparisons, checks and revises inconsistent comparisons, and derives the weights from the principal eigenvector of the comparison matrix. Extensive experiments demonstrate that ICW consistently outperforms uniform weighting and competitive weighting methods, and translates into better policy models in downstream reinforcement learning.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19915 [cs.CL] |
| (or arXiv:2610.19915v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19915 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Haocheng Yang [view email]
[v1]
Wed, 14 Oct 2026 10:05:43 UTC (506 KB)
来源：arXiv cs.CL · arxiv.org