arXiv cs.CL· Feifei Qian, Xuan Chen, Lixin Cui, Chaoyang Mei, Fanlin Meng, Shuyao Xiao, Ming Cui, Junxi Yin, Lu Bai·· 6 天前AI 评分24
从结构到效度：机制引导的 LLM 评分标准生成与精炼
From Structure to Validity: Mechanism-Guided Generation and Refinement of LLM Rubrics
AI 导读
研究对 LLM 评分标准设计开展机制分析，发现结构化组织能构建更全面的评分准则，仅靠语义规范不足以保证准则级判断有效，复杂分数聚合带来的额外收益有限。基于此提出 ARTS-Hybrid，将自适应树构建与语义优先、响应校准的精炼相结合，在多个基准上同时提升端到端评估表现与评分标准本身的质量，其生成的评分标准还能为模型优化提供有效信号。
正文 · 原文
Abstract:Rubrics have been widely used to evaluate and optimize language models on open-ended tasks. However, existing work primarily focuses on generating better rubrics, with limited understanding of what makes rubric design effective. In this work, we conduct a systematic mechanistic analysis of rubric design across criterion organization, criterion operationalization, and score aggregation. Our analysis shows that structural organization enables more comprehensive criterion construction, semantic specification alone does not ensure effective criterion-level judgments, and sophisticated score aggregation provides limited additional gains. Guided by these findings, we introduce Adaptive Rubric Tree Search with Hybrid refinement (ARTS-Hybrid), which combines adaptive tree construction with semantic-first, response-calibrated refinement. Experiments across benchmarks show that ARTS-Hybrid improves both end-to-end evaluation performance and intrinsic rubric quality. Further experiments suggest that, beyond evaluation, rubrics generated by ARTS-Hybrid provide effective signals for model optimization.
| Comments: | Submitted to ICLR 2027 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.13375 [cs.CL] |
| (or arXiv:2610.13375v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13375 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xuan Chen [view email]
[v1]
Fri, 9 Oct 2026 10:48:51 UTC (605 KB)
来源：arXiv cs.CL · arxiv.org