arXiv cs.AI· Wei Chen, Yubing Wu, Shigui Li, Delu Zeng, Jingfeng Liu, Junmei Yang, Zhiheng Zhou, Bo Li, Min Chen, Zhou Wang·· 13 小时前AI 评分26
RubricOpt：让策略自我修订语义评分标准以优化偏好训练
RubricOpt: Self-Revision of Semantic Rubrics for Preference Optimization
AI 导读
RubricOpt 框架让策略在偏好优化过程中自行诊断无评分标准时的输出，保留已覆盖的要求并修订未解决反复失败问题的条款，再由修订后的评分标准在训练中引导 DPO，评估时移除。在能力与开放式生成基准上，评分标准条件化可优于普通 DPO，但增益与退化因模型和数据集而异；匹配的延续实验显示该编辑带来的增量效果不一。
正文 · 原文
Abstract:A model's failures can reveal gaps in the behavioral criteria guiding its training. We introduce RubricOpt, a framework that lets a policy revise these criteria within preference optimization. The policy diagnoses its own rubric-free outputs, retains requirements already covered by the rubric, and edits clauses that leave recurring failures unaddressed. A shared semantic representation connects task demands, preference-data contrasts, and rubric clauses, while a deterministic gate checks evidence references and fixed semantic anchors. The rubric then guides Direct Preference Optimization (DPO) during training, with preference pairs, labels, and the objective held fixed, and is removed at evaluation. Our analysis relates the local effect on rubric-free success to semantic alignment and a residual beyond the measured factors. Experiments on capability and open-ended generation benchmarks show that rubric conditioning can improve outcomes over ordinary DPO, with gains and regressions varying across models and datasets. A matched continuation isolates the contribution of revision from additional training and finds mixed incremental effects for the tested edit, whose feedback overlaps the evaluation set. RubricOpt thus brings the behavioral specification itself into the policy's feedback loop.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.29491 [cs.AI] |
| (or arXiv:2610.29491v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29491 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Wei Chen [view email]
[v1]
Wed, 21 Oct 2026 09:46:37 UTC (246 KB)
来源：arXiv cs.AI · arxiv.org