arXiv cs.CL· Xiuyi Lou, Zicheng Xu, Yu-Neng Chuang, Hoang Anh Duy Le, Zhaozhuo Xu, Guanchu Wang, Vladimir Braverman·· 2 天前AI 评分23
TACO：面向 RLVR 的尾部感知信用校准方法
Right Answer, Wrong Tokens: Tail-Aware Credit Calibration for RLVR
AI 导读
针对 GRPO 类方法中"正信用污染"问题，研究者提出尾部感知信用校准方法 TACO，让策略自身作为参考，评估每个采样 token 相对策略局部预期的稀有度以计算尾部风险分数，并据此软性降低高尾部风险 token 的正信用、同时保留非零权重。在四个 LLM 和八个推理基准上，TACO 持续优于 GRPO 类基线，并提升训练稳定性，支持长程 RL 的持续性能增益。
正文 · 原文
Abstract:Although reinforcement learning (RL) improves the reasoning capabilities of large language models (LLMs), GRPO-style methods can suffer from a positive-credit contamination failure mode. Specifically, uniform credit assignment can reinforce locally erroneous tail tokens alongside useful tokens within successful trajectories. To address this problem, we propose Tail-Aware Credit calibratiOn (TACO), which calibrates the positive credit assigned to such tokens without token-level error labels. To this end, TACO lets the policy serve as its own reference, assessing each sampled token's rarity relative to the policy's local expectations to compute a tail-risk score. Using this score as a proxy for potential local errors, TACO softly reduces positive credit for high tail-risk tokens while retaining nonzero weights, limiting their reinforcement without eliminating opportunities to learn useful rare patterns. Experiments across four LLMs and eight reasoning benchmarks show that TACO consistently improves over GRPO-style baselines. Notably, TACO also improves training stability, supporting sustained performance gains in long-horizon RL. The source code is available at: this https URL.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.07976 [cs.CL] |
| (or arXiv:2607.07976v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.07976 arXiv-issued DOI via DataCite |
Submission history
From: Zicheng Xu [view email]
[v1]
Wed, 8 Jul 2026 23:00:54 UTC (1,901 KB)
[v2]
Thu, 15 Oct 2026 04:57:05 UTC (3,203 KB)
来源：arXiv cs.CL · arxiv.org