arXiv cs.AI· Chu-Cheng Lin, Eugene Ie·· 11 小时前AI 评分31
模型该多快进入监督？用 Tsallis 损失连续体训练推理模型
How Fast Should a Model Commit to Supervision? Training Reasoning Models on the Tsallis Loss Continuum
AI 导读
研究者提出基于 Tsallis q-对数的损失族 J_Q，在 RLVR（q=0）与潜轨迹对数边际似然（q=1）之间连续插值，并证明两端的冷启动逃逸时间分别为 Ω(1/p₀) 与 Θ(log(1/p₀))，从而解释标准流程为何先 SFT 后 RLVR。
正文 · 原文
Abstract:The standard recipe for training reasoning models---SFT on rationale annotations followed by RLVR---relies on rationale annotations being available and clean. When no annotated rationales exist, SFT cannot start at all, and when the model is too unaligned, RLVR alone stalls at cold start; when annotations are weak, SFT memorizes the errors. Using the Tsallis $q$-logarithm, we define a loss family $J_Q$ that interpolates between RLVR (at $q{=}0$, the exploitation pole) and the log-marginal-likelihood over latent trajectories (at $q{=}1$, the density-estimation pole). The standard pipeline corresponds to a stepwise $q{=}1 \to q{=}0$ schedule, and intermediate $q$ enables training in regimes where the two-step pipeline cannot apply. All members share the same gradient direction, differing only by a per-instance amplification $P_{\theta}^{-q}$ that reweights each instance independently of the learning rate. Under gradient flow analysis, we show that the exploitation pole requires $\Omega(\frac{1}{p_0})$ time to escape cold start but is robust to label noise, while the density-estimation pole escapes in $\Theta\big(\log(\frac{1}{p_0})\big)$ but memorizes label noise. This separation explains why SFT ($q{=}1$) precedes RLVR ($q{=}0$) in the standard pipeline. We further derive two Monte Carlo estimators that directly optimize fixed-$q$ on the $J_Q$ continuum, without annotated rationales or external verifiers: Gradient-Amplified RL (GARL) and Posterior-Attenuated Fine-Tuning (PAFT), with shared bias $O\big(\frac{q}{M P_{\theta}^q}\big)$ but different variance and stability properties. On FinQA, HotPotQA, and MuSiQue, GARL at sufficiently high $q$ escapes cold start where GRPO fails entirely under Qwen 3 and Gemma 4 architectures.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2604.25907 [cs.LG] |
| (or arXiv:2604.25907v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.25907 arXiv-issued DOI via DataCite |
Submission history
From: Chu-Cheng Lin [view email]
[v1]
Tue, 28 Apr 2026 17:52:38 UTC (258 KB)
[v2]
Thu, 7 May 2026 17:16:40 UTC (256 KB)
[v3]
Fri, 30 Oct 2026 03:56:49 UTC (152 KB)
来源：arXiv cs.AI · arxiv.org