arXiv cs.CL· Jinyuan Li, Chengsong Huang, Langlin Huang, Donghong Cai, Shiping Gao, Yuyi Yang, Jiaxin Huang·· 10 天前AI 评分28
推理模型自我进化为何崩溃？R-Quest 用问题有效性与新颖性反馈维持十轮稳定提升
Questioning the Questions: Sustaining Self-Evolution in Reasoning Models
AI 导读
针对自我进化推理模型多轮自训练后性能崩溃的问题，研究者提出 R-Quest，用问题有效性与新颖性反馈引导进化：先训练求解器识别并拒绝无效问题，再以其判断引导出题方奖励并过滤训练数据，同时用冻结基座模型比较问题对以提供新颖性反馈。该方法在数学推理、通用推理和代码生成共 12 个基准上取得最高平均性能，十轮自我进化中性能持续上升并在末轮达到峰值，超过 R-Zero 17.32 分。
正文 · 原文
Abstract:Self-evolving reasoning models learn from their own generated questions, yet repeated self-training can lead to performance collapse. In this paper, we investigate why performance deteriorates over successive rounds and how to sustain self-evolution. Our analysis identifies two recurring quality problems in self-generated questions: invalid questions and repeated variants of the same mathematical questions. First, invalid questions become more prevalent across rounds, and answer-consistency filtering further increases their proportion in training data. Second, existing question diversity controls based on lexical similarity can miss mathematically equivalent questions expressed in different ways, which leads to question diversity collapse in later training rounds. Building on these findings, we introduce R-Quest, which uses question validity and novelty feedback to guide self-evolution. We first train the solver to recognize and reject invalid questions, then use its judgments to guide questioner rewards and filter solver training data. To avoid question repetition, we use a frozen base model to compare sampled question pairs and provide novelty feedback. Empirically, our method consistently achieves the highest average performance on 12 benchmarks in mathematical reasoning, general-domain reasoning, and code generation across two model families. Additionally, R-Quest maintains stable performance gains over ten rounds of self-evolution, peaking in the final round and outperforming R-Zero by 17.32 points.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.04299 [cs.LG] |
| (or arXiv:2610.04299v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.04299 arXiv-issued DOI via DataCite |
Submission history
From: Jinyuan Li [view email]
[v1]
Sat, 3 Oct 2026 05:24:47 UTC (16,059 KB)
[v2]
Tue, 6 Oct 2026 21:32:46 UTC (16,059 KB)
来源：arXiv cs.CL · arxiv.org