arXiv cs.AI· Azwar Abdulsalam, Nishil Patel, Andrew Saxe·· 4 小时前AI 评分38
RL 后训练如何构建组合式推理策略
RL Post-Training Builds Compositional Reasoning Strategies
AI 导读
在完全可观测的重写语法环境中,RL 后训练能让 Transformer 解决预训练模型即便加大采样预算也极少解决的留出问题,而拒绝微调(RFT)早期有提升但随后停滞。
正文 · 原文
Abstract:Does RL post-training merely amplify primitive skills already latent in a base model, or can it compose primitive skills into new higher-level strategies? We study this question in a fully observable rewrite-grammar environment where the pretraining distribution is known and every generated rewrite can be audited. A Transformer is pretrained on primitive symbol-rewrite chains and post-trained on a Trace-based reasoning task with only a binary final-answer reward. RL solves held-out problems that remain rarely solved by the pretrained model even under much larger sampling budgets, while rejection fine-tuning improves early but plateaus. Trace analysis shows that RL reorganizes primitive competence through a phased compositional mechanism: it first strengthens primitive reductions, then discovers valid composed procedures. These include sequential compositions, which collapse ordered chains of primitive contractions, and parallel compositions, which combine independent primitive contractions in a single step. The composed procedures are not isolated samples; they are reused and consolidated into a stable repertoire. Comparing RL with rejection fine-tuning shows that the key difference is not exploration volume but selectivity: RFT produces many shortcut-like rewrites, much of them invalid, whereas RL concentrates exploration into valid reusable structure. Pretraining ablations show that the emergence of compositional strategies is gated not by primitive exposure alone, but by whether pretraining organizes primitive competence into reduction procedures that RL can later compress. The base model provides weak procedural ingredients; RL builds them into reliable higher-level strategies.
| Comments: | Accepted at NeurIPS 2026 (poster) |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.07646 [cs.AI] |
| (or arXiv:2607.07646v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.07646 arXiv-issued DOI via DataCite |
Submission history
From: Nishil Patel [view email]
[v1]
Wed, 8 Jul 2026 17:04:42 UTC (1,054 KB)
[v2]
Tue, 17 Nov 2026 17:20:07 UTC (1,042 KB)
来源:arXiv cs.AI · arxiv.org