arXiv cs.CL· Zijian Zhang, Rizhen Hu, Jiangweizhi Peng, Shiyang Li, Sujay Bhatt, Sihan Zeng, Mingyi Hong·· 10 小时前AI 评分38
AutoScaffold：AI 引导的脚手架用于语言模型强化学习
AutoScaffold: AI-Guided Scaffolding for Reinforcement Learning of Language Models
AI 导读
AutoScaffold 是一个 AI-for-AI 框架，由冻结的教师模型周期性诊断学生训练经验中的瓶颈，自动设计跨任务策略与单任务提示两类脚手架，并在评估时移除。
正文 · 原文
Abstract:Reinforcement learning with verifiable rewards (RLVR) produces rich training experience, yet outcome rewards reduce each attempt to a single number and leave most of this information unused. Training-time scaffolds, such as strategies or hints added to the prompt during training, offer a natural way to turn this information into guidance, but their design and adaptation are largely left to humans and tailored to specific domains. We introduce AutoScaffold, a unified AI-for-AI framework that turns a student's training experience into adaptive scaffolds, with minimal domain-specific information and no human intervention during training. A frozen teacher periodically examines this experience, diagnoses bottlenecks, and designs scaffolds in two forms: (i) strategies shared across a class of tasks and (ii) hints for individual hard tasks, adjusting the content and amount of support as the student learns. The scaffolds are removed at evaluation. We apply the same framework to ALFWorld, search-based QA, mathematical reasoning, and GPU kernel generation. When evaluated without scaffolds, students trained with AutoScaffold outperform those trained with the corresponding RL baselines in all four domains, with gains of 7.2 points in ALFWorld success and 4.0 points in average QA exact match. On GPU kernel generation, AutoScaffold increases the fraction of tasks with a correct kernel that runs faster than the PyTorch reference by up to 5.0 points. On math, our student surpasses QuestA's reported accuracy using approximately one-third as many student training steps. Analyses show that the teacher identifies a different failure in each task class and shifts its scaffolds from step-by-step workflows to targeted fixes as the student improves, and that its revised scaffolds help the student learn more than keeping the previous scaffold or using no scaffold.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.14515 [cs.LG] |
| (or arXiv:2611.14515v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14515 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zijian Zhang [view email]
[v1]
Tue, 10 Nov 2026 18:58:46 UTC (251 KB)
来源：arXiv cs.CL · arxiv.org