arXiv cs.AI· Dongyang Chen, Shudong Liu, Shubo Li, Duoxun Tang, Jinwei Liang, Zehuan Wu, Lewei Lu·· 6 天前AI 评分22
BranchOPD：分支式在线策略蒸馏
Branching On-Policy Distillation
AI 导读
BranchOPD 通过学生模型不确定性和师生分歧选择决策点，从共享前缀重采样学生策略续写，并对主轨迹与分支同时施加 token 级教师监督。
正文 · 原文
Abstract:On-policy distillation (OPD) provides teacher supervision on student-generated trajectories, but a single trajectory realizes only one continuation at each decision, limiting supervision coverage under a finite sampling budget. We propose BranchOPD (Branching On-Policy Distillation), which selects informative decision points using student uncertainty and teacher-student disagreement, resamples student-policy continuations from shared prefixes, and applies token-level teacher supervision to both main trajectories and branches. BranchOPD operates at token-level decisions in mathematics and within-turn decisions in agent tasks. Shared-prefix reuse reduces student-generated token costs by approximately 19% on ALFWorld and 31% on WebShop relative to seven independent trajectories. We also introduce protocol-boundary repair (PBR), which uses targeted supervision under protocol constraints to mitigate repetition and wasted generation. BranchOPD outperforms Vanilla OPD and Multi-Rollout OPD (n=7) across all evaluated benchmarks, with gains over Vanilla OPD of 18.55 percentage points in mean mathematics accuracy over 16 sampled answers (avg@16) and 13.8 points in mean ALFWorld ID/OOD success. Mechanistic analyses and selection ablations show that branching at uncertain decisions expands subsequent behavior coverage and outperforms random selection and turn-initial branching.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.13481 [cs.AI] |
| (or arXiv:2610.13481v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13481 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Dongyang Chen [view email]
[v1]
Fri, 9 Oct 2026 12:13:34 UTC (1,304 KB)
来源：arXiv cs.AI · arxiv.org