哪些成功轨迹适合做参考？多轮 on-policy 蒸馏中的动作序列对齐

Authors:Mufei Li, Nikki Lijing Kuang, Zhuoer Wang, Samson Koelle, Jie Yuan, Manqing Mao, Hong Wang, Yanjun Lin, James Feng, Daniel Edmiston, Pan Li, Wei Niu

View PDF HTML (experimental)

Abstract:On-policy distillation (OPD) reduces train-inference distribution mismatch by supervising a student on its own trajectories. For multi-turn tool-use agents, conditioning the teacher on a validated same-task successful trajectory as a reference can provide useful hindsight, but different successes may reach incompatible interaction states and offer misleading supervision. We cast successful-reference selection as a free-target optimal transport problem, which reduces to assigning each failed trajectory its most compatible successful reference. Because latent state paths are hidden, we use action-sequence alignment as an observable proxy for compatibility and instantiate it with longest common subsequence (LCS), which identifies a shared ordered backbone of actions while allowing recoverable detours. This perspective makes trajectory compatibility the central criterion for reference selection in multi-turn OPD. Experiments on ALFWorld and the airline domain of $\tau^3$-bench show that LCS is the only evaluated reference selector to improve over reference-free OPD on both ALFWorld overall scores and all three airline metrics. Without further optimization, the airline-trained models also transfer to the retail domain, where LCS improves all three metrics.

Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Cite as:	arXiv:2611.13135 [cs.LG]
	(or arXiv:2611.13135v1 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2611.13135 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Mufei Li [view email]
[v1] Mon, 9 Nov 2026 19:18:25 UTC (4,005 KB)