arXiv cs.AI· Yubing Wu, Wei Chen, Junmei Yang, Delu Zeng, Tiewen Pan·· 13 小时前AI 评分29
DPO 计算图的随机剪枝：训练提速 1.4 倍并降低峰值内存
Stochastic Pruning of the DPO Computation Graph: 1.4x Faster Training and Lower Peak Memory
AI 导读
研究者提出 DPO 计算图的随机剪枝框架，在打分、序列执行和偏好分支反向传播前分别独立采样响应位置、响应前缀和偏好分支，从而跳过各阶段计算。分支剪枝相比拼接式 DPO 基线实现 1.426 倍单步提速，在两个模型族上训练时间较原生 DPO 减少约 30%，但平均准确率略降、开放式评分随基准变化。位置与序列剪枝主要降低峰值内存。
正文 · 原文
Abstract:Direct preference optimization (DPO) incurs training costs in output scoring, sequence execution, and backpropagation, so reducing supervision alone does not necessarily save computation. We introduce a unified framework for stochastic pruning of the DPO computation graph that independently samples response positions for scoring, response prefixes for execution, and preference branches for backpropagation. By making these selections before the corresponding operations, the framework omits computation at each stage of a training step. Our analysis characterizes the objective changes induced by position and sequence pruning and uses inverse-probability weighting to obtain a conditionally unbiased single-branch gradient estimator for the sampled objective. Controlled measurements show that position and sequence pruning primarily reduce peak memory, with most output-memory savings attributable to an implementation that retains full response supervision. Branch pruning yields a $1.426\times$ step-time speedup over a concatenated DPO baseline and reduces training time by about 30\% relative to stock DPO across two model families, with slightly lower mean accuracy and benchmark-dependent changes in open-ended scores. These findings relate the resource benefits of stochastic graph pruning to the computation omitted and the resulting changes in the training objective.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.29538 [cs.LG] |
| (or arXiv:2610.29538v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29538 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yubing Wu [view email]
[v1]
Wed, 21 Oct 2026 10:25:15 UTC (191 KB)
来源：arXiv cs.AI · arxiv.org