arXiv cs.CL· Jialin Yang, Jiankun Wang, Jiajun Wu, Henry Leung, Jiayu Zhou, Steve Drew·· 12 小时前AI 评分31
D-RPC：通过推理空间压缩实现结构化理由蒸馏
Structural Rationale Distillation via Reasoning Space Compression
AI 导读
研究者提出 D-RPC（Distillation through Reasoning Path Compression），通过让教师模型遵循一个动态维护的可复用高层推理路径库来生成更一致的推理理由，从而降低蒸馏中的监督噪声。
正文 · 原文
Abstract:When distilling reasoning from large language models (LLMs) into smaller ones, teacher rationales for similar problems often vary wildly in structure and strategy. Like a chef who makes the same dish differently each time, this inconsistency burdens the student with noisy supervision that is hard to internalize. We propose Distillation through Reasoning Path Compression (D-RPC), which constrains the teacher to follow a compact, dynamically maintained bank of reusable high-level reasoning paths. For each training question, D-RPC retrieves the most relevant path and conditions the teacher to follow it, producing rationales that are consistent across similar problems yet diverse enough to cover different problem types. A PAC-Bayes analysis formalizes a trade-off between bank size and coverage, with smaller banks limiting path diversity at the risk of coverage gaps. Our ablations favor an intermediate bank size, consistent with this trade-off. Across five math and commonsense reasoning benchmarks with two student models, D-RPC achieves the highest average accuracy among the evaluated distillation methods, while using fewer tokens than template-heavy alternatives.
| Comments: | NeurIPS 2026 |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.07139 [cs.CL] |
| (or arXiv:2605.07139v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.07139 arXiv-issued DOI via DataCite |
Submission history
From: Jialin Yang [view email]
[v1]
Fri, 8 May 2026 02:15:52 UTC (549 KB)
[v2]
Wed, 4 Nov 2026 02:18:35 UTC (546 KB)
来源：arXiv cs.CL · arxiv.org