arXiv cs.CL· Mingyeong Kim, Gyusang Cho, Juho Lee, Giung Nam·· 3 天前AI 评分20
面向分布式连续隐式推理的流匹配原理分布蒸馏
Rationale Distribution Distillation via Flow Matching for Distributional Continuous Latent Reasoning
AI 导读
研究者提出原理分布蒸馏（RDD），通过流匹配从多条文本推理路径中蒸馏，训练一个在定长隐序列上的条件流，并用 within-query mixup 在完整隐式推理之间插值以扩展蒸馏目标。该流非自回归地生成整段隐序列，再由冻结的语言模型自回归解码答案。实验显示，这一分布式推理器在答案覆盖率上优于所评估的点值隐式推理基线，在高等数学任务上尤为明显。
正文 · 原文
Abstract:Sampling multiple chain-of-thought rationales from a language model can uncover correct solutions missed by a single attempt. However, these rationales are typically lengthy, making repeated autoregressive generation costly. Continuous latent reasoning offers a compact alternative, but point-valued methods condition repeated answer sampling on a single rationale, which can limit exploration of diverse paths and answer coverage. We thus propose rationale distribution distillation (RDD), training a conditional flow over fixed-length latent sequences by distilling from multiple textual reasoning paths. Furthermore, we expand the distillation targets through within-query mixup, interpolating between complete latent rationales for the same query. The flow nonautoregressively generates the entire latent sequence, and a frozen language model autoregressively decodes answers from sampled rationale contexts. Empirically, the resulting distributional reasoner achieves higher answer coverage than the evaluated point-valued latent reasoning baselines, particularly on advanced mathematical tasks.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.19871 [cs.CL] |
| (or arXiv:2610.19871v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19871 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Giung Nam [view email]
[v1]
Wed, 14 Oct 2026 09:30:04 UTC (441 KB)
来源：arXiv cs.CL · arxiv.org