arXiv cs.AI· Zhenxing Dou, Kun He, Mingyang Yi, Yong Liu·· 13 小时前AI 评分29
SRA：面向 LLM 推理增强的选择性残差适配框架
SRA: Selective Residual Adaptation for LLM Reasoning Enhancement
AI 导读
研究者提出即插即用框架 SRA，通过 Discrepancy Gate 定位师生模型推理路径上的稀疏分歧位置，并用 Candidate Residual Adapter 对 top-K logit 做校准，无需更新学生主干或运行时调用教师模型。在 Qwen3 0.6B 至 4B 的数学基准上，SRA 将蒸馏学生模型的平均 Pass@16 最高提升 4.37 个百分点，同时保持通用能力。
正文 · 原文
Abstract:Knowledge distillation transfers reasoning capabilities from large language models to lightweight students. However, distilled small models still struggle with multi-step reasoning. We observe that the distribution divergences between teacher and student models along reasoning paths are highly sparse, and target tokens almost always fall within the student top-$K$ candidate set. Based on this finding, unlike approaches updating student backbones or requiring runtime teacher computation, we propose Selective Residual Adaptation (SRA), a plug-and-play framework that internalizes localized corrections. Concretely, SRA employs a Discrepancy Gate to identify divergence positions alongside a Candidate Residual Adapter for top-$K$ logit calibration to correct the mismatch. Theoretically, we establish error bounds ensuring strict sequence level performance gains. Across mathematical benchmarks on Qwen3 models from 0.6B to 4B, SRA consistently enhances distilled student models, raising average Pass@16 by up to 4.37 percentage points while preserving general capabilities.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.34199 [cs.LG] |
| (or arXiv:2610.34199v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34199 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhenxing Dou [view email]
[v1]
Sun, 25 Oct 2026 10:30:10 UTC (2,208 KB)
来源：arXiv cs.AI · arxiv.org