arXiv cs.AI· Tiago Martinelli, Jo\~ao Paulo Papa, Adriano Jos\'e Pereira·· 3 天前AI 评分22
蒸馏前先审查：将推理能力迁移到紧凑医疗语言模型的受控工作流
Audit Before You Distill: A Controlled Workflow for Reasoning Transfer into Compact Medical Language Models
AI 导读
研究提出 Distill-SFT 方法，将答案与推理依据合并为单一训练目标，在巴西葡萄牙语住院医师考试题上仅用更少训练样本，就在此前推理答错的题目上略优于逐步蒸馏。
正文 · 原文
Abstract:Teacher rationales can help a smaller model answer health professions questions, but they can also reinforce errors. We examined when they help and how this evidence can guide training using Brazilian Portuguese questions from medical and multiprofessional residency examinations. First, we measured how a teacher rationale changed the student's probability of selecting the correct answer and compared this effect with rationales drawn from other questions. The complete rationale increased this probability by 21~p.p. when the teacher was correct and decreased it by 14~p.p. when the teacher was wrong. By intervening on the reasoning content, we showed that shuffled rationales did not produce the same pattern. Guided by this result, we trained only with rationales attached to correct teacher answers and compared answer-only training, step-by-step distillation, and our proposed Distill-SFT method. Distill-SFT showed a modest advantage over step-by-step distillation on previously missed questions under reasoning, while using fewer training instances by combining the answer and rationale into a single target. We also investigated catastrophic forgetting using the out-of-domain ENEM dataset. Answer-only training reduced reasoning accuracy by 12~p.p. relative to baseline, compared with approximately 4~p.p. for either rationale-based training method. Together, these findings support a workflow that audits rationale content before distillation, verifies teacher answers before supervision, and trains compact models efficiently for health professions question answering.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.16713 [cs.CL] |
| (or arXiv:2610.16713v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.16713 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tiago Martinelli [view email]
[v1]
Mon, 12 Oct 2026 12:47:07 UTC (97 KB)
来源：arXiv cs.AI · arxiv.org