arXiv cs.AI· Debjyoti Saha Roy·· 10 天前AI 评分22
先刻画再蒸馏：大输出空间中的机制推理
Characterize Then Distill: Mechanistic Reasoning in Large Output Spaces
AI 导读
研究揭示推理训练语言模型如何在数千至数十万个候选中零样本完成多标签选择：在 MIMIC-IV 临床编码任务上，5,651 个候选诊断码全部置于上下文时，一组小型、全局、分阶段的注意力头对每个摘要的决策既必要又充分。作者提出 MISTILL 蒸馏方法，除教师推理文本外还监督学生模型在这些决策事件上的池化注意力，在跨家族学生模型中使对比决策的因果恢复近乎翻倍。
正文 · 原文
Abstract:Reasoning-trained language models can perform, zero-shot, multi-label tasks that require selecting a small set of relevant labels from a universe of thousands to hundreds of thousands of candidates. We ask how they do it mechanistically, and whether the mechanism can be distilled. We make the question measurable by treating each decision as a token-level event scored by the model's own decision margin: the token that picks a coarse region of the label space, the tokens that pick a label within it, and the token where the output departs from a close alternative (a near-miss) named earlier in the reasoning. Attribution, exact mean-ablation, knock-in into another example's context, and a null calibration that discounts generic heads then give individual attention heads causal standing. On clinical coding of hospital discharge summaries (MIMIC-IV), with all 5,651 candidate diagnosis codes in context, a small, global, phase-structured set of heads is necessary and sufficient, by ablation and knock-in, on essentially every summary; distinct head families attend to the candidate region and back to the near-miss named earlier; and, for the mentions decided in the reasoning, the region can already be elicited several tokens before the code, from a disjoint mid-layer set that reads the input. We introduce MISTILL: unlike chain-of-thought distillation, which transfers only the teacher's reasoning text, it also supervises the student's pooled attention at exactly these decision events. Read on heads found after training, it nearly doubles the causal recovery of the contrastive decision in a cross-family student and adds a small, seed-stable gain in one that already carries most of it, with no detected task difference when both objectives train bf16 weights and a task cost with fp32 master weights.
| Comments: | substantially revised and extended; supersedes v1. New analysis (token-level decision events, head-level causal tests on MIMIC-IV clinical coding), new distillation method (MISTILL), experiments and text; the author list reflects authorship of this version. 58 pages, 6 figures. Code: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2606.06840 [cs.CL] |
| (or arXiv:2606.06840v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.06840 arXiv-issued DOI via DataCite |
Submission history
From: Debjyoti Saha Roy [view email]
[v1]
Fri, 5 Jun 2026 02:32:24 UTC (334 KB)
[v2]
Tue, 6 Oct 2026 02:10:23 UTC (301 KB)
来源：arXiv cs.AI · arxiv.org