arXiv cs.CL· Kavinda Kehelella, Sanghamitra Dutta·· 6 天前AI 评分21
COFFEQA：面向 LLM 问答的反事实引导少样本蒸馏
COFFEQA: Counterfactual-Guided Few-Shot Distillation for LLM Question Answering
AI 导读
研究提出 COFFEQA，一种用反事实样本引导 LLM 少样本蒸馏的策略，通过让教师模型改变答案的最小修改上下文，为困难事实样本提供额外监督。作者还提出 Context Following Difficulty（CFD）指标，在蒸馏前验证反事实样本。
正文 · 原文
Abstract:Knowledge distillation aims to train a smaller student model to mimic a larger teacher model for resource-constrained environments. However, the process of distillation is typically data-intensive and computationally expensive, particularly for generative tasks. Few-shot distillation offers an alternative by transferring teacher knowledge using limited training samples. However, we observe that few-shot distillation performance is heavily dependent on the choice of these training samples. In fact, we demonstrate that using more uncertain samples near the teacher's decision boundary can significantly deteriorate distillation performance. Motivated by this observation, we propose Counterfactual-guided Few-shot Distillation in Open-book Question Answering (COFFEQA), a novel strategy that aptly uses counterfactuals to provide additional supervision around challenging factual samples. In the generative QA setting, we define a counterfactual as a minimally modified context that induces the teacher to change its response to an alternative answer. We provide geometric insights on how factual-counterfactual pairs can constrain the deviation between the teacher's and student's decision boundaries. We further introduce a new metric, Context Following Difficulty (CFD) that further validates counterfactuals prior to distillation based on agreement with the intended alternative answer and dependence on the modified context. COFFEQA uses CFD-validated factual-counterfactual pairs for few-shot distillation. Experiments across five in-domain and three out-of-domain QA benchmarks using T5 and Qwen2.5 families show that COFFEQA outperforms factual-only distillation baselines under the same few-shot budget. For instance, on TriviaQA, COFFEQA improves over the factual-only baseline by 8.77 average EM points and 9.98 average F1 points under the T5 setting with 32 few-shot pairs.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.12837 [cs.CL] |
| (or arXiv:2610.12837v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12837 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Kavinda Kehelella [view email]
[v1]
Fri, 9 Oct 2026 00:41:00 UTC (571 KB)
来源：arXiv cs.CL · arxiv.org