arXiv cs.CL· Dipto Sumit, Zobeida Zaman Chowdhury, Syeda Fabiayee Jahan, Farig Sadeque·· 11 小时前AI 评分24
门控知识蒸馏在低资源孟加拉语生成中无效？45 组对比未显著优于普通 KD
Label Smoothing in Disguise? Gated Knowledge Distillation for Low-Resource Bangla Generation
AI 导读
研究测试 EWAD 与 CHAD 两种门控知识蒸馏方法，在六个孟加拉语任务、三个 60M 至约 2B 参数学生模型和两类教师模型上，45 组对比中无一显著优于普通 KD。
正文 · 原文
Abstract:Gated knowledge distillation (KD) weights teacher signals per token or example, aiming to trust an imperfect teacher only where it seems reliable. We test two gates: EWAD (entropy-weighted agreement-aware distillation), which gates tokens by teacher confidence, and CHAD (counterfactual harm-aware distillation), which gates examples by gradient alignment with validation data. Across six Bangla-centric tasks, three student models (60M to about 2B parameters), and two teacher families, no gate significantly improves plain KD in 45 comparisons. KD also fails to beat cross-entropy (CE) fine-tuning on gold outputs in 40 single-seed comparisons or three-seed runs on four tasks; CE leads on every summarisation metric. The main benefit of KD is regularisation: over ten fixed epochs, CE students lose 0.030 to 0.044 token-F1 from their early-stopped checkpoint, versus 0.005 to 0.025 for distilled students. Teacher-free label smoothing closes most of this gap. In one comparison, a teacher 0.29 token-F1 worse yields students at most 0.009 worse. Thus, gating may change regularisation strength but has little teacher knowledge to select. Findings are limited to Bangla-centric tasks and small students. We will release code, data splits, predictions, and scores.
| Comments: | 18 pages, 5 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.01065 [cs.CL] |
| (or arXiv:2611.01065v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.01065 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Syeda Fabiayee Jahan [view email]
[v1]
Sat, 31 Oct 2026 16:01:58 UTC (145 KB)
来源：arXiv cs.CL · arxiv.org