arXiv cs.CL· Vedant Pandhare, Piyush Dhoka·· 2 天前AI 评分22
校准语言如何影响多语言模型的 Block Influence 剪枝
Calibration Language Shapes Block-Influence Pruning in Multilingual Language Models
AI 导读
研究在 Qwen2.5-1.5B-Instruct、Llama-3.2-1B 和 Gemma-2-2B 上发现，用英语校准的 Block Influence（BI）会系统性低估印度语言文本的深层网络活动，三种印度语言的后期 block BI 比英语高 20.7-55.5%。
正文 · 原文
Abstract:Block Influence (BI) ranks Transformer blocks for depth pruning from their input-output cosine change, but the ranking depends on the calibration distribution. We study whether English calibration transfers to Indian languages in Qwen2.5-1.5B-Instruct, Llama-3.2-1B, and Gemma-2-2B. English, Hindi, Marathi, and Tamil use disjoint Wikipedia articles for calibration and evaluation; every selector receives the same 150-chunk budget, and Tamil is excluded from multilingual aggregation to test cross-script transfer. Across all three models, late-block BI is 20.7-55.5% higher for the three Indian languages than for English, and every predeclared late-minus-middle contrast has a 95% bootstrap interval above zero. This mismatch changes the low-BI blocks selected for removal despite high global rank correlations. Averaging BI over equal English, Hindi, and Marathi budgets mitigates the resulting perplexity damage for Qwen at six removed blocks and for Gemma at all tested depths, including held-out Tamil. The mitigation is not universal: on Llama, balancing improves Hindi and Marathi at two removed blocks but severely harms Tamil, while larger pruning budgets select the same blocks and collapse perplexity regardless of calibration. Rank-mean and minimax-rank aggregators do not dominate the raw mean. These results support a diagnostic claim rather than a universal pruning rule: English-only BI systematically misses late-layer activity for Indian-language text, but multilingual calibration must still be validated per architecture and pruning budget.
| Comments: | 16 pages, 3 figures. Code and results: this https URL |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.21383 [cs.CL] |
| (or arXiv:2610.21383v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21383 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Vedant Pandhare [view email]
[v1]
Thu, 15 Oct 2026 07:01:55 UTC (82 KB)
来源：arXiv cs.CL · arxiv.org