机制可解释性视角下的 LoRA：通过路径修补实现自适应微调的层稀疏性

View PDF HTML (experimental)

Abstract:Low-Rank Adaptation (LoRA) has emerged as the dominant paradigm for parameter-efficient fine-tuning of large language models. Standard all-layer LoRA applies adaptation throughout the transformer, despite heterogeneous contributions from its components. In this work, we conduct a systematic mechanistic interpretability analysis to understand how LoRA affects model internals. We observe that LoRA's adaptation effect is highly non-uniform, with significant component-wise and layer-wise sparsity. Most critically, we find that important layers are highly task-dependent, with limited overlap across tasks. Motivated by these insights, we propose Adaptive-LoRA, a calibration-driven method that automatically identifies task-specific layers via frozen-base zero-ablation, enabling efficient layer selection without exhaustive search. Extensive experiments on Qwen2.5-1.5B across 11 diverse reasoning tasks demonstrate that Adaptive-LoRA outperforms full-layer LoRA while using only 30% of layers, achieving a $3.5\times$ reduction in adapter parameters. This work bridges mechanistic interpretability and practical efficiency, providing both fundamental insights and a principled approach for adaptive fine-tuning.

Comments:	Accepted at NeurIPS 2026 (Main Track). 31 pages, including appendices and the paper checklist
Subjects:	Machine Learning (cs.LG); Computation and Language (cs.CL)
Cite as:	arXiv:2610.34756 [cs.LG]
	(or arXiv:2610.34756v1 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2610.34756 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Fei Zuo [view email]
[v1] Mon, 26 Oct 2026 02:58:22 UTC (273 KB)