arXiv cs.CL· Qince Luan, Jianhui Xu, Yang Xu·· 13 小时前AI 评分29
谱平滑与尖峰扰动重塑大语言模型的能力分布与选择性任务表现
Spectral Smoothing and Spiking Perturbations Reconfigure Capability Profiles and Selective Task Performance in Large Language Models
AI 导读
研究对 Llama 和 Qwen 五个开源指令模型的 up_proj 与 down_proj 矩阵施加谱平滑与尖峰扰动，在保持奇异向量和奇异值几何均值不变的前提下改变谱分布。结果显示扰动带来的能力变化呈结构化而非随机：大幅正向扰动更稳定地抑制语言与演绎推理表现，适度负向扰动有时能提升事实性或局部任务表现，敏感度随模型家族、评测模块和层间谱不均衡程度显著变化。
正文 · 原文
Abstract:Singular-value spectra encode important structure in transformer feed-forward projections, yet their causal role in downstream behavior remains underexplored. We study post-hoc spectral smoothing and spiking interventions on the up_proj and down_proj matrices. Positive perturbation stretches deviations from the geometric mean, making large singular values larger and small ones smaller (spiking), and vice versa. The intervention preserves singular vectors and the geometric mean of the singular values. Across five open Llama and Qwen instruction models, we evaluate the perturbed systems on capability-labeled subtasks, standard and chain-of-thought reasoning benchmarks, judge-scored truthfulness, hallucination, and safety tasks, and TRAIT personality probes. To interpret score shifts, we fit responses in a three-dimensional capability basis consisting of factual knowledge, language understanding, and deductive reasoning. The induced changes suggest structured rather than purely random sensitivity: large positive perturbations more reliably suppress language and deductive responses, moderate negative perturbations can sometimes improve factual or task-local performance on the evaluated grids, and sensitivity varies strongly by model family, benchmark module, and underlying layerwise spectral inequality. These results position spectral perturbation as a lightweight intervention for probing how capabilities are organized and traded off in large language models.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.30676 [cs.CL] |
| (or arXiv:2610.30676v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.30676 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Qince Luan [view email]
[v1]
Thu, 22 Oct 2026 03:42:07 UTC (5,997 KB)
来源：arXiv cs.CL · arxiv.org