arXiv cs.CL· Yukiya Horiba, Hakaze Cho, Takahiro Yakoh·· 11 小时前AI 评分29
任务向量可线性分解为可解释、可控的组件
Task Vectors Can Be Linearly Decomposed into Interpretable and Controllable Components
AI 导读
研究提出将单个任务向量（TV）按任务识别（TR）与任务学习（TL）分解为 TR、TL 和残差组件，并用贝叶斯优化调整 TR/TL 系数。在多个模型族、分类任务及基于指令和示例的 TV 上，该分解找到的干预方向优于原始 TV，其中基于指令的 TV 增益更稳定。对分解组件的单独干预产生的 logit 层面效果与其预期功能角色一致。
正文 · 原文
Abstract:Task Vectors (TVs) provide a compact way to steer large language models, but they are typically treated as atomized directions merely extracted from hidden states, leaving their internal functional structure poorly understood. Motivated by prior work that decomposes in-context learning into Task Recognition (TR) and Task Learning (TL), we ask whether a single TV can likewise be decomposed into functionally meaningful components. We formulate a TR/TL-based decomposition from candidate-label unembeddings, separating each TV into TR, TL, and residual components, and try to optimize the TR and TL coefficients using Bayesian optimization for more effective intervention. We evaluate this formulation across multiple model families, classification tasks, and both instruction- and demonstration-based TVs. Across these settings, TR/TL-based optimization identifies intervention directions that outperform the original TVs. Although optimization over random directions can also improve performance, TR/TL-based search yields more reliable gains, particularly for instruction-based TVs. Moreover, isolated interventions on the decomposed components produce distinct logit-level effects that are consistent with their intended functional roles. Together, these results suggest that TVs are not monolithic steering directions, but can be decomposed into functionally meaningful components that support both interpretation and control.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.41743 [cs.LG] |
| (or arXiv:2610.41743v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.41743 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yukiya Horiba [view email]
[v1]
Fri, 30 Oct 2026 13:48:48 UTC (318 KB)
来源：arXiv cs.CL · arxiv.org