arXiv cs.CL· Ailiang Lin, Zhuoyun Li, Yusong Wang, Keyu Mao, Kotaro Funakoshi, Manabu Okumura·· 12 小时前AI 评分31
PRSR：通过配对残差子空间移除改进多语言免训练文本嵌入
Steering beyond Language: Improving Multilingual Training-Free Text Embeddings via Paired-Residual Subspace Removal
AI 导读
研究者提出免训练推理时干预方法 Paired-Residual Subspace Removal（PRSR），通过多语言平行句构造配对残差、估计逐层低秩子空间并抑制末 token 隐状态中的语言特定成分，将 LLM 表示导向更语言无关的语义空间。
正文 · 原文
Abstract:While deriving text embeddings directly from frozen LLMs has emerged as a promising direction, existing methods focus almost exclusively on English, leaving multilingual settings largely unexplored. In this work, we present the first systematic study of multilingual training-free text embeddings in LLMs. Our analysis reveals a language-conditioned embedding geometry in which semantically equivalent sentences remain misaligned across languages. We hypothesize that language-specific variation can be disentangled from semantic content to improve embedding quality. To this end, we propose Paired-Residual Subspace Removal (PRSR), a training-free inference-time intervention that steers LLM representations toward a more language-invariant semantic space. PRSR first constructs paired residuals from multilingual parallel sentences to capture language-specific deviations. It then estimates layer-wise low-rank subspaces from these residuals and suppresses the corresponding components in the last-token hidden states. We conduct more than 200 monolingual evaluations across 18 languages using MTEB and MMTEB, along with cross-lingual bitext mining between English and over 200 other languages. Empirical results demonstrate that PRSR consistently improves the performance of existing training-free baselines in both monolingual and cross-lingual settings across different LLMs.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.23807 [cs.CL] |
| (or arXiv:2610.23807v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.23807 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ailiang Lin [view email]
[v1]
Fri, 16 Oct 2026 17:30:43 UTC (4,295 KB)
来源：arXiv cs.CL · arxiv.org