arXiv cs.CL· Laya Iyer, Chen Shani·· 13 小时前AI 评分20
词性决定效果：LLM 后训练中概念增强的 POS 影响研究
Every Part Has Its Role: Part-of-Speech Effects in LLM Post-Training
AI 导读
研究复现并扩展了概念增强方法，将替换目标词的范围从名词扩展到动词和形容词，考察承载概念信号的词性（POS）如何影响困惑度与下游表现。结果显示，概念增强在新闻文本上最稳定地改善困惑度且与词性无关，而动词带来最可靠的下游收益。CKA 与线性探针的表示分析表明，下游增益最大的条件未必是内部表示改变最大的条件。
正文 · 原文
Abstract:Large language models are typically trained with next-token prediction (NTP), which rewards predicting specific tokens rather than explicitly accounting for the semantic similarities between different words or expressions. Prior work has addressed this limitation by augmenting training data with minimally varying sentences that replace a target word with conceptually related alternatives, e.g., "The tired coach ran home" $\rightarrow$ {"The tired trainer ran home", "The tired instructor ran home", ...}. Such concept augmentation improves performance, but it remains unclear whether its effectiveness depends on where in the sentence the concept is expressed. We reproduce and extend prior work $\textit{beyond nouns}$ to verbs and adjectives, asking whether the part of speech (POS) carrying the concept signal affects perplexity and downstream performance. We find different patterns across evaluation settings: concept augmentation improves perplexity most consistently on news, independent of the POS used, whereas verbs provide the most reliable downstream gains. Despite these behavioral differences, representational analyses using CKA and linear probing show that the conditions yielding the largest downstream gains are not necessarily those that most alter the model's internal representations. Together, these results suggest that the effectiveness of concept augmentation depends on where and how conceptual information is introduced during training.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.29843 [cs.CL] |
| (or arXiv:2610.29843v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29843 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Laya Iyer [view email]
[v1]
Wed, 21 Oct 2026 15:05:51 UTC (1,568 KB)
来源：arXiv cs.CL · arxiv.org