arXiv cs.AI· Robert Pasternak·· 14 小时前AI 评分31
语言模型语言学评估中的提示词与概率：语义焦点敏感性的证据
Prompts and probabilities in the linguistic assessment of language models: Evidence from semantic focus-sensitivity
AI 导读
研究在语义焦点敏感性任务上对比了显式提示与 token 概率两种评估方式，测试不同规模模型并使用粗体、斜体等文本焦点标记。token 概率在所有模型上均高于随机水平，提示词结果大体与既往发现一致，但部分较大模型以较大幅度超过概率方法。作者认为原因是 LM 不倾向产生无端焦点标记，从而干扰 token 概率结果；将标准焦点标记替换为随机标记后，概率方法仍高于随机但性能下降显著大于提示词。
正文 · 原文
Abstract:As language models (LMs) have improved, an increasingly common means of evaluating their linguistic competence is through explicit prompting (e.g., "Which sentence is more acceptable?"). Prior work indicates that such prompting underperforms compared to standard token-probability-based methods in smaller models, but improves in larger models. We test this in a new domain: semantic focus-sensitivity, responsible for contrasts like "Lucy only SWAM yesterday" vs. "Lucy only swam YESTERDAY". Testing models of various sizes using text-based focus markers like boldface and italics, we find universal above-chance performance for token probabilities. Prompting results mostly align with previous findings, but in some larger models they outperform probabilities by a wide margin. We propose that this is because LMs are not prone to produce out-of-the-blue focus marking, which muddies results from token probabilities in a way that does not affect prompting. We test this by replacing standard focus markers with randomly generated ones; token probabilities still beat chance but show significantly greater performance decrease compared to prompting. These results suggest that metalinguistic prompting, while non-trivial for LMs, can also be more robust when testing stylistically atypical data. We argue that this has important consequences for the assessment of LMs' language competence, even if LMs do not perform metalinguistic judgments through human-like introspection.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.39923 [cs.CL] |
| (or arXiv:2610.39923v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.39923 arXiv-issued DOI via DataCite |
Submission history
From: Robert Pasternak [view email]
[v1]
Thu, 29 Oct 2026 12:28:54 UTC (369 KB)
来源：arXiv cs.AI · arxiv.org