arXiv cs.CL· Yaning Jia, Chunhui Zhang, Chiyu Ma, Zhongyu Ouyang, Ivory Yang, Soroush Vosoughi·· 11 小时前AI 评分38
规模即能力：大模型如何掌握“不可能语言”
Scaling Makes It Possible: How Large Models Master Impossible Languages
AI 导读
研究提出“语义连贯性”指标，通过在不可能语言样本上微调不同规模模型发现，十亿参数级大模型在不可能语言与可能语言上表现相近，而百万参数级小模型在不可能语言上性能显著下降，且语义复杂度越高差距越大。扩大模型规模使学习重心从语法规则转向语义理解，十亿参数级模型对低层语言扰动的敏感度降低三倍，同时在高层次语义扰动下保持稳定表现。
正文 · 原文
Abstract:While language models (LMs) excel in natural language tasks, their ability to learn impossible languages--those that violate the structural, cognitive, or logical constraints that make human languages learnable and functional--remains debatable. This controversy is further fueled by a divide between linguistic theorists and empirical researchers working with small-scale LMs. To address this controversy, we introduce semantic coherence. This concept measures how effectively semantic information is expressed in language structures and examines the learning capabilities of LMs across multiple scales. By fine-tuning billion-parameter LMs on impossible language samples, we demonstrate that larger LMs achieve similar performance on impossible and possible languages, whereas million-parameter LMs exhibit significant degradation when trained on impossible languages. This performance gap widens as semantic complexity increases. We further show that scaling up the LM size fundamentally shifts the learning focus from grammatical rules to semantic understanding, with billion-scale LMs exhibiting a threefold reduction in sensitivity to low-level linguistic perturbations while maintaining stable performance under high-level semantic perturbations.
| Comments: | 17 pages, 7 figures, including 6 pages of supplementary information |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.02015 [cs.CL] |
| (or arXiv:2611.02015v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.02015 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yaning Jia [view email]
[v1]
Sun, 1 Nov 2026 15:58:51 UTC (923 KB)
来源：arXiv cs.CL · arxiv.org