arXiv cs.AI· Pengcheng Wang, Zhexuan Zhang, Chao Lou, Kewei Tu·· 3 天前AI 评分27
句法语言模型的大规模实证研究：1B 参数下结构掩码优势消失
A Scaled-Up Empirical Study of Syntactic Language Models
AI 导读
一项针对句法语言模型（SLM）的大规模实证研究，在最高 1B 参数、100B 训练 token 规模下对比了 12 种配置（1 个终端基线 + 11 个不同线性化与注意力掩码的 SLM 变体），并在 11 个下游任务和两个句法基准上评测。
正文 · 原文
Abstract:Syntactic language models (SLMs) that generate sentences along with their syntactic tree structures have shown promise in both syntactic generalization and downstream tasks. Prior studies, however, have mostly evaluated smaller models trained on less pretraining data and tested on narrower task suites. In addition, the design space of tree linearization and attention masks remains underexplored. We conduct a large-scale empirical study of 12 controlled configurations---one terminal baseline and 11 SLM variants spanning different linearizations and masks---at scales up to 1B parameters and 100B training tokens, evaluated across 11 downstream tasks and two syntactic benchmarks. The experimental results show that SLMs with standard causal attention match or exceed structural-mask variants across downstream and syntactic benchmarks. They require no architectural modifications and remain compatible with FlashAttention. We also observe that when scaling up to the 1B-parameter scale, the top-performing SLM variant no longer shows a clear advantage over a compute-matched non-structural baseline, suggesting that SLMs may become less competitive with scale. Our code is available at this https URL.
| Comments: | Accepted to EMNLP 2026 |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.16837 [cs.CL] |
| (or arXiv:2610.16837v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.16837 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Pengcheng Wang [view email]
[v1]
Mon, 12 Oct 2026 14:16:58 UTC (44 KB)
来源：arXiv cs.AI · arxiv.org