arXiv cs.CL· Michael Foster, Barbara McGillivray, Gard B. Jenset, Jakub M\'acha, Jason Harding, Marcus Perlman, Devyani Sharma, Patrick Skeels, Jana Willer-Gold, Jeannette Littlemore·· 12 小时前AI 评分34
LALA 基准：评估 LLM 的文学与语言学分析能力
The LALA Benchmark: Evaluating Literary and Linguistic Analysis in LLMs
AI 导读
研究者推出 Literature and Linguistics Analysis Benchmark（LALA），包含 236 道由语言学与文学专家撰写的开放式问题及专家参考答案，并用人类专家和 Sol、Qwen 两个 LLM 评判器测试了 17 个语言模型。
正文
Abstract:We introduce the Literature and Linguistics Analysis Benchmark (LALA), comprising 236 open-ended questions written by experts in linguistics and literary studies, together with expert-provided ideal answers. By comparing models within families and at similar training scales using human and LLM judges, LALA identifies differences in linguistic competence associated with post-training, offering a framework for evaluating future models and guiding targeted improvements. We administered questions to 17 language models and evaluated their responses using both human expert assessment and two LLM evaluators, Sol and Qwen. Base models displayed little of the linguistic competence assessed by the benchmark, whereas their post-trained versions performed markedly better, suggesting that post-training plays a central role in making linguistic knowledge accessible and usable. Parameter count, by contrast, showed no statistically significant association with partial-credit performance, and competencies remained unevenly distributed across linguistic domains. We interpret this pattern as evidence that post-training adds linguistic competencies piecemeal, without producing a general linguistic intelligence capable of extending them beyond the learned policies that support them. Models performed best on logical relations between statements and selected discourse-level tasks, while even the strongest post-trained models continued to struggle with ambiguity, literary style, reference, and misleading assumptions. Human and Sol evaluations produced nearly identical model rankings in both the original and ReliableEval runs, supporting the replicability of the comparative results.
| Comments: | 49 pages, 10 figures |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.34080 [cs.CL] |
| (or arXiv:2610.34080v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34080 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Michael Foster [view email]
[v1]
Sun, 25 Oct 2026 05:38:57 UTC (3,947 KB)
来源：arXiv cs.CL · arxiv.org