arXiv cs.CL· Isaac Saxton-Knight, Gabriel Saxton-Knight, Diego Oliveira, Tidor-Vlad Pricope, Aston Garratt, Jocelyn D'Arcy·· 10 小时前AI 评分29
CulturalCrosswords：评估答案约束与填格能力的多语言基准
CulturalCrosswords: A Multilingual Benchmark of Answer Constraints and Grid Completion
AI 导读
CulturalCrosswords 用 42 种语言、众包谜语构建的 53 个填字游戏，评测 11 种 LLM 配置在答案约束下的表现。在 962 条存在替代答案的线索上，给出长度要求使模型与参考答案的一致率提升 22.8 个百分点，若同时接受替代答案则仅提升 8.4 个百分点。填字输入使所有交叉格答案兼容比例提高 9.7 个百分点，而要求填满整格则暴露出模型间的巨大差异。
正文 · 原文
Abstract:When several answers fit a riddle, exact-match scoring can penalise a valid response. CulturalCrosswords tests how answer constraints affect large language models (LLMs) solving cultural riddles, whose solutions draw on knowledge specific to a community. We constructed 53 crosswords from crowdsourced riddles in 42 languages and evaluated 11 LLM configurations on five versions of each clue set: plain batches, batches with lengths, batches with lengths and partial-answer hints, and crosswords requesting an answer list or a completed grid. On 962 clues with alternative answers, lengths improve agreement with the reference by 22.8 percentage points, compared with 8.4 points when accepted alternatives also receive credit. Part of the gain therefore appears to come from selecting a required answer form. Crossword input increases the share of all crossings with compatible answers by 9.7 points; requiring a completed grid reveals large differences between models. The benchmark distinguishes agreement with a reference from the ability to produce answers that fit together and occupy the correct cells.
| Comments: | 23 pages, 7 figures, including appendices |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.10044 [cs.CL] |
| (or arXiv:2611.10044v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.10044 arXiv-issued DOI via DataCite |
Submission history
From: Tidor-Vlad Pricope [view email]
[v1]
Thu, 5 Nov 2026 21:45:08 UTC (197 KB)
[v2]
Tue, 10 Nov 2026 03:31:27 UTC (197 KB)
来源：arXiv cs.CL · arxiv.org