arXiv cs.CL· Arslan Bisharat, Mujtaba Nazari, Khushboo Bhadauria, Brian Ortiz, Eric Spencer, Beatriz Santos, TaiNing Wang, George K. Thiruvathukal, Konstantin Laufer, Mohammed Abuhamad·· 13 小时前AI 评分31
ALIAS：评估 LLM 编写 TLA+ 规范时的性能与记忆化
ALIAS: Evaluating Performance and Memorization in LLM-Written TLA+ Specifications
AI 导读
研究团队提出 ALIAS，用 TLA+ 模型检查器作为 oracle 检验一致标识符重命名（CIR）能否测出 LLM 的记忆化。在 111 个 TLA+-Bench 任务上，重命名使 Claude Opus 4.8 正确率从 28% 降至 18%、GPT-5 从 9% 降至 7%，但改按行为评分后降幅分别缩至约 3 个百分点和 0。
正文 · 原文
Abstract:Memorization inflates the benchmark scores of large language models (LLMs), since a model can reproduce answers it saw in training. Researchers test for it with consistent identifier renaming (CIR), which gives every identifier in a task a new name, and treat a lower score on the renamed task as evidence of recall. We test this method on the Temporal Logic of Actions Plus (TLA+) language, whose model checker acts as an oracle by confirming whether a renamed task preserves the exact behaviors of the original. Across 111 TLA+-Bench tasks, renaming lowers the correctness of Claude Opus 4.8 from 28% to 18% and of GPT-5 from 9% to 7%. Most of this decrease comes from how the test grades, which requires exact names that most descriptions never state. Grading by behavior shrinks the decrease to about 3 percentage points for Opus 4.8 and to 0 for GPT-5. In two model families fine-tuned on disjoint halves of the tasks, renaming removes about half of the training gain in likelihood and costs 4.9 percentage points (Qwen) and 1.3 percentage points (Gemma) on unseen tasks. A lower score after renaming therefore mixes memorization, name-based grading, and the cost of new names.
| Comments: | 19 pages (8 main + appendix), 6 figures, 9 tables. Code and data: this https URL |
| Subjects: | Computation and Language (cs.CL); Software Engineering (cs.SE) |
| Cite as: | arXiv:2610.29767 [cs.CL] |
| (or arXiv:2610.29767v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29767 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Arslan Bisharat [view email]
[v1]
Wed, 21 Oct 2026 14:02:02 UTC (88 KB)
来源：arXiv cs.CL · arxiv.org