arXiv cs.CL· Omid Saghatchian, Ali Mahzoun, Khashayar Etemadi, Saeed Reza Kheradpisheh·· 11 小时前AI 评分34
SCRBench：超越等价性检查，评估代码语义变更推理
SCRBench: Beyond Equivalence Checking for Semantic Change Reasoning in Code
AI 导读
研究者构建 SCRBench，包含 67,658 对通过施加代码变更序列生成的 Python 程序对，并在 4,500 对分层子集上评测五个语言模型。等价性准确率达 94.71% 的模型仅能诊断 54.44% 的语义变更，最佳模型也只能精确重建 39.27% 的变更序列，施加五次变更后无模型超过 1.11%。高等价性准确率因此高估了模型对代码变更语义影响的实际推理能力。
正文 · 原文
Abstract:Understanding how code changes affect program semantics is a core aspect of code reasoning in language models. It is commonly evaluated by asking whether two programs are semantically equivalent, but a yes/no answer does not reveal whether a model knows which code change caused a semantic difference, or which sequence of code changes produced the modified program. We study two tasks that ask these questions directly, diagnosing semantic change and reconstructing sequences of code changes, alongside detecting semantic equivalence. We build SCRBench, 67,658 Python program pairs generated by applying sequences of code changes that preserve or alter program semantics, and evaluate five language models on a stratified subset of 4,500 pairs. Detecting semantic equivalence exposes weaker models, whose accuracy declines as changes accumulate, but the strongest models remain near the ceiling. Diagnosing semantic change reveals weaknesses that equivalence accuracy hides: a model with 94.71% equivalence accuracy diagnoses only 54.44% of semantic changes. Reconstructing sequences of code changes is much harder: the best model reconstructs the exact sequence for only 39.27% of pairs, and no model exceeds 1.11% when five changes are applied. High equivalence accuracy therefore overestimates how well models reason about the semantic effects of code changes.
| Comments: | 17 pages, including references and appendices |
| Subjects: | Software Engineering (cs.SE); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.41562 [cs.SE] |
| (or arXiv:2610.41562v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.41562 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Omid Saghatchian [view email]
[v1]
Fri, 30 Oct 2026 10:56:30 UTC (76 KB)
来源：arXiv cs.CL · arxiv.org