arXiv cs.CL· Neeva Oza, Ishaan Govil, Parul Gupta, Dinesh Khandelwal, Dinesh Garg, Parag Singla·· 12 小时前AI 评分35
LLMs 难以应对简单代码变换：CETBench 代码等价性检查基准发布
LLMs are Brittle to Simple Code Transformations: Introducing CETBench - A Benchmark for Code-Equivalence Checking
AI 导读
研究者提出 CETBench（Code Equivalence with Transformations Benchmark），通过采样程序对并施加随机代码变换生成等价或非等价样本，用于评估 LLM 判断两段程序功能是否等价的能力。
正文 · 原文
Abstract:We study how well LLMs can determine whether two programs are functionally equivalent. This is an important problem because benchmarking code equivalence helps assess LLM capability in tasks such as code rewriting and translation. To this end, we introduce CETBench - Code Equivalence with Transformations Benchmark - built from a repository of programs that may solve the same or different tasks. Each dataset instance is created by sampling a program pair and applying a random sequence of predefined code transformations, yielding either equivalent or non-equivalent pairs. Our analysis shows that even simple transformations cause a significant performance drop in state-of-the-art LLMs on code-equivalence checking. These challenges are further amplified in the cross-lingual setting when comparing programs written in different languages. To remedy this, we present a simple fine-tuning-based approach to boost LLM performance on the transformed pairs of programs. Our approach for dataset generation is generic, supporting cross-lingual equivalence checking, the generation of program pairs with varying difficulty levels, and the application of diverse transformations. In our experiments, we perform ablations over the difficulty level of original programs, as well as the kind of transformations used in generating pairs for equivalence checking. Our analysis presents deep insights into the working of LLMs for the task of code-equivalence, and points to the fact that they may still be far from what could be termed as a semantic understanding of the underlying code.
| Comments: | Published in ACL Findings 2026 |
| Subjects: | Software Engineering (cs.SE); Computation and Language (cs.CL); Machine Learning (cs.LG); Programming Languages (cs.PL) |
| MSC classes: | 68-02 (Primary) 68T50, 68T07, 68N19, 68N30 (Secondary) |
| ACM classes: | I.2.7; I.2.6; I.2.5; D.3.0; D.3.3; D.3.1; F.3.2; F.3.1; F.3.3; D.2.3; D.2.5 |
| Cite as: | arXiv:2506.04019 [cs.SE] |
| (or arXiv:2506.04019v2 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2506.04019 arXiv-issued DOI via DataCite |
|
| Journal reference: | Findings of the Association for Computational Linguistics: ACL 2026 |
| Related DOI: | https://doi.org/10.18653/v1/2026.findings-acl.2070
DOI(s) linking to related resources |
Submission history
From: Neeva Oza [view email]
[v1]
Wed, 4 Jun 2025 14:47:14 UTC (166 KB)
[v2]
Tue, 20 Oct 2026 16:57:53 UTC (1,162 KB)
来源：arXiv cs.CL · arxiv.org