arXiv cs.AI· Ruiqi Wang, Xinchen Wang, Cuiyun Gao, Chun Yong Chong, Xin Xia, Qing Liao·· 2 天前AI 评分24
AXIOM：通过约束扰动与校准对代码 LLM-as-a-Judge 进行基准测试
Benchmarking Code LLM-as-a-Judge via Constrained Perturbation and Calibration
AI 导读
研究者提出 AXIOM，一个基于扰动的框架，用于构建代码评估基准。它通过质量约束语义扰动控制程序质量分布，再经混合质量校准筛选语义多样的程序并简化人工标注，据此构建了包含 1,962 道多语言代码评估问题的 AXIOMBENCH。
正文 · 原文
Abstract:Code from LLM-assisted development may contain defects like API hallucinations, motivating the development of code evaluation metrics to assess such code before deployment. As conventional rule-based metrics merely score code based on similarity to reference programs, and execution-based metrics require expensive manual preparation for tests, researchers have developed LLM-as-a-judge metrics for both behavioral and nonbehavioral code evaluation, and curated multiple benchmarks to validate their effectiveness. However, such benchmarks suffer from critical limitations: Some feature coarse-grained binary labels of passing/failing unit tests, unable to distinguish between fundamental and minor flaws; others utilize fine-grained criteria with manually annotated scores as ground-truth, but such criteria are often vaguely defined, introducing subjectivity and unreliability in human scores. Furthermore, they often directly prompt LLMs to generate code for evaluation, which may lead to unbalanced code quality distributions, failing to test the metrics' discriminative capability across varying quality levels.
To curate a diverse benchmark with a balanced program quality distribution and streamline manual annotation, we propose AXIOM, a novel perturbation-based framework. It reframes program scores as the refinement effort needed for deployment, with two stages: (1) Quality-constrained semantic perturbation, prompting LLMs to apply sequences of predefined perturbation rules to existing programs, precisely controlling each program's target quality to achieve a balanced distribution. (2) Hybrid quality calibration, selecting semantically distinct programs for benchmark diversity, and performing streamlined manual annotation with multisource context. With AXIOM, we curate AXIOMBENCH, a large-scale multilingual benchmark with 1,962 code evaluation problems in a more balanced score distribution...
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2512.20159 [cs.SE] |
| (or arXiv:2512.20159v2 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2512.20159 arXiv-issued DOI via DataCite |
Submission history
From: Ruiqi Wang [view email]
[v1]
Tue, 23 Dec 2025 08:39:22 UTC (852 KB)
[v2]
Thu, 15 Oct 2026 03:29:16 UTC (1,668 KB)
来源：arXiv cs.AI · arxiv.org