arXiv cs.AI· Rupam Patir, Keyan Guo, Suvadra Barua, Abhijeet Pathak, Dinesh Gudimetla, Jiawei Guo, Hongxin Hu, Haipeng Cai·· 14 小时前AI 评分49
DualGauge：LLM 与编程智能体仅凭规格生成代码的自动化安全-功能联合基准测试
DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code Generation by LLMs and Coding Agents
AI 导读
DualGauge 是首个全自动联合评估“仅凭规格生成代码”功能正确性与安全性的框架，配套 DualGauge-Bench 包含 307 个编码任务，每个任务的功能与安全测试均源自同一规格。
正文 · 原文
Abstract:Large language models (LLMs) and LLM-based coding agents are now used to generate code from natural-language specifications, yet ensuring such code is both functionally correct and secure remains a challenge. We present DualGauge, the first fully automated framework for jointly evaluating correctness and security of specification-only code generation, supported by DualGauge-Bench, a language-agnostic benchmark of 307 coding tasks each paired with functional and security tests derived from the same specification. Evaluating 10 representative LLMs across Python, C++, and JavaScript, we find that functional correctness substantially overestimates reliable code generation: even the strongest model remains below 15% joint security-functionality success in every language. Common model-side factors--scale, extended thinking, quantization, instruction tuning, and code specialization--do not reliably improve joint performance, suggesting secure-and-correct code generation does not simply emerge from stronger coding capability. Evaluation of 3 leading coding agents (Codex, OpenHands, and Claude Code) shows that iterative scaffolding is not a reliable remedy on specification-only tasks: results are strongly scaffold-dependent, with two of the three agents failing to outperform a comparable direct model. A qualitative audit reveals that these failures concentrate at the output contract boundary and in guards that exist but are insufficient--patterns that only joint benchmarking reliably exposes.
| Comments: | Accepted to Findings of the Association for Computational Linguistics: EMNLP 2026 |
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR) |
| Cite as: | arXiv:2511.20709 [cs.SE] |
| (or arXiv:2511.20709v3 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2511.20709 arXiv-issued DOI via DataCite |
Submission history
From: Rupam Patir [view email]
[v1]
Mon, 24 Nov 2025 22:26:14 UTC (776 KB)
[v2]
Mon, 15 Jun 2026 14:19:33 UTC (407 KB)
[v3]
Wed, 28 Oct 2026 21:06:32 UTC (980 KB)
来源：arXiv cs.AI · arxiv.org