arXiv cs.AI· Binhang Qi, Yiming Liu, Yun Lin, Yangjie Zhou, Hailong Sun, Jin Song Dong·· 13 小时前AI 评分38
Req2Test-Bench：评估 LLM 智能体从软件需求生成行为充分测试的能力
Benchmarking LLM Agents for Behaviorally Adequate Test Generation from Software Requirements
AI 导读
研究者推出 Req2Test-Bench，包含从 8 个开源项目 BDD 产物中回收的 5,086 个执行验证实例，覆盖 5 种编程语言和 4 个 BDD 框架，要求智能体在行为实现被隐藏时仅凭 Gherkin 需求生成完整测试实现。
正文 · 原文
Abstract:As large language model agents increasingly compile natural-language requirements into repository-level code, requirement-to-test generation grows in importance to provide validation constraints (or harness) to guide code generation. Yet it remains unclear whether current agents can generate tests that validate the required behavior when the implementation of that behavior is withheld. Existing test-generation benchmarks mostly focus on white-box testing, whereas the requirement--test-related study usually requires evaluation on semantic alignment between the given documents and the generated tests.
We introduce Req2Test-Bench, a benchmark containing 5,086 execution-validated instances recovered from developer-authored behavior-driven development (BDD) artifacts in 8 open-source projects, spanning 5 programming languages and 4 BDD frameworks. Each instance pairs a natural-language requirement in Gherkin with its developer-written Test Implementation. Given the requirement, target interfaces of the required behavior, an agent must generate a complete, potentially multi-file Test Implementation. Evaluation of 8 agent--model--reasoning configurations reveals a substantial gap between execution success and behavioral adequacy, which is measured strictly as kill-completeness: a kill-complete test kills every mutant that the ground-truth test kills. The best-performing configuration, Codex with DeepSeek-V4.1-Flash, produces passing tests for 82.8% of instances, yet only 33.8% are behaviorally adequate. We find that (1) scaling up agents yields sharply diminishing returns in behavioral adequacy relative to generation cost; (2) under-specified requirements cause the information gap, including ambiguous setup and implicit expected outcomes; and (3) requirement-grounded harnesses that steer agent non-determinism offer a promising direction for improvement.
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.36584 [cs.SE] |
| (or arXiv:2610.36584v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36584 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Binhang Qi [view email]
[v1]
Tue, 27 Oct 2026 03:41:55 UTC (339 KB)
来源：arXiv cs.AI · arxiv.org