arXiv cs.CL· Meriem Terki, Nouhaila Innan, Minghao Shao, Muhammad Kashif, Alberto Marchisio, Muhammad Shafique·· 6 天前AI 评分22
QTestBench：评估 LLM 生成量子代码的测试框架
QTestBench: A Test Framework for Evaluating LLM-Generated Quantum Code
AI 导读
研究团队提出 QTestBench，用独立基准检查器评估 LLM 生成的量子测试套件，而非将测试本身视为真值。在八个 PennyLane QHack 挑战和十个 Qiskit HumanEval 任务上测试五个 LLM，生成套件几乎都能执行，但最佳生成器的测试质量仅为 QHack 0.71、Qiskit HumanEval 0.34。
正文 · 原文
Abstract:Large language models (LLMs) have shown strong results in classical code generation and are increasingly used to write the unit tests that safeguard that code. Whether they can do the same for quantum software is less clear, because quantum programs are probabilistic, depend on numerical tolerances, and are judged by domain-specific checkers, so a well-formed test can still disagree with the benchmark's verdict. Existing quantum benchmarks measure whether a model produces a correct solution, but not whether its tests can distinguish correct implementations from faulty ones. We introduce QTestBench, a framework that evaluates LLM-generated quantum test suites against independent benchmark checkers, never treating the tests themselves as ground truth. For each task, QTestBench prompts models to produce syntactic, semantic, and behavioral tests, injects candidate solutions at runtime, and compares each test verdict with the checker's label under three validation cohorts, each answering a distinct diagnostic question. We benchmark five LLMs on eight PennyLane QHack challenges and ten Qiskit HumanEval tasks. Generated suites almost always execute, but discrimination lags behind: the best generator reaches a test quality of 0.71 on QHack and 0.34 on Qiskit HumanEval, so executability alone overstates test quality. Oracle-verified synthetic mutants stabilize bug-detection estimates when natural faulty solutions are scarce, as on three of ten Qiskit tasks that yield none. QTestBench offers a reproducible, cross-framework benchmark for the trustworthiness of LLM-generated quantum tests.
| Comments: | Accepted at the 2026 IEEE 2nd International Conference on Quantum Artificial Intelligence (QAI), 2026 |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL); Quantum Physics (quant-ph) |
| Cite as: | arXiv:2610.13811 [cs.LG] |
| (or arXiv:2610.13811v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13811 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nouhaila Innan [view email]
[v1]
Fri, 9 Oct 2026 16:23:46 UTC (207 KB)
来源：arXiv cs.CL · arxiv.org