测试驱动代码生成的理论分析：环境交互策略的概率框架与遗憾界

View PDF HTML (experimental)

Abstract:Code assistants are increasingly utilized in test-driven software development, yet the theoretical mechanisms behind their environment-interaction strategies remain underexplored. We provide a probabilistic framework for two dominant paradigms: code selection after generation using the execution environment, and code generation conditioned on environment feedback. First, we formalize several well-established selection heuristics as environment-aware estimators of code correctness. We theoretically prove that estimators based on fuzzy functional similarity add an inductive bias and strictly dominate estimators based on functional equivalence in terms of signal-to-noise ratio. Second, we frame backprompting as an in-context approximation of Thompson sampling. We derive a novel regret bound for reward functions with unobservable components, theoretically explaining why the effectiveness of backprompting is limited by the ambiguity of the informal task description (an irreducible regret). Using seven state-of-the-art open weight models, we corroborate these findings across BigCodeBenchHard, LeetCodeDataset, QiskitHumanEvalSim, and SWE-bench Verified. Our formalization also suggests how to improve task descriptions effectively, leading to an improved benchmark for LLM-assisted quantum programming, QiskitHumanEvalSimX.

Comments:	Accepted at NeurIPS 2026 as a spotlight paper
Subjects:	Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as:	arXiv:2602.06098 [cs.SE]
	(or arXiv:2602.06098v4 [cs.SE] for this version)
	https://doi.org/10.48550/arXiv.2602.06098 arXiv-issued DOI via DataCite

Submission history

From: Nicolas Menet [view email]
[v1] Thu, 5 Feb 2026 13:49:42 UTC (635 KB)
[v2] Sun, 29 Mar 2026 15:10:52 UTC (637 KB)
[v3] Thu, 7 May 2026 12:50:23 UTC (703 KB)
[v4] Fri, 16 Oct 2026 09:26:12 UTC (736 KB)