arXiv cs.CL· Clarissa Miranda-Pena, Andrew Reeson, C\'ecile Paris, Josiah Poon, Jonathan K. Kummerfeld·· 11 小时前AI 评分40
静态分析方法检测与缓解代码库幻觉的实证分析
An Empirical Analysis of Static Analysis Methods for Detection and Mitigation of Code Library Hallucinations
AI 导读
在需要调用库的 NL-to-code 基准上，LLM 有 8.1-40% 的回复使用了不存在的库功能。静态分析工具可检测 16-70% 的全部错误和 14-85% 的库幻觉，性能随模型与数据集变化；人工分析显示其理论上限为 48.5%-77%。该研究已被 TMLR 接收。
正文 · 原文
Abstract:Despite extensive research, Large Language Models continue to hallucinate when generating code, particularly when using libraries. On NL-to-code benchmarks that require library use, we find that LLMs generate code that uses non-existent library features in 8.1-40% of responses. One intuitive approach for detection and mitigation of hallucinations is static analysis. In this paper, we analyse the potential of static analysis tools, both in terms of what they can solve and what they cannot. We find that static analysis tools can detect 16-70% of all errors, and 14-85% of library hallucinations, with performance varying by LLM and dataset. Through manual analysis, we identify cases a static method could not plausibly catch, which gives an upper bound on their potential from 48.5% to 77%. Overall, we show that static analysis methods are cheap method for addressing some forms of hallucination, and we quantify how far short of solving the problem they will always be.
| Comments: | Accepted for publication in Transactions on Machine Learning Research (TMLR) |
| Subjects: | Computation and Language (cs.CL); Software Engineering (cs.SE) |
| Cite as: | arXiv:2604.07755 [cs.CL] |
| (or arXiv:2604.07755v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2604.07755 arXiv-issued DOI via DataCite |
Submission history
From: Clarissa Miranda Pena [view email]
[v1]
Thu, 9 Apr 2026 03:21:30 UTC (229 KB)
[v2]
Fri, 10 Apr 2026 05:53:25 UTC (229 KB)
[v3]
Fri, 6 Nov 2026 01:28:41 UTC (566 KB)
来源：arXiv cs.CL · arxiv.org