arXiv cs.CL· Shanchao Liang, Yiran Hu, Nan Jiang, Lin Tan·· 12 小时前AI 评分44
REPOCOD 基准测试：语言模型能否取代程序员？答案仍是“不能”
Can Language Models Replace Programmers for Coding? REPOCOD Says 'Not Yet'
AI 导读
新基准 REPOCOD 显示，十个 LLM 在其上的 pass@1 均未超过 30%。该基准包含来自 11 个流行项目的 980 个整函数生成任务，其中 50.8% 需要仓库级上下文，每个实例配有 314 个开发者编写的测试用例。研究还发现，检索增强生成的效果优于直接使用目标函数依赖作为上下文。
正文 · 原文
Abstract:Recently, a number of repository-level code generation benchmarks-such as CoderEval, DevEval, RepoEval, RepoBench, and LongCodeArena-have emerged to evaluate the capabilities of large language models (LLMs) beyond standalone benchmarks like HumanEval and MBPP. Thus, a natural question is, would LLMs have similar performance in real world coding tasks as their performance in these benchmarks? Unfortunately, one cannot answer this question, since these benchmarks consist of short completions, synthetic examples, or focus on limited scale repositories, failing to represent real-world coding tasks.
To address these challenges, we create REPOCOD, a Python code-generation benchmark containing complex tasks with realistic dependencies in real-world large projects and appropriate metrics for evaluating source code. It includes 980 whole-function generation tasks from 11 popular projects, 50.8% of which require repository-level context. REPOCOD includes 314 developer-written test cases per instance for better evaluation. We evaluate ten LLMs on REPOCOD and find that none achieves more than 30% pass@1 on REPOCOD, indicating the necessity of building stronger LLMs that can help developers in real-world software development. In addition, we found that retrieval-augmented generation achieves better results than using target function dependencies as context.
| Subjects: | Software Engineering (cs.SE); Computation and Language (cs.CL) |
| Cite as: | arXiv:2410.21647 [cs.SE] |
| (or arXiv:2410.21647v5 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2410.21647 arXiv-issued DOI via DataCite |
Submission history
From: Shanchao Liang [view email]
[v1]
Tue, 29 Oct 2024 01:21:05 UTC (1,243 KB)
[v2]
Thu, 31 Oct 2024 07:31:31 UTC (1,243 KB)
[v3]
Sun, 3 Nov 2024 21:24:10 UTC (1,243 KB)
[v4]
Tue, 24 Jun 2025 20:49:51 UTC (2,722 KB)
[v5]
Tue, 20 Oct 2026 04:42:17 UTC (2,731 KB)
来源：arXiv cs.CL · arxiv.org