arXiv cs.AI· Lorenz Brehme, Benjamin Steenhoek, Ruth Breu·· 13 小时前AI 评分34
SWE-DIVE：用动态用户模拟评估多轮编码智能体
SWE-DIVE: Multi-Turn Coding Agent Evaluation with Dynamic User Simulation
AI 导读
SWE-DIVE 是一个面向编码智能体的动态交互式评测框架，用于在多轮环境中检验智能体能力。它引入可模拟不同行为模式的用户模拟器，以及判断模拟器输出是否满足初始需求、并引导智能体完成任务的对话校验器。整个框架运行在 Harbor 环境中，负责编排编码智能体、用户模拟器与对话校验器并执行测试环境。
正文 · 原文
Abstract:Coding agents are typically evaluated using one-shot, well-formulated prompts. Benchmarks such as SWE-bench use existing GitHub issues as input for a coding agent and evaluate whether the agent successfully solves the task. However, in real-world use cases, prompts are often not as well-formulated or structured as existing GitHub issues. Additionally, a coding agent session is not limited to a single prompt; it typically involves a multi-turn interaction in which the user iteratively adds new tasks and requirements that the agent must fulfill.
To benchmark these abilities, we developed SWE-DIVE, a dynamic, interactive validation and evaluation framework for benchmarking coding agents in a multi-turn environment. To achieve this, we introduce a user simulator that can simulate different behavioral patterns and communicate with the coding agent. We also introduce a conversation validator that evaluates the user simulator's outputs to determine whether they fulfill the initial requirements and guide the agent toward successfully solving the task. The entire framework is embedded in a Harbor environment, which orchestrates the coding agent, user simulator, and conversation validator, and executes the test environment to determine whether the agent successfully solves the task. Our results and source code are available in our GitHub repository (this https URL).
| Comments: | Accepted at the NeurIPS 2026 workshop "Grounded User Simulation for Model Evaluation and Training: Diversity, Fidelity, and Validity'' (non-archival) |
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.30047 [cs.SE] |
| (or arXiv:2610.30047v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.30047 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Lorenz Brehme [view email]
[v1]
Wed, 21 Oct 2026 17:06:34 UTC (161 KB)
来源：arXiv cs.AI · arxiv.org