RefactorPlatform：用于仓库级重构智能体受控评估的开源测试框架

View PDF HTML (experimental)

Abstract:Repository-scale refactoring requires coding agents to propagate a single change across many interdependent files without altering program behavior, yet to our knowledge no existing harness isolates the design choices that determine agent success on this task. We present RefactorPlatform, an open-source evaluation harness that holds the environment fixed and varies each design axis explicitly: model backbone (via OpenRouter and GitHub Copilot CLI), execution regime (baseline, retrieval-augmented, and multi-agent), and prompt specificity. Each run executes in an isolated workspace with live terminal streaming, per-task logging of tokens, diffs, and transcripts, AST-based verification, and exportable telemetry for audit and reproduction. Demonstrating the platform on 100 multi-file RefactorBench tasks across four model families, we illustrate the analyses it supports: AST-aware chunking outperforms naive token-window chunking by 25-30 percentage points (pp) across prompt modes, whereas naive retrieval falls below the retrieval-free baseline; a lean retrieval-augmented single agent (86%) beats the sub-agent configuration we evaluated (66%) on matched tasks with no task passing under delegation that fails under retrieval; and retrieval's accuracy gains absorb its token overhead, leaving cost per successful refactoring unchanged. RefactorPlatform is open-sourced to make refactoring-agent evaluation reproducible and auditable.

Comments:	Accepted at EMNLP 2026 System Demonstrations
Subjects:	Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2609.04898 [cs.CL]
	(or arXiv:2609.04898v2 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2609.04898 arXiv-issued DOI via DataCite

Submission history

From: Vijayasri Iyer [view email]
[v1] Fri, 4 Sep 2026 08:58:10 UTC (185 KB)
[v2] Thu, 15 Oct 2026 12:45:09 UTC (185 KB)