arXiv cs.AI· Yichen Zhao, Minghao Shao, Zhuoyan Yu, Yijun Shen, Boyuan Chen, Yik-Cheung Tam, Muhammad Shafique·· 1 天前AI 评分34
ROSTrans：面向 ROS1 到 ROS2 迁移的 LLM 基准测试
ROSTrans: Empirically Benchmarking LLMs for ROS1-to-ROS2 Migration
AI 导读
ROSTrans 基准测试评估 LLM 将 ROS1 代码迁移到 ROS2 的能力，包含来自 70+ 个开源 ROS1 仓库的 108 个脚手架补全任务，按接口级、行为级和系统级正确性分层。
正文 · 原文
Abstract:Although large language models have shown strong capabilities in domain-specific code generation, their performance in robotics software development, especially for the Robot Operating System (ROS), remains largely unexplored. ROS code generation presents distinctive challenges, including complex inter-node coordination, middleware configuration, and lifecycle semantics that go far beyond standard coding benchmarks, and many important correctness properties become observable only once migrated nodes are launched and communicate. We present ROSTrans, a benchmark for evaluating LLMs on ROS1-to-ROS2 code migration, a representative and practically important setting for ROS code generation. ROSTrans comprises 108 scaffolded completion tasks drawn from a diverse set of 70+ open-source ROS1 repositories and organized into a hierarchical taxonomy assessing interface-level, behavioral-level and system-level correctness. For every task we establish an executable reference: a verified ROS2 implementation, a runtime test suite that launches the nodes and exercises their topics, services, actions and parameters, and a container ensuring reproducibility. Candidates are scored by injecting only their target files into an otherwise fixed copy of the reference package, so a candidate can neither supply nor weaken the tests it is judged by. Evaluating six frontier LLMs over five stochastic generations per task, we find pass@1 below 10.9%, and that 56.5% of all samples build successfully and then fail at runtime,and 32.0% fail runtime assertions on the migrated interfaces, a class of error no compile-only oracle can detect. Execution-guided repair further improves task success, reaching 49.1% and 47.2% for Claude Opus 4.6 and GPT-5.5 after three rounds, respectively.
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.41168 [cs.SE] |
| (or arXiv:2610.41168v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.41168 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yichen Zhao [view email]
[v1]
Fri, 30 Oct 2026 02:44:28 UTC (29,720 KB)
来源：arXiv cs.AI · arxiv.org