arXiv cs.AI· Wenqian Zhang, Yangyi Huang, Weiyang Liu, Zhen Liu·· 3 天前AI 评分30
用 LLM 做程序合成：BesiegeField 基准测试组合式机械设计
Compositional Machine Design as Program Synthesis with LLMs
AI 导读
研究者提出"组合式机械设计"，将机器写成组合标准化部件的程序，并以模拟物理行为判定成败，同时发布基于游戏 Besiege 的测试平台 BesiegeField。在单智能体生成、迭代编辑和分层工作流下，强模型能恢复任务相关结构，但在空间精确装配与机制级规划上仍吃力。团队用模拟奖励对开源模型 Qwen2.5-14B 做强化学习微调，固定生成预算下最佳机器得到改善。
正文 · 原文
Abstract:Large language models (LLMs) have shown strong abilities in writing and revising programs, yet many program-synthesis benchmarks still evaluate programs in symbolic or digital environments. We introduce compositional machine design, a physically grounded form of program synthesis where machines are written as programs that compose standardized parts, and success is determined by simulated physical behavior. To study this problem, we present BesiegeField, a testbed built on the machine-building game Besiege. In BesiegeField, LLM agents generate machine programs from textual functional demands, execute the resulting machines in simulation, and receive rewards and state feedback. We benchmark LLM agents across representative machine-design tasks under single-agent generation, iterative editing, and hierarchical workflows. Strong models recover task-relevant structures and sometimes achieve nontrivial physical performance, but often struggle with spatially precise assembly, mechanism-level planning, and translating feedback into useful structural edits. We further finetune Qwen2.5-14B, an open-source LLM, with reinforcement learning from simulation-derived rewards. We find that, under a fixed generation budget, RL improves the best machine discovered. We additionally evaluate human performance to provide a reference point for task difficulty. These results establish compositional machine design as a testbed for studying LLM agents that synthesize executable machine programs and improve them through physical feedback.
| Comments: | 75 pages, 44 figures, Project Page: this https URL, accepted at EMNLP 2026 Main Conference (Oral) |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); Graphics (cs.GR); Machine Learning (cs.LG) |
| Cite as: | arXiv:2510.14980 [cs.AI] |
| (or arXiv:2510.14980v4 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2510.14980 arXiv-issued DOI via DataCite |
Submission history
From: Zhen Liu [view email]
[v1]
Thu, 16 Oct 2025 17:59:58 UTC (27,792 KB)
[v2]
Sun, 19 Oct 2025 05:35:03 UTC (23,882 KB)
[v3]
Tue, 1 Sep 2026 06:03:40 UTC (27,235 KB)
[v4]
Sun, 11 Oct 2026 15:51:38 UTC (27,234 KB)
来源：arXiv cs.AI · arxiv.org