arXiv cs.CL· Marianne Menglin Liu, Rongguang Wang, Zheng Wang, Yewen Wang, Leonid Boytsov, Tao Sheng, Sujith Ravi, Dan Roth·· 11 小时前AI 评分38
LLM 智能体能否胜任前置工程师？企业级 Text-to-SQL 的需求发现与交付研究
Can LLM Agents Be Forward-Deployed Engineers? Requirement Discovery and Delivery for Enterprise Text-to-SQL
AI 导读
一项基于真实企业 text-to-SQL 场景的研究测试了 LLM 智能体能否充当前置工程师（FDE），在 15 个模型中，最强模型能发现超过 80% 的隐藏需求，开源权重模型为 40-50%。
正文 · 原文
Abstract:LLM coding agents can increasingly automate model post-training, but whether they can act as forward-deployed engineer (FDE) agents, discovering customer requirements and turning them into deployment decisions, remains unclear. We study this on a real enterprise text-to-SQL engagement: a simulated customer, grounded in the real use case's requirements, starts with an ambiguous request. An agent must discover acceptance criteria, data permissions, and hardware constraints in the discovery phase, then plan and execute a post-training recipe in the delivery phase. Across 15 models, the strongest discover over 80% of withheld requirements and open-weight models discover 40-50%; in the delivery phase, only 25% of runs across eight coding-agent configurations pass all acceptance criteria. We identify four bottlenecks in this engagement: (1) incomplete discovery, with 40% of withheld requirements missing or misrepresented; (2) unreliable execution; (3) underused compute budgets and a lack of systematic comparisons across base models and post-training methods; and (4) benchmark accuracy does not guarantee production readiness. We suggest human review of discovered requirements, monitoring of the agent's plan during execution, and an explicit experiment-environment specification covering hardware, budget scope, and production checks.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.03551 [cs.CL] |
| (or arXiv:2611.03551v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.03551 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Rongguang Wang [view email]
[v1]
Mon, 2 Nov 2026 18:13:38 UTC (168 KB)
来源：arXiv cs.CL · arxiv.org