arXiv cs.AI· Dipin Khati, Shubham Mehrotra, Bin Bi, Zhou Yu, Denys Poshyvanyk, James Zhu, Sitaram Asur, Phil L. Mui·· 10 小时前AI 评分31
AgentScript-Eval：面向企业 DSL 代码生成的 LLM 与智能体基准测试
AgentScript-Eval: Benchmarking LLMs and Agents for Code Generation in an Enterprise DSL
AI 导读
Salesforce 推出 AgentScript-Eval，用 100 个真实任务评测 LLM 与智能体编写企业 DSL AgentScript 的能力，最佳模型 pass@3 仅 25.0%，所有模型从零编写任务得分均为 0%。
正文 · 原文
Abstract:Domain-specific languages (DSLs) are widely used in enterprise software, giving a compact interface to complex tasks, from data access to infrastructure automation. Platforms increasingly extend them to agents, where a developer declares the topics, permitted actions, and guardrails of an agent rather than writing general-purpose code. Large language models (LLMs) write general-purpose code well, but that ability has been measured almost entirely on languages abundant in public pretraining data, and enterprise DSLs are neither abundant nor public. We introduce AgentScript-Eval, an execution-graded benchmark of 100 real tasks for AgentScript, the declarative DSL that Salesforce uses in Agentforce to build conversational agents. Tasks span authoring an agent from a specification, debugging a broken definition, and optimizing a working one. Evaluating five models zero-shot, we find that the regime is hard: the best model reaches only 25.0% pass@3, and every model scores 0% on authoring from scratch. We further find that model scale and price are weak predictors of success -- a 31B open-weight model attains the highest compilation rate of any model tested (48.0%) and matches the strongest frontier model on partial-credit accuracy at the lowest token cost. Adding an agentic compile-and-edit loop is the single largest lever, roughly doubling partial-credit accuracy. Beyond that gain, the wording of domain guidance matters more than its packaging. The same skill pack scores 28% as plain documentation but only 14% as a live plugin. Rewriting one skill inside that plugin then raises it to 37%, our best result, with catalog and tooling untouched. Larger models did not reliably outperform smaller ones; what mattered was the harness surrounding the model and the guidance supplied to it.
| Comments: | Accepted to 40th Conference on Neural Information Processing Systems (NeurIPS 2026). Workshop: 2nd Workshop on Agentic AI Benchmarks and Applications for Enterprise Tasks (AABA4ET), NeurIPS 2026 |
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.13183 [cs.SE] |
| (or arXiv:2611.13183v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13183 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Dipin Khati [view email]
[v1]
Mon, 9 Nov 2026 20:04:32 UTC (19 KB)
来源：arXiv cs.AI · arxiv.org