arXiv cs.AI· Prabal Vashisht, Zheng Hu, Ying Yang·· 12 小时前AI 评分31
可复现评估：生产级智能体安全提示词压缩的基础
Reproducible Evaluation as the Substrate for Safe Prompt Compression in Production Agents
AI 导读
研究者提出一套面向生产级对话智能体的离线评估框架，即使所依赖的实时企业数据变化也能保持可复现，并可在基准过期后按需重新生成。以该框架作为 GEPA 提示词优化的适应度函数，并加正确性门控防止奖励黑客，在面向商家的生产智能体上将完整提示词规模压缩 3.5-7.6%，三个独立被测模型族均未出现留出任务正确率回退，其中一个正确率还测得提升。
正文 · 原文
Abstract:Evaluating conversational agents that act on live, mutable enterprise data is hard: replayed sessions drift as backend state changes, and a benchmark built today can silently go stale as the data it describes moves on. We present an offline evaluation harness for a production conversational agent that stays reproducible despite the live data it operates on changing, and that can be regenerated on demand once it goes stale. We then use this harness as the fitness function for GEPA-based prompt optimization under a correctness gate that guards against reward hacking. Applied to a production merchant-facing agent, correctness-gated compression shrinks the agent's full prompt surface by 3.5-7.6% without regressing held-out task correctness, consistently across three independent subject-model families, and for one, correctness measurably improves.
| Comments: | 8 pages (4 pages main text plus references and appendices), 1 table. Accepted at the NeurIPS 2026 Workshop on Agents in the Wild |
| Subjects: | Artificial Intelligence (cs.AI) |
| ACM classes: | I.2.7; I.2.6 |
| Cite as: | arXiv:2610.31972 [cs.AI] |
| (or arXiv:2610.31972v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.31972 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Prabal Vashisht [view email]
[v1]
Thu, 22 Oct 2026 22:24:35 UTC (22 KB)
来源：arXiv cs.AI · arxiv.org