arXiv cs.AI· Bohe Chen·· 23 小时前AI 评分34
用 Demonstration-Conditioned Prompt Search 审计提示词优化器:简单方法也能打
Simplicity Goes Far: Auditing Prompt Optimizers with Demonstration-Conditioned Prompt Search
AI 导读
研究将 APE 配方重构为 Demonstration-Conditioned Prompt Search(DCPS),作为机制归因对照,冻结生成器、以随机采样示例为条件、在验证集上打分选优,不做学习更新、反思或进化。
正文 · 原文
Abstract:Automatic prompt optimization is growing more elaborate. Recent methods stack policy-gradient updates, trace reflection, and evolutionary search. These methods share a generate-and-select step; without controlling for it, reported gains may conflate the contributions of optimizer mechanisms and larger sampling budgets. We isolate this step as Demonstration-Conditioned Prompt Search (DCPS), repurposing the Automatic Prompt Engineer (APE) recipe as a mechanism-attribution control: it freezes the generator, conditions it on randomly sampled demonstrations, scores candidates on validation data, and keeps the best---no learned update, reflection, or evolution. Across two optimizer families, DCPS replaces Proximal Policy Optimization (PPO) in StablePrompt, matching classification accuracy within 0.5 pp at roughly one-third the measured cost; in DSPy-based compound AI systems, under the reported budgets, it matches GEPA's macro accuracy on GPT-4.1-mini while using 55% fewer rollouts at 32% lower measured API cost on average. The audit reveals a task-dependent pattern: proposal and selection recover much of heavier optimizers' performance on structured and instruction-following tasks, while additional mechanisms retain advantages in some open-ended and reasoning-heavy settings. We recommend benchmarking prompt optimizers against generate-and-select controls and reporting rollout counts, token usage, and dollar costs alongside accuracy. Code and artifacts: this https URL.
| Comments: | 13 pages, 2 figures, 4 tables |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.22605 [cs.AI] |
| (or arXiv:2611.22605v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22605 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Bohe Chen [view email]
[v1]
Tue, 17 Nov 2026 11:22:27 UTC (50 KB)
来源:arXiv cs.AI · arxiv.org