arXiv cs.CL· Lin Sun, Heming Zhang·· 11 小时前AI 评分34
生产环境 LLM 系统的外部经验服务：面向部署的质量-成本权衡研究
External Experience Serving in Production LLM Systems: A Deployment-Oriented Study of Quality-Cost Trade-offs
AI 导读
研究将外部经验服务视为面向部署的质量-成本权衡问题，在真实生产审核场景中对比无经验基线、随机经验对照、全局提示词注入与基于检索的选择性注入。结果显示，当经验依赖具体案例时，选择性检索比无条件全局注入提供更优的运行点，且检索质量比单纯提高 Top-K 更重要。相同服务策略在短输出与解码密集型场景下的成本收益差异显著，外部经验只有在服务接口与任务成本结构匹配时才值得付出在线成本。
正文 · 原文
Abstract:Production LLM systems accumulate reusable operational experience, but the practical deployment issue is not merely whether such experience can help. It is how different serving strategies trade off quality against online cost under realistic constraints. Injecting external experience can improve task quality, yet it also increases prompt burden, latency, and serving pressure. We study \textit{external experience serving} as a deployment-oriented quality-cost trade-off problem. We evaluate this question in a real production moderation setting, with tool-use and GPQA as supporting contrast tasks that expose different output-cost regimes. We compare no-experience baselines, random experience controls, global prompt injection, and retrieval-based selective injection, and analyze both task quality and serving cost. The results show that, once experience becomes case-dependent, selective retrieval provides a stronger operating point than unconditional global injection. They further show that retrieval quality matters more than simply increasing Top-$K$, and that the same serving policy can exhibit substantially different cost-benefit profiles across short-output and decode-heavy regimes. These findings suggest that external experience is best treated as a selective, cost-aware serving decision rather than as a universal add-on. Overall, in the settings studied here, external experience pays off only when both the serving interface and the task-specific cost structure make its quality gains worth the online cost.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2606.11806 [cs.CL] |
| (or arXiv:2606.11806v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.11806 arXiv-issued DOI via DataCite |
Submission history
From: Lin Sun [view email]
[v1]
Wed, 10 Jun 2026 08:38:55 UTC (2,147 KB)
[v2]
Mon, 2 Nov 2026 01:33:55 UTC (246 KB)
来源：arXiv cs.CL · arxiv.org