arXiv cs.CL· Danny Kim, Fernando Koch·· 11 小时前AI 评分31
AI 智能体究竟需要多少对话历史？BEAM 基准上的 token 预算与得分权衡研究
How Much Conversation History Does an AI Agent Need?
AI 导读
研究在 1,100 道 BEAM 问题上固定检索排序与 reader，测试五种 passage 预算，发现平均得分随预算增大而上升，但高预算区间每增加一个 token 的得分增益递减。在带明确对话状态目标的受控任务中，保留用户消息时 Qwen 更常匹配完整状态，但当答案依赖对早前提议的引用时其完整状态匹配率下降；Mistral 在各保留条件下结果接近下限。评分审计未能确定 BEAM 答案的正确性。
正文 · 原文
Abstract:How does using less conversational history change reader-token use and benchmark scores? We examine this question on 1,100 BEAM questions. We held the retrieval ranking and reader constant across five fixed passage budgets, and mean assigned scores rose as the passage budget grew. The score gain per additional token became smaller at the upper tested budgets. Answerable questions generally received higher scores with more context. Pooled abstention scores sometimes rose as the passage budget shrank. We also tested message retention in a controlled task with explicit conversational-state targets and compared retained user messages with token-matched recency. Qwen matched the full state more often for direct requirements when user messages were retained. Its full-state matching declined when answers depended on references to earlier offers. Mistral produced near-floor results across the retention conditions. A scoring audit leaves the correctness of BEAM answers unresolved. These experiments characterize the token and score consequences of the tested memory budgets. Reliable budget selection depends on retaining the evidence each question requires and validating the measure used to assess answer quality.
| Comments: | 10 pages, 6 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.09662 [cs.CL] |
| (or arXiv:2611.09662v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.09662 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Fernando Koch [view email]
[v1]
Fri, 6 Nov 2026 16:09:04 UTC (165 KB)
来源：arXiv cs.CL · arxiv.org