arXiv cs.AI· Yuexi Pan, Yongchao Chen·· 14 小时前AI 评分31
SciValidityBench：面向科学智能体记忆的“使用有效性”基准
The Right Fact, the Wrong Decision: SciValidityBench for Scientific Agent Memory
AI 导读
研究者提出 SciValidityBench，一个可执行基准工厂，含 64 个 TaskSpecs，覆盖五个有效性模块与六个科学领域，用于检验智能体记住的事实是否“有资格”影响决策。配套 SciStateForge 编译器-读取器框架将分布式历史转为显式决策治理状态，在四个模型家族上相比通用 Event Graph 提升有效决策 9.2-18.8 个百分点，且仅靠检索无法复现该增益。
正文 · 原文
Abstract:Most agent-memory benchmarks frame long-term memory as retrieval: can an agent recover the relevant record from a long and evolving history? Scientific agents face a stricter problem: a correctly remembered fact may still be invalid to use. The same reported test result can support confirmation when the outcome and stopping rule were fixed in advance, yet only exploration when the outcome was selected from twenty inspected results. We call this scientific use validity: whether remembered evidence is eligible to influence a decision under the state governing its interpretation. We introduce SciValidityBench, an executable benchmark factory with 64 TaskSpecs spanning five validity modules and six scientific domains. TaskSpecs couple generated experimental histories with scientific-state definitions, programmatic decision oracles, controlled interventions, and automatic scoring. The SciStateForge compiler-reader framework turns distributed histories into explicit decision-governing state. Counterfactual pairs and targeted corruption-repair controls then test whether models use those dependencies and localize failures. Across four model families, compiled scientific state improves valid decision-making by 9.2-18.8 percentage points over a generic Event Graph. Retrieval alone does not reproduce this cross-family gain, while content-identical flat and typed views show that the improvement comes from exposing governing state rather than a particular graph serialization. SciValidityBench thus shifts memory evaluation from finding the right fact to deciding whether that fact should control the decision.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.34767 [cs.AI] |
| (or arXiv:2610.34767v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34767 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yuexi Pan [view email]
[v1]
Mon, 26 Oct 2026 03:12:57 UTC (295 KB)
来源：arXiv cs.AI · arxiv.org