arXiv cs.CL· Zhitong Chen, Weifeng Zhang, Kai Yin, Zhiyuan Wang, Ali Mostafavi, James Caverlee·· 11 小时前AI 评分31
从表面搜索增益到选择失败：审计结构化工作流规划中的测试时推理
From Apparent Search Gains to Selection Failures: Auditing Test-Time Inference in Structured Workflow Planning
AI 导读
研究在 DisasterBench 上把测试时推理拆成发现、结构验证与最终选择三阶段，发现名义上的树搜索配置实际只是非自适应的完整工作流采样，用其扩展提示词的独立采样即可复现候选发现而无需建树。多数投票在每个候选池中会让 10-19% 任务已发现的正确工作流未被选中，抽查 60 例中 57 例确有工具选择或信息流错误。工具规格感知的验证器能挽回部分损失，但推翻的正确决策多于修正的。
正文 · 原文
Abstract:Test-time search can improve structured planning, but when methods differ in prompts, post-processing, and selection, it is unclear what an observed gain should be attributed to. We study this question on DisasterBench, where models compose typed multi-tool workflows, by decomposing inference into three observable stages: whether a correct workflow is discovered, whether it survives structural validation, and whether it is finally selected. Comparing independent sampling, a nominal tree-search configuration, and multi-operator search under controlled inference budgets on two model families, we find that (i) the nominal tree-search configuration realizes non-adaptive complete-workflow sampling, and independent sampling with its expansion prompt reproduces its candidate discovery without constructing a tree; (ii) structural validation never removes the last correct workflow in our evaluated candidate pools, whereas majority voting leaves a discovered correct workflow unselected on 10-19% of tasks per pool, with an audit confirming substantive tool-choice or information-flow errors in 57 of 60 examined cases; and (iii) a tool-specification-aware verifier recovers some of these losses but overturns more correct decisions than it fixes. Our results motivate auditing realized rather than nominal inference pipelines, and reporting discovery, validity, and selection separately.
| Comments: | 37 pages. Code and reproducibility artifacts: this https URL |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.00579 [cs.CL] |
| (or arXiv:2611.00579v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.00579 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhitong Chen [view email]
[v1]
Sat, 31 Oct 2026 03:10:12 UTC (625 KB)
来源：arXiv cs.CL · arxiv.org