arXiv cs.CL· Bo Zou, Chao Xu·· 12 小时前AI 评分31
WORTH Benchmark：非完全协作信息下的证据发现——两集试点实验
WORTH Benchmark: Evidence Discovery under Not-Fully-Cooperative Information -- A Two-Episode Pilot
AI 导读
研究者提出 WORTH Benchmark，让模型在无线索标注的原创悬疑叙事中自主选取关键线索句、识别案件参与者并解释推理过程，两集共完成 70 次正式运行，其中 67 次可内容评分。
正文 · 原文
Abstract:Existing reasoning evaluations typically provide task-relevant information to models as available premises, rarely examining concurrently how models discover evidence autonomously and how information presentation within narratives guides or obscures judgment. This paper proposes the WORTH Benchmark: models read original mystery narratives without clue annotations, autonomously select key clue sentences, identify case participants, and explain their reasoning process. The narrative materials are designed by professional authors such that evidence facilitating reasoning coexists with narrative arrangements that delay the revelation of truth. We report a pilot experiment under full-story conditions across two episodes, conducting a total of 70 formal runs on models, of which 67 runs are content-scorable. Scoring separately records hits on author-designated key clue sentences and participant selections; natural language explanations are used for case diagnosis and are not scored independently. The results show that clue hits and participant judgments are not synchronized; across 37 content-scorable runs in Episode 02, 31 runs simultaneously selected the same set of incorrect participants, while adjacent suspicion-oriented and exoneration-oriented sentences were selected 25 times and 0 times, respectively. These phenomena illustrate the diagnostic value of component-wise analysis and suggest that not-fully-cooperative informational relationships may influence evidence selection and judgment. This study only implements full-story conditions for two episodes, and does not establish stable model rankings or infer general mechanisms on this basis; human player records have not yet been cleaned, and this version does not report human-model comparisons.
| Comments: | 31 pages. Pilot study, two episodes |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.27597 [cs.CL] |
| (or arXiv:2610.27597v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.27597 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Bo Zou [view email]
[v1]
Tue, 20 Oct 2026 04:04:06 UTC (46 KB)
来源：arXiv cs.CL · arxiv.org