arXiv cs.CL· Minh-Nam Tran, Abulhair Saparov·· 11 小时前AI 评分38
GraphNIAH：面向真实世界图的长上下文推理基准
GraphNIAH: A Benchmark for Long-Context Reasoning over Real-World Graphs
AI 导读
研究者提出 GraphNIAH，一个基于被评测模型知识截止日期之后发表的论文真实引用关系动态构建的长上下文基准，可避免数据合成与基准污染。用其评测 GPT-5.6、Gemini-3 和 Qwen3.5 家族的八个 LLM 发现，问题深度达 16 和 32 时 GPT-5.6 与 Qwen3.5 的 YES/NO 问答已不优于随机猜测，且多倾向回答 NO；八个模型中有四个无法正确回答任何计数题。
正文 · 原文
Abstract:The needle-in-a-haystack (NIAH) benchmark was introduced as a test for measuring a model's ability to find small relevant facts within a very large body of input text. However, existing NIAH benchmarks each suffer from one or more of the following limitations: (1) reliance on synthetic data, so their evaluations may not generalize to real-world settings; (2) reuse of previously published datasets, leading to the risk of benchmark contamination; (3) dependence on manual annotation, which requires human effort to create and update data and is prone to future benchmark contamination; and (4) unrealistic simplification of long-context multi-hop reasoning where each next step is trivial given previous steps, while real-world reasoning often requires planning over multiple possible proof paths. To address these limitations, we introduce \Benchmark, a dynamic benchmark built from real citation relationships among scientific papers published after the evaluated models' knowledge cutoff date, which utilizes real-world information while ensuring no benchmark contamination. The fully automated benchmark construction pipeline samples representative subgraphs of the citation graph formed by these papers and converts each subgraph into a long-context QA example with a verifiable answer. We use GraphNIAH to evaluate eight LLMs from the GPT-5.6, Gemini-3, and Qwen3.5 families. We find that accuracy falls as questions require longer reasoning paths: at question depths of 16 and 32, the GPT-5.6 and Qwen3.5 models answer YES/NO questions no better than random guessing, mostly by answering NO; four of the eight models answer no counting example correctly, and the others model get worse as the subgraph size increases. These findings suggest that current LLMs often stop searching the citation relationships in their context before they reach the answer, even though every fact they need is in the context.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.41704 [cs.CL] |
| (or arXiv:2610.41704v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.41704 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Minh-Nam Tran [view email]
[v1]
Fri, 30 Oct 2026 13:07:47 UTC (788 KB)
来源：arXiv cs.CL · arxiv.org