arXiv cs.AI· Alexander Hickey, Joanna Knight, James Bishop·· 6 天前AI 评分32
语言模型能玩维基百科游戏吗？GPT-o3 mini、GPT-4o、QwQ:32B 与 DeepSeek-R1:70B 的导航能力对比
Can Language Models Play the Wikipedia Game?
AI 导读
研究用维基百科游戏测试语言模型的信息检索能力，让模型仅靠文章链接抵达目标条目，并在五种维基百科语言和五个学术文献数据集上对比了句嵌入模型与 LLM。GPT-o3 mini、GPT-4o、QwQ:32B 和 DeepSeek-R1:70B 表现最佳，成功率与人类相当，但无模型显著超越人类，且性能与排名随语言和主题变化。
正文 · 原文
Abstract:Language models increasingly support or automate information retrieval from document collections too large to index. This task is difficult when it requires specialist linguistic or scientific knowledge. We used the Wikipedia Game to test language models on an analogous information retrieval task. In the game, players must reach a target article using only article links. We compared sentence embedding models and LLMs across five Wikipedia languages and five academic-literature datasets. We construct very large local graphs of Wikipedia pages and of paper citations using OpenAlex and provide code to build and navigate them. This allows the study of language models' navigation capabilities without relying on third-party APIs. We compared the models with human and TF-IDF benchmarks. The Wikipedia Game format enables robust comparisons across techniques, languages, and domains. No model significantly outperformed humans, although the best models (GPT-o3 mini, GPT-4o, QwQ:32B, and DeepSeek-R1:70B) achieved comparable success. Performance and rankings varied across languages and topics, with implications for practitioners utilising these techniques.
| Comments: | 64 pages (14-page main text, references and 16 appendices), 17 figures, 10 tables |
| Subjects: | Artificial Intelligence (cs.AI); Information Retrieval (cs.IR) |
| Cite as: | arXiv:2610.13948 [cs.AI] |
| (or arXiv:2610.13948v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13948 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Alexander Hickey [view email]
[v1]
Fri, 9 Oct 2026 17:59:56 UTC (1,204 KB)
来源：arXiv cs.AI · arxiv.org