arXiv cs.CL· Dayam Nadeem, Safdar Mustafa, Ameen Ur Rehman, Shahab Saquib Sohail, Mohammad Nadeem, Usman Naseem·· 11 小时前AI 评分34
开放权重 LLM 圣训复述双语幻觉基准：3 万条回答无一精确匹配
They Know Where It Is, But Not What It Says: A Bilingual Hallucination Benchmark of Hadith Recall in Open LLMs
AI 导读
一项双语（英语/阿拉伯语）基准测试了 Mistral-7B、Gemma3-4B、Jais-7B、ALLaM-7B 和 Phi4-mini-3.8B 五款开放权重模型对六大逊尼派圣训集（每集 500 条、每模型每语言 3000 条，共 30000 条回答）的精确复述能力，结果 0% 的回答同时达到高词汇保真度（BLEU）与高语义保真度。
正文 · 原文
Abstract:Large language models are increasingly consulted for religious knowledge, including the recall of specific Hadith (recorded sayings and actions of the Prophet Muhammad, peace be upon him). Unlike open-ended generation, Hadith recall is a high-stakes, verifiable task: a reference (collection, chapter, and Hadith number) has exactly one correct text, and any deviation from it constitutes a fabrication that can be mistaken for authentic scripture. This study introduces a bilingual (English/Arabic) benchmark that probes five open-weight, locally-deployable language models (Mistral-7B, Gemma3-4B, Jais-7B, ALLaM-7B, and Phi4-mini-3.8B) across all six canonical Sunni Hadith collections (Sahih al-Bukhari, Sahih Muslim, Sunan Abi Da'ud, Jami' al-Tirmidhi, Sunan an-Nasa'i, Sunan Ibn Majah), sampling 500 Hadith per collection (3,000 per model per language, 30,000 responses total). Each model is given only the bibliographic reference and asked to reproduce the exact Hadith text, with no retrieval augmentation, and is scored on lexical fidelity (BLEU) and semantic fidelity (multilingual cosine similarity), with refusals flagged separately. Across all five models and both languages, 0% of 30,000 responses achieve both high lexical and high semantic fidelity, i.e., an exact, verifiable match to the source text. Arabic responses fare uniformly worse than English. These findings suggest that current open-weight LLMs, used without retrieval grounding, are unreliable and potentially dangerous sources for exact religious-text recall, and that deployment for Islamic scholarship should mandate retrieval-augmented generation over a verified corpus rather than parametric memory. The complete code, prompts, evaluation pipeline, and scored outputs are publicly available at this https URL.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.02587 [cs.CL] |
| (or arXiv:2611.02587v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.02587 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shahab Saquib Sohail PhD [view email]
[v1]
Mon, 2 Nov 2026 06:12:57 UTC (24,760 KB)
来源：arXiv cs.CL · arxiv.org