arXiv cs.CL· Sabah Nushra, Md Kishor Morol·· 11 小时前AI 评分38
BILAL-BENCH：开放语言模型会在孟加拉语中编造伊斯兰教义来源吗
Do Open Language Models Invent Scripture? BILAL-BENCH: Benchmarking Islamic-Source Fabrication in Bengali
AI 导读
研究者推出 BILAL-BENCH，一个 240 题、专家核验的孟加拉语开放式伊斯兰来源编造基准，覆盖《古兰经》归属、圣训、教法、当代议题与对抗性虚假前提提示。
正文 · 原文
Abstract:When a language model invents a Qur'anic verse or hadith, it gives false authority to text that readers may treat as sacred. Existing Islamic hallucination benchmarks are largely Arabic and often closed-ended, while Bengali resources mainly study general-domain hallucination or factual knowledge. We introduce BILAL-BENCH, a 240-question expert-verified benchmark for open-ended Islamic-source fabrication in Bengali. It covers Qur'anic attribution, hadith, fiqh, contemporary issues, and adversarial false-premise prompts, and uses a two-axis rubric that separates source errors from assertiveness. We evaluate seven open-weight models from 1B to 120B on 1,680 double-annotated responses. Six of the seven models confidently fabricate on 79.3-95.3% of source-production questions, with no monotonic improvement from scale. Hadith provenance is the weakest category across models: only two of 315 responses are correct and correctly sourced. The two 8B BanglaLLaMA variants have similar confident-fabrication rates but different error profiles. Models vary far more in whether they withhold a claim they cannot support than in source accuracy: appropriate abstention or false-premise correction ranges from 6 to 46 responses across the benchmark, and does not track parameter count.
| Comments: | 15 pages, 6 tables. Accepted at the 7th Muslims in ML (MusIML) Workshop at NeurIPS 2026. Code and dataset: this https URL |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.40996 [cs.CL] |
| (or arXiv:2610.40996v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.40996 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sabah Nushra [view email]
[v1]
Thu, 29 Oct 2026 21:25:31 UTC (79 KB)
来源：arXiv cs.CL · arxiv.org