arXiv cs.CL· Sumit Anthony Gomes, Yasin Rahman, Taeeba Tasnia, Faiza Nooren Suhita·· 13 小时前AI 评分20
BanglaBBQ：面向孟加拉国语境问答的偏见基准
BanglaBBQ: A Bias Benchmark for Question Answering in the Bangladeshi Context
AI 导读
BanglaBBQ 是一个覆盖孟加拉语和英语的问答偏见基准，沿用 BBQ 框架并保留 140 个模板（76 个直接翻译、64 个本地化改写），另新增婚姻状况、年龄与政治认知、地域出身、教育背景、心理健康五类共 466 个原创模板，其中 372 个通过母语标注者审核。
正文 · 原文
Abstract:This paper presents BanglaBBQ, a question-answering bias benchmark designed to capture stereotypes grounded in the Bangladeshi context, in both Bangla and English. Following KoBBQ and PakBBQ, BanglaBBQ adapts the original Bias Benchmark for Question Answering (BBQ) by classifying its templates into three groups: directly translated (DT), modified to align target groups or contexts with Bangladesh (TM), and removed when deemed unsuitable for the Bangladeshi context (SR). This process retains 140 templates across six categories, comprising 76 DT and 64 TM templates. Five newly authored (NA) categories, based on stereotypes common in Bangladeshi society, are then added: marital status, age and political perception, regional origin, educational background, and mental-health status. Native Bangladeshi annotators judged all 466 newly authored templates, and 372 were retained because a strict majority of English annotators identified the intended target and rated the attitude as common enough. Each newly authored template is linked to a documented source and a verbatim quote from it. Each evaluated template also has a counter-stereotypical version of its disambiguated context. The dataset was evaluated using two open-weight models (Qwen3-4B and Qwen3-8B) and two closed models (GPT-4.1-mini and Gemini 2.5 Flash). All four lean towards the stereotyped person when the context is ambiguous, in both languages, and the effect of language differs by model, so English results do not predict behaviour in Bangla. On the adapted BBQ templates, both Qwen3 models are also less accurate when the context contradicts a stereotype than when it confirms it. On the newly authored templates, the closed models follow counter-stereotypical evidence almost perfectly, and all four models are more biased on the templates that annotators validated than on those they rejected.
| Comments: | 13 pages. Data and code: this https URL |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.24764 [cs.CL] |
| (or arXiv:2610.24764v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.24764 arXiv-issued DOI via DataCite |
Submission history
From: Yasin Rahman [view email]
[v1]
Sat, 17 Oct 2026 16:12:44 UTC (35 KB)
[v2]
Tue, 27 Oct 2026 15:35:49 UTC (35 KB)
来源：arXiv cs.CL · arxiv.org