MBBQ：面向生成式 LLM 刻板印象跨语言比较的数据集

View PDF HTML (experimental)

Abstract:Generative large language models (LLMs) have been shown to exhibit harmful biases and stereotypes. While safety fine-tuning typically takes place in English, if at all, these models are being used by speakers of many different languages. There is existing evidence that the performance of these models is inconsistent across languages and that they discriminate based on demographic factors of the user. Motivated by this, we investigate whether the social stereotypes exhibited by LLMs differ as a function of the language used to prompt them, while controlling for cultural differences and task accuracy. To this end, we present MBBQ (Multilingual Bias Benchmark for Question-answering), a carefully curated version of the English BBQ dataset extended to Dutch, Spanish, and Turkish, which measures stereotypes commonly held across these languages. We further complement MBBQ with a parallel control dataset to measure task performance on the question-answering task independently of bias. Our results based on several open-source and proprietary LLMs confirm that some non-English languages suffer from bias more than English, even when controlling for cultural shifts. Moreover, we observe significant cross-lingual differences in bias behaviour for all except the most accurate models. With the release of MBBQ, we hope to encourage further research on bias in multilingual settings. The dataset and code are available at this https URL.

Comments:	Accepted to COLM 2024
Subjects:	Computation and Language (cs.CL)
Cite as:	arXiv:2406.07243 [cs.CL]
	(or arXiv:2406.07243v4 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2406.07243 arXiv-issued DOI via DataCite

Submission history

From: Vera Neplenbroek [view email]
[v1] Tue, 11 Jun 2024 13:23:14 UTC (550 KB)
[v2] Tue, 18 Jun 2024 15:33:33 UTC (550 KB)
[v3] Wed, 17 Jul 2024 08:49:22 UTC (560 KB)
[v4] Thu, 15 Oct 2026 13:59:12 UTC (561 KB)