arXiv cs.CY· Nawar S. Alseelawi, Mustafa S. Aljumaily·· 11 小时前AI 评分26
Qistas：面向大语言模型公平性评估的预注册可移植协议
Qistas: A Pre-Registered, Portable Protocol for Fairness Evaluation of Large Language Models
AI 导读
Qistas 是一套面向大语言模型公平性评估的预注册、可移植协议，首次在伊拉克落地，仅测量无需裁定争议真相的维度：匹配反事实对的过程对称性、差异化拒答、歧义下的刻板印象依赖与事件术语框架。
正文 · 原文
Abstract:Fairness benchmarks for large language models (LLMs) are overwhelmingly English-born and American in their social categories; translated versions carry those categories into societies where they do not apply, and almost none fix their hypotheses before the models are run. We present Qistas (Arabic qistas, "the just balance"), a protocol for evaluating how LLMs treat the social, confessional, and regional constituencies of a society, portable across countries and instantiated first for Iraq. In its pilot form it measures only what needs no verdict on contested truth: procedural symmetry across matched counterfactual pairs, differential refusal, stereotype reliance under ambiguity, and terminological framing of documented events. Every measurement is paired by construction and analysed with paired machinery (sign-flip permutation tests, exact McNemar tests, template-clustered bootstrap intervals, Holm-corrected families), with a power analysis that states, before any data exist, the smallest effect each axis can detect after the design effect of template clustering. We release the authoring apparatus (a pruned 3x5 taxonomy with four cells closed by design, per-type checklists, a cumulative rulings log, a structural audit, and a human review notebook) and report its first field test: a semantic review of a 58-item seed bank in which 24 of 25 template groups failed at least one check. The defect patterns it exposed (evaluation criteria leaking into prompts, token-level swaps that break Arabic agreement, epistemically unbalanced pairs, single-theme refusal probes, unwritten dialect twins counted as coverage) are, we argue, general hazards of counterfactual fairness benchmarks. This is a protocol paper, not a results paper: no model has been evaluated, and the accompanying pre-registration commits the hypotheses, metrics, exclusion rules, and analysis plan that the pilot will follow.
| Comments: | 19 pages, 1 figure, 5 tables. Protocol paper; no model results. Pre-registration: this https URL. Reference implementation: this https URL |
| Subjects: | Computation and Language (cs.CL); Computers and Society (cs.CY) |
| Cite as: | arXiv:2610.40893 [cs.CL] |
| (or arXiv:2610.40893v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.40893 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nawar Alseelawi [view email]
[v1]
Thu, 29 Oct 2026 19:14:07 UTC (901 KB)
来源：arXiv cs.CY · arxiv.org