arXiv cs.CY· Mir Tafseer Nayeem, Davood Rafiei·· 3 天前AI 评分27
谁的数据，谁的基准？一个以用户为中心的评估资源审计框架
Whose Data, Whose Benchmark? A User-Centric Framework for Auditing Evaluation Resources
AI 导读
研究者提出一个以用户为中心的审计框架，区分贡献基准内容的用户与评估结果所代表的目标用户，并在内容、元数据、文档和评分流程四个层面操作化七个维度。对 16 个资源的审计发现，11 个文本基准中美国中心惯例占匹配信号的 62.4%-94.4%，5 个图文资源的地理可见度与各国 GDP 份额强相关（Pearson r 均值 0.780）。
正文 · 原文
Abstract:Benchmarks shape how model capabilities are measured and interpreted, yet their scores can obscure whose conventions, contexts, and judgments define successful performance. We introduce a user-centric audit framework that distinguishes users who contribute benchmark content from those whose needs evaluation results are intended to represent. The framework operationalizes seven dimensions across benchmark content, metadata, documentation, and scoring procedures. Across 16 resources, we find that American-centered conventions account for 62.4-94.4% of matched signals in 11 text benchmarks, while geographic visibility in five image-text resources is concentrated and strongly associated with national GDP share (mean Pearson r = 0.780; Spearman p = 0.778). Documentation reveals a systematic source-target gap: all 16 resources operationalize source-user context, but none operationalize target-user context. Paired scoring experiments further show that meaning-preserving answer variants reduce DROP exact-match accuracy from 100.0% to 33.3%, while TruthfulQA exhibits metric-dependent reductions in reference similarity. Our framework and documentation checklist make user assumptions auditable, clarifying what benchmark scores represent and where evaluation claims are supported.
| Comments: | Preprint |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY); Emerging Technologies (cs.ET); Human-Computer Interaction (cs.HC) |
| Cite as: | arXiv:2610.19268 [cs.CL] |
| (or arXiv:2610.19268v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19268 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mir Tafseer Nayeem [view email]
[v1]
Tue, 13 Oct 2026 22:39:55 UTC (2,314 KB)
来源：arXiv cs.CY · arxiv.org