arXiv cs.CY· Louise Yorke·· 11 小时前AI 评分34
超越准确率：AI 保障在公共服务中需要确立什么
Beyond accuracy: what AI assurance needs to establish in public services
AI 导读
研究用 8 种 GPT-4o 配置对 1,200 条英国首相问答做 21 类政策主题分类，准确率 68.2%–70.3%，但 24.3% 的问题在重复运行中标签发生变化，且各配置之间的一致性高于与人工编码的一致性。
正文 · 原文
Abstract:Public bodies increasingly rely on AI-produced evidence, including classifications, summaries and assessments. These outputs can inform policy decisions and service delivery, making their reliability a matter of public accountability. Establishing whether this evidence can be relied upon requires testing its outputs and examining how organisations decide whether those tests are sufficient. We combine technical evaluation with analysis of AI governance requirements, informed by measurement theory and management-based regulation. We compare eight GPT-4o configurations classifying 1,200 UK Prime Minister's Questions into 21 policy topics, against human-coded labels and across unchanged repeats. Accuracy ranges from 68.2% to 70.3%, yet 24.3% of questions receive different labels, with changes in topic shares and rankings. Configurations agree more with one another than with human coding: agreement does not guarantee correctness. Under the implemented score-extraction procedure, the proportion flagged for human review ranges from 0.9% to 41.0%. We analyse 37 UK and EU policy documents. Both sets expect testing and oversight, but reporting expectations do not consistently cover changing answers, shared errors, selective evaluation, substantive conclusions or review routing. Some UK guidance explicitly addresses stability. The UK sample emphasises public bodies checking their own use; the EU places substantial testing duties on providers while leaving them important choices about measures and acceptance thresholds. Without appropriate checks, evidence could misrepresent policy issues, redirect attention or resources, and leave errors undetected. Public policy should require organisations to justify reliance through checks beyond aggregate accuracy, clear responsibility, access to expertise and scrutiny of results.
| Comments: | 20 pages, 6 tables, 2 figures |
| Subjects: | Computers and Society (cs.CY); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.06655 [cs.CY] |
| (or arXiv:2611.06655v1 [cs.CY] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06655 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Louise Yorke [view email]
[v1]
Wed, 4 Nov 2026 15:31:51 UTC (86 KB)
来源：arXiv cs.CY · arxiv.org