arXiv cs.CL· Cl\'ea Chataigner, Rebecca Ma, Prakhar Ganesh, Yuhao Chen, Afaf Ta\"ik, Elliot Creager, Golnoosh Farnadi·· 9 小时前AI 评分31
AUGMENT:用基于用户行为的自动改写框架审计 LLM
Say It Another Way: Auditing LLMs with a User-Grounded Automated Paraphrasing Framework
AI 导读
研究者提出 AUGMENT 框架,基于用户行为生成受控的提示词改写,并通过指令遵循、语义相似度和真实性检查保证质量,用于 LLM 审计。在 BBQ 和 MMLU 数据集上的案例显示,受控改写能揭示无约束改写下被掩盖的系统性弱点。该工作已被 EACL 2026 接收。
正文 · 原文
Abstract:Large language models (LLMs) are highly sensitive to subtle changes in prompt phrasing, posing challenges for reliable auditing. Prior methods often apply unconstrained prompt paraphrasing, which risk missing linguistic and demographic factors that shape authentic user interactions. We introduce AUGMENT (Automated User-Grounded Modeling and Evaluation of Natural Language Transformations), a framework for generating controlled paraphrases, grounded in user behaviors. AUGMENT leverages linguistically informed rules and enforces quality through checks on instruction adherence, semantic similarity, and realism, ensuring paraphrases are both reliable and meaningful for auditing. Through case studies on the BBQ and MMLU datasets, we show that controlled paraphrases uncover systematic weaknesses that remain obscured under unconstrained variation. These results highlight the value of the AUGMENT framework for reliable auditing.
| Comments: | Accepted to EACL 2026 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2505.03563 [cs.CL] |
| (or arXiv:2505.03563v4 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2505.03563 arXiv-issued DOI via DataCite |
|
| Journal reference: | Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2026), Volume 1: Long Papers, pp. 1441-1467, 2026 |
Submission history
From: Cléa Chataigner [view email]
[v1]
Tue, 6 May 2025 14:17:30 UTC (959 KB)
[v2]
Wed, 21 May 2025 20:55:08 UTC (5,857 KB)
[v3]
Wed, 8 Oct 2025 23:43:57 UTC (2,629 KB)
[v4]
Mon, 16 Nov 2026 19:18:58 UTC (3,994 KB)
来源:arXiv cs.CL · arxiv.org