arXiv cs.CL· Jiaming Yu, Xiaotian Xie, Jian Ma, Yingjie Li·· 13 小时前AI 评分26
法语输入变体如何影响 LLM 准确率与可靠性：一项基准测试
Small Edits, Shifting Decisions: Benchmarking LLM Accuracy and Reliability on French Input Variants
AI 导读
研究者构建了覆盖 7 类任务的法语表层变体基准，含 4,200 组 original-low-high 三元组、共 12,600 条输入，评测两个 LLM 家族的六个模型。
正文 · 原文
Abstract:French users may omit apostrophes and diacritics or make keyboard typos in everyday input. The effects of these common variations on language model performance and prediction reliability remain insufficiently understood. To address this gap, we construct a controlled French surface-variant benchmark covering seven task types, with 4,200 original-low-high triplets (12,600 input instances). Each high-intensity variant retains all low-intensity edits and adds further changes. We evaluate six models from two LLM families. The results show that accuracy decreases and Brier score and negative log-likelihood (NLL) increase with perturbation intensity. Larger models show smaller accuracy losses under high intensity, but increases in Brier score and NLL do not consistently decrease with scale. Keyboard typos cause the largest losses in accuracy and the quality of predicted probabilities at the predefined high-intensity settings. Finally, calibration temperatures fitted on original inputs transfer effectively to surface variants. Combining calibration with input restoration further improves Brier score and NLL over either method alone while preserving restored accuracy. Our benchmark provides a unified framework for jointly evaluating task accuracy and prediction reliability under common French surface variations.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.33493 [cs.CL] |
| (or arXiv:2610.33493v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33493 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yingjie Li [view email]
[v1]
Sat, 24 Oct 2026 06:46:00 UTC (179 KB)
来源：arXiv cs.CL · arxiv.org