arXiv cs.CL· Aarne Talman, Stergios Chatzikyriakidis·· 10 小时前AI 评分34
捷径还是记忆?重新审视大语言模型在自然语言理解中的数据扰动实验
Shortcuts or Memorisation? Revisiting Data Corruption in Natural Language Understanding with Large Language Models
AI 导读
研究用 8 个 LLM 与微调编码器在 GLUE 和三轮 ANLI 上重做扰动实验,发现直接作答的系统在 MNLI 去掉名词后仍保留 36-72% 的超随机水平性能,而在 ANLI 上仅 14-29%。
正文 · 原文
Abstract:Pre-trained encoders fine-tuned on natural language inference (NLI) and other language understanding tasks keep much of their accuracy after the input is perturbed, e.g. by removing all nouns or shuffling the words. Does the same hold for large language models (LLMs), and if so, is it because they have seen the benchmarks during pre-training? In this paper we revisit these perturbation experiments with eight LLMs alongside fine-tuned encoders on GLUE and the three rounds of ANLI. Regardless of model size, every system that answers directly retains 36-72% of its above-chance performance without nouns on MNLI compared to only 14-29% on ANLI. Only a model that reasons before answering departs from this pattern. We also check contamination indicators (including exact $n$-gram counts in the public pre-training data of two of the models) and find no link between exposure and robustness. Our strongest evidence comes from 368 new, human-validated items that none of the models can have seen. All eight LLMs keep more of their performance without nouns on items written in MNLI's style than on items written without its cues. When the perturbation instead removes the function words carrying such cues, this advantage reverses. In light of these results, we argue that robustness to perturbation on familiar NLI benchmarks reflects dataset shortcuts rather than memorisation, and tells us more about the benchmark than about the model.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.16843 [cs.CL] |
| (or arXiv:2611.16843v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.16843 arXiv-issued DOI via DataCite |
Submission history
From: Aarne Talman [view email]
[v1]
Thu, 12 Nov 2026 14:18:44 UTC (69 KB)
[v2]
Thu, 19 Nov 2026 18:55:33 UTC (69 KB)
来源:arXiv cs.CL · arxiv.org