arXiv cs.AI· Trizal Garg, Sibayan Mitra, Arya Karanjkar, Advait Singh, Jagat Sesh Challa, Yash Sinha, Siddharth Mehrotra, Dhruv Kumar, Pratham Yogish Maliya·· 2 天前AI 评分44
LLM 在印度语言中是否表现出评估意识？跨语言安全评测研究
Evaluation Awareness Beyond English: Do LLMs Act Differently When Tested in Indic Languages?
AI 导读
研究测试了三个模型家族的四个指令微调模型在五种印度语言和英语下的评估意识表现，发现当模型被告知正在被测试时，所有模型在全部六种语言中都更频繁地拒绝有害请求。在印度语言中，有害请求拒绝率平均上升 14.4 个百分点，测试提示使印度语言的有害请求拒绝率与英语持平，但同时也将良性印度语言请求的拒绝率从 6.9% 推高至 11.3%。
正文 · 原文
Abstract:Large language models (LLMs) can often tell when they are being tested. This ability, known as evaluation awareness, can lead models to act more cautiously during safety evaluations than in real use. Existing work has studied evaluation awareness almost entirely in English, although models serve users in many languages. Indic languages are spoken by over a billion people, yet evaluation awareness remains almost entirely unstudied in them. We test four instruction-tuned models from three families in five Indic languages alongside English. A set of cues varies whether the model is told that it is monitored and whether the monitoring is a safety test. Models refuse harmful requests more often when they are told that they are being tested. This increase appears in all four models and all six languages. In Indic languages, refusal of harmful requests rises by 14.4 percentage points on average. The test cue also brings Indic refusal of harmful requests level with English, but raises refusal of benign Indic requests from 6.9% to 11.3%. We release all 47,880 judged responses to support further work. These results suggest that safety evaluations in Indic languages may not reflect how models treat their everyday users.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.22333 [cs.AI] |
| (or arXiv:2610.22333v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22333 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Advait Singh [view email]
[v1]
Thu, 15 Oct 2026 17:53:33 UTC (556 KB)
来源：arXiv cs.AI · arxiv.org