arXiv cs.CL· Madhu Sudhan Rao Beesetty, Surbhi Sharma, Devesh Maheshwari, Lokesh Meesala·· 11 小时前AI 评分31
研究揭示罗马化印度语言生成评测盲区：自动指标高分难掩模型实际失败
The Invisible Gap: Romanized Indic Generation Fails Where Automatic Metrics Succeed
AI 导读
研究显示 BGE-M3、BERTScore 等自动指标会系统性误判罗马化印地语、泰卢固语和卡纳达语生成质量：在 72 条诊断样本上母语者给 GPT-5.5 的评分比 Claude Opus 4.8 高 1.43 分（12 分制），而全数据集 BGE-M3 均值仅差 0.004。
正文 · 原文
Abstract:Hundreds of millions of Indians communicate with digital systems in romanized Latin characters -- typing "aap kaise hain" rather than native script -- yet romanized Indic generation is absent from NLP benchmarks and evaluation frameworks. We demonstrate that automatic metrics (BGE-M3, BERTScore) systematically misrepresent the quality of romanized Hindi, Telugu, and Kannada generation. We evaluate four LLMs (Claude Opus 4.8, GPT-5.5, Gemini 2.5-flash, Qwen3.5-35B) on 142 OpenAssistant query-response pairs. On a 72-response diagnostic subset, native speakers rate GPT-5.5 1.43 points higher than Claude Opus 4.8 on a 12-point rubric, whereas their full-dataset BGE-M3 means differ by only 0.004. Two distinct model-specific failure modes emerge: Gemini produces script spillage (16.9% of Telugu responses leak native script), while Qwen collapses to English-only output (16.2% of Kannada responses). We introduce a failure-aware evaluation policy that assigns zero to empty outputs, script-corrupted responses, and instruction collapses -- revealing failures metrics alone cannot detect. Our results show that evaluating romanized Indic generation requires a coordinated bundle: semantic similarity, script compliance, failure coverage, and native-speaker acceptability.
| Comments: | 11 pages, 2 figures, 5 tables. All authors contributed equally |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.09085 [cs.CL] |
| (or arXiv:2611.09085v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.09085 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Madhu Sudhan Rao Beesetty [view email]
[v1]
Fri, 6 Nov 2026 07:17:32 UTC (1,488 KB)
来源：arXiv cs.CL · arxiv.org