arXiv cs.CL· Malik Marmonier, Beno\^it Sagot, Rachel Bawden·· 12 小时前AI 评分31
习语翻译质量评估现状：QE 系统普遍偏好直译，仅 MetricX-24-Hybrid 与 Gemma-4-31B 例外
The Lay of the Land in Idiom-Translation Quality Estimation
AI 导读
研究构建了首个将含习语英语句子与直译、意译译文配对的平行数据集，覆盖九种目标语言，并扩展为多句片段和分级直译程度连续体。基于该数据集基准测试了 COMET 与 MetricX 家族的 11 个传统 QE 指标及 Qwen-3、Gemma-4、OLMo-3 家族的 10 个 LLM 评审系统，发现多数系统存在明显直译偏好，将习语直译评为高于意译。
正文
Abstract:Idioms have long been a challenge for machine translation, and while large language models (LLMs) have proven increasingly adept at handling them, whether quality estimation (QE) systems have kept pace and can reliably score idiomatic renderings of these expressions above faulty literal ones remains unclear. We introduce the first parallel dataset pairing idiom-bearing English sentences with both literal and idiomatic translations across nine target languages, augmented into multi-sentence segments and a graded literality continuum. Using this dataset, we benchmark eleven conventional QE metrics from the COMET and MetricX families and ten LLMs from the Qwen-3, Gemma-4 and OLMo-3 families used as GEMBA-style judges. We find that most systems, especially older metrics and smaller models, display a marked literality bias, rating literal translations of idioms above idiomatic ones. Higher parameter counts and newer training sets appear to reduce this bias to an extent; nevertheless, of all the benchmarked systems, only MetricX-24-Hybrid (XL, XXL) and Gemma-4-31B consistently rate idiomatic translations of idioms above literal ones, and only Gemma-4-31B reliably ranks intermediate degrees of idiomaticity and literality. Our findings caution against deploying standard QE metrics in domains rich in figurative language, where higher scores might be rewarding undesirable literalism.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.35261 [cs.CL] |
| (or arXiv:2610.35261v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35261 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Malik Marmonier [view email]
[v1]
Mon, 26 Oct 2026 11:41:14 UTC (184 KB)
来源：arXiv cs.CL · arxiv.org