arXiv cs.CL· Dmitry Popov, Roman Bokhyan, Ekaterina Enikeeva, Artem Mekhraliev, Stepan Vysotsky, Nikolay Karpachev·· 12 小时前AI 评分31
超越提示词：LLM 机器翻译评估器的系统性研究
Beyond Prompts: A Systematic Study of LLM-Based Machine Translation Evaluators
AI 导读
研究系统考察了指令设计、few-shot 示例、基座模型选择与多步流程对 span 级和排序式机器翻译评估的影响，并在 WMT24、WMT25 英俄翻译上收集了新的专家 RATE 标注。
正文
Abstract:LLM-based evaluators have rapidly become a standard tool for machine translation evaluation, yet the design choices that shape their behavior remain poorly understood. We present a systematic study of how instruction design, few-shot demonstrations, backbone model choice, and multi-step workflows affect span-level and ranking-based MT evaluation. We collect new expert RATE annotations on WMT24 and WMT25 English-Russian translations, with transfer validation on MQM- and ESA-annotated benchmarks across five additional language pairs. Our experiments reveal four robust findings: instruction structure shapes evaluator behavior, high-quality demonstrations matter more than quantity, frontier backbones exhibit distinct evaluator profiles (recall-, precision-, or ranking-oriented), and fusing outputs across diverse backbones outperforms single-evaluator multi-call setups. Although evaluator design exposes a large combinatorial space, several discovered design principles transfer consistently across annotation protocols and language pairs. We synthesize these findings into ORBIT, a family of LLM-based MT evaluators. Our strongest configurations achieve expert-level performance in error span detection and in Accuracy and Fluency ranking at roughly 20x lower cost than expert human annotation.
| Comments: | Accepted to Findings of EMNLP 2026. 17 pages, 1 figure. Dataset and prompts: this https URL |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.33741 [cs.CL] |
| (or arXiv:2610.33741v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33741 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Dmitry Popov [view email]
[v1]
Sat, 24 Oct 2026 15:30:59 UTC (73 KB)
来源：arXiv cs.CL · arxiv.org