arXiv cs.CL· Josef Jon, Rawan Bondok, Ond\v{r}ej Bojar·· 11 小时前AI 评分23
LLM 在阿拉伯方言机器翻译中的现状评估：16 种方言、Jais/Nile/GPT-4.1 等模型对比
Current state of LLMs for Arabic dialectal machine translation
AI 导读
一项基于 MADAR 数据集的研究评估了 LLM 在 16 种阿拉伯方言上的英阿双向机器翻译能力，覆盖 Jais、Nile 等阿拉伯专用模型，Gemma、Command-R、Mistral、Aya 等多语言模型以及 GPT-4.1 商业 API。
正文 · 原文
Abstract:This work presents an evaluation of large language models (LLMs) for English to dialectal Arabic machine translation on the MADAR dataset. We evaluate both translation directions (English to Arabic and vice-versa) on 16 Arabic dialects. Our experiments cover a diverse set of models, including specialized Arabic models (Jais, Nile), multilingual models (Gemma, Command-R, Mistral, Aya), and commercial APIs (GPT-4.1). We employ multiple evaluation metrics: BLEU, CHRF, COMET (both reference-based and reference-less variants) and GEMBA (LLM-as-a-judge), as well as a small-scale manual evaluation, to assess translation quality. We discuss the challenges of automatic MT evaluation, especially in the context of Arabic dialects. We also evaluate the ability of LLMs to classify the dialect used in a text. The study offers insights into the capabilities and limitations of current LLMs for dialectal Arabic machine translation, particularly highlighting the difficulty of handling dialectal diversity, although the results may be influenced by possible training data contamination, which is always a concern with LLMs.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.02271 [cs.CL] |
| (or arXiv:2611.02271v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.02271 arXiv-issued DOI via DataCite (pending registration) |
|
| Related DOI: | https://doi.org/10.18653/v1/2026.abjadnlp-1.41
DOI(s) linking to related resources |
Submission history
From: Josef Jon [view email]
[v1]
Sun, 1 Nov 2026 22:52:16 UTC (7,396 KB)
来源：arXiv cs.CL · arxiv.org