arXiv cs.CL· Khan Mahi Al Atahar, Nazifa Tasneem, Ajwad Abrar·· 3 天前AI 评分22
八款 LLM 的英语与孟加拉语摘要能力评测：GPT-5.6 Luna、LLaMA 3.3 70B、Gemini 2.5 Flash 表现分化
Evaluating Large Language Models for Multilingual Abstractive Summarization: A Comprehensive Study of English and Bengali
AI 导读
一项研究对八款闭源与开放权重 LLM 在规模匹配的英语和孟加拉语 XL-Sum 测试集上做了零样本评测，结合词汇与语义指标、无参考事实性验证及 LLM-as-a-Judge 多维评估。
正文 · 原文
Abstract:Abstractive summarization remains challenging across languages, where linguistic quality and factual consistency may diverge, particularly in low-resource settings. We present a standardized zero-shot benchmark of eight closed-source and open-weight LLMs on size-matched English and Bengali XL-Sum test sets. Our evaluation combines lexical and semantic metrics with reference-free factuality verification and multidimensional LLM-as-a-Judge assessment, providing a comprehensive analysis of both surface-level quality and factual reliability in multilingual summarization. Results reveal substantial cross-lingual variation: GPT-5.6 Luna leads several English lexical metrics but does not maintain the same advantage in Bengali, where LLaMA 3.3 70B achieves the highest ROUGE and BLEU scores. Gemini 2.5 Flash obtains the highest observed Bengali factuality score. Furthermore, evaluation methods produce different model rankings, highlighting the limitations of relying on a single metric for multilingual summarization assessment. These findings demonstrate that strong English summarization performance does not necessarily transfer to Bengali and emphasize the need for comprehensive evaluation frameworks for low-resource languages.
| Comments: | Accepted in MCT4SD 2026 (Manila, Philippines) |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.19003 [cs.CL] |
| (or arXiv:2610.19003v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19003 arXiv-issued DOI via DataCite |
Submission history
From: Nazifa Tasneem [view email]
[v1]
Tue, 13 Oct 2026 18:05:03 UTC (11 KB)
来源：arXiv cs.CL · arxiv.org