arXiv cs.CL· Jos\'e Roberto Homeli da Silva, Felipe Jorge do Nascimento, Lucas Coutinho Freitas, William Alberto Cruz Casta\~neda, Marcellus Amadeus·· 10 小时前AI 评分22
Brasilidade：面向大语言模型的巴西身份认同评测基准
Brasilidade: Measuring Brazilian Identity for Large Language Models
AI 导读
研究者推出 Brasilidade 基准，用于评估 LLM 在巴西葡萄牙语上的表现，并已开放首个领域的任务，覆盖语言细微差别、语法与巴西葡萄牙语变体。结果显示模型规模本身并不决定表现，多个中等规模模型在评测中超过更大模型；针对葡萄牙语专门训练的模型总体也未达到平均水平。不同任务难度差异显著，部分任务对多数模型很容易，另一些则构成极端挑战。
正文 · 原文
Abstract:This work presents a new benchmark that aims to address the limitations of current methods for evaluating LLMs in Brazilian Portuguese. We introduced Brasilidade, an assessment methodology comprised of standardized tasks distributed across four different domains, and we are making available the tasks from its first domain, which address nuances of the language, grammar, and variations of Brazilian Portuguese. The results show that model size, by itself, does not determine performance; several medium-sized models outperformed larger models in the protocols evaluated. Specific models for Portuguese, in general, did not perform above average, despite their targeted training. Different task levels revealed significant nuances: while some tasks proved easy for most models, others posed extreme challenges, and some yielded mixed results, with wide variations in performance. This clearly illustrates the need for benchmarks that address the limitations of the models in Brazilian Portuguese, allowing future training cycles to seek to overcome them. Project repository this https URL
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.12906 [cs.CL] |
| (or arXiv:2611.12906v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.12906 arXiv-issued DOI via DataCite |
Submission history
From: William Alberto Cruz Castañeda PhD [view email]
[v1]
Thu, 15 Oct 2026 02:14:13 UTC (4,237 KB)
来源：arXiv cs.CL · arxiv.org