arXiv cs.CL· Aidan McCarthy·· 12 小时前AI 评分18
LAMB：面向拉丁语的现代上下文语言模型
LAMB: A Modern Contextual Language Model for Latin
AI 导读
基于 ModernBERT 架构的拉丁语编码器 LAMB 发布，采用形态学感知掩码方案，在两阶段预训练中使用了超过 74B 拉丁语 token。研究同时推出 Acervus（24B BERT WordPiece token，含超 760K 条经 ByT5 修复的严重损坏 OCR 样本）和首个拉丁语语义三元组数据集 Corpus Trinum（超 100M 强弱三元组）。
正文 · 原文
Abstract:This article introduces Latin Modern BERT (LAMB), an encoder for the Latin language based on the ModernBERT architecture, pretrained in two phases on more than 74B Latin tokens using a novel morphologically informed masking regimen. We also present two corpora for the Latin language. Acervus contains 24B BERT WordPiece tokens, including over 760K samples from heavily damaged OCR transcriptions, repaired by a ByT5 model trained for Latin text repair. The second dataset, Corpus Trinum, contains a mixture of over 100M strong and weak Latin language semantic triplets. Corpus Trinum is the first Latin semantic triplet dataset, opening the door for Latin language models to be adapted to general sentence semantic similarity tasks. We construct an experimental grid using five tasks and seventeen subtasks, applying a variety of finetuning configurations and subword aggregation techniques for 15 pretrained encoders, 6 LAMB training checkpoints, and an ablation model designed to quantify the impact of different masking regimens. Using this benchmark, we rank Latin encoders and find that LAMB outperforms all other evaluated models on 16 of 17 subtasks, coming within 0.8 percentage points of state of the art on the seventeenth. We provide additional analyses relating Latin encoder performance to tokenizer behavior, syntactic structure, finetuning configuration, pretraining state, subword aggregation, and variation in source texts and domains. In these, we identify unique acquisition dynamics for Latin open class words during terminal decay phases, and establish that for some task categories, finetuning configuration selection can have a greater effect on downstream scores than additional pretraining tokens.
| Subjects: | Computation and Language (cs.CL) |
| ACM classes: | I.2.7; I.2.6 |
| Cite as: | arXiv:2610.31723 [cs.CL] |
| (or arXiv:2610.31723v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.31723 arXiv-issued DOI via DataCite |
Submission history
From: Aidan McCarthy [view email]
[v1]
Thu, 22 Oct 2026 11:07:14 UTC (404 KB)
来源：arXiv cs.CL · arxiv.org