跳到正文
arXiv cs.AI· Eeham Khan, Richard Khoury, Leila Kosseim·· 10 小时前AI 评分31

持续预训练何时有效?针对魁北克法语的方言适配受控研究

When Does Continual Pre-Training Help? A Controlled Study of Dialectal Adaptation for Qu\'ebec French

AI 导读

一项针对 9 个不同家族与规模 LLM 的受控研究显示,持续预训练(CPT)对魁北克法语适配的收益高度取决于基座模型对目标变体已有的能力。在每次实验 3000 万 token 的预算匹配协议下,CPT 有效时提升主要来自语料构成而非数据量,文学文本贡献了大部分增益。配对评测还表明 CPT 主要改善表达理解而非语法能力,并伴随聚合分数可能掩盖的阅读理解下降等能力权衡。

正文 · 原文

View PDF HTML (experimental)

Abstract:Continual pre-training (CPT) is widely used to adapt pretrained language models to dialectal and non-standard language varieties, yet it remains unclear when this additional training is necessary, what drives its benefits, and whether aggregate evaluation scores adequately capture its effects. In this paper, we present a controlled study of CPT for Québec French language understanding across nine LLMs spanning different families and scales. Using a budget-matched protocol of 30M tokens per experiment, our results show that the benefit of CPT depends strongly on the starting model's existing competence in the target variety. When CPT is beneficial, the improvements are driven more by corpus composition than by data volume, with literary text accounting for most of the gain. Paired Québec and standard French evaluations further show that CPT primarily improves expression comprehension rather than grammatical competence, and that corpus register determines which regional expressions are acquired. These gains also involve capability-specific trade-offs, including declines in reading comprehension that aggregate scores can conceal. Overall, these findings show that CPT for dialectal adaptation should not be treated as a uniformly beneficial default. Instead, adaptation strategies should begin with a careful assessment of the base model's existing competence, accompanied by source-level data analysis and task-level evaluation.
Comments: EACL 2027 Main Conference
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as: arXiv:2611.24699 [cs.CL]
  (or arXiv:2611.24699v1 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2611.24699

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Eeham Khan [view email]
[v1] Wed, 18 Nov 2026 16:26:03 UTC (51 KB)

来源:arXiv cs.AI · arxiv.org