arXiv cs.CL· Bruce Changlong Xu, Muhammed Shahir Abdurrahman, Hashem Elezabi·· 12 小时前AI 评分18
Typhoon：面向掩码语言模型的基于梯度的任务特定掩码策略
Typhoon: Gradient-Based Task-Specific Masking for Masked Language Models
AI 导读
研究提出 Typhoon，利用 MLM 损失对 one-hot token 输入的梯度在线估计各 token 类型的贡献，并以指数移动平均维护显著性、校准成期望掩码率匹配目标预算的掩码分布。
正文 · 原文
Abstract:The choice of which tokens to mask is an under-examined design decision in masked language modeling (MLM). Standard pretraining masks tokens uniformly at random, whereas content-aware masking schemes can improve downstream transfer. We study masking as a task-adaptive component of the fine-tuning pipeline and introduce Typhoon, a masking strategy that uses the gradient of the MLM loss with respect to one-hot token inputs to estimate online how much each token type contributes to the objective. Typhoon maintains an exponential moving average of per-token-type saliency and calibrates these scores into a masking distribution whose expected masking rate matches a target budget under a token-independence approximation. We formalize the method and evaluate it against random masking and whole-word masking on two GLUE tasks, MRPC and CoLA, across three BERT-family backbones (TinyBERT, DistilBERT, and BERT-base) and five random seeds per configuration ($90$ training runs in total). Once seed variance is accounted for, no masking strategy is reliably better than the others on these tasks: on MRPC the gap between Typhoon and the best baseline stays within $0.004$ $F_1$, across all twelve Typhoon comparisons no paired test reaches significance, and every $95\%$ confidence interval contains zero. Inspecting Typhoon's learned distribution explains why: budget-preserving min-max calibration compresses the gradient ranking into a narrow band whose Kullback-Leibler divergence from uniform random masking is $7\times 10^{-4}$ nats per token, and widening the calibration spread by an order of magnitude in KL still leaves downstream accuracy unchanged within seed noise. Typhoon's apparent advantage in single-run experiments does not survive multi-seed evaluation, showing that gradient-based task-adaptive masking is competitive with, but does not outperform, uniform random masking at this scale.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2303.15619 [cs.CL] |
| (or arXiv:2303.15619v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2303.15619 arXiv-issued DOI via DataCite |
Submission history
From: Bruce Xu [view email]
[v1]
Mon, 27 Mar 2023 22:27:23 UTC (857 KB)
[v2]
Mon, 1 Jun 2026 20:44:51 UTC (364 KB)
[v3]
Tue, 20 Oct 2026 16:55:47 UTC (63 KB)
来源：arXiv cs.CL · arxiv.org