arXiv cs.CL· Giacomo Bais, Tejaswini Deoskar·· 2 天前AI 评分22
TagInsert：Transformer 序列生成中的动态顺序，低资源标注场景实验
Dynamic ordering in sequence generation in Transformers: experiments on low-resource Tagging scenarios
AI 导读
基于 Transformer 的 TagInsert 模型移除解码器侧注意力掩码，让模型在推理时自行决定标签生成顺序，从而打破常见的从左到右（L2R）生成范式。
正文 · 原文
Abstract:This paper introduces TagInsert, a Transformer-based model that challenges the common left-to-right (L2R) generation paradigm in Transformers, for sequence labelling tasks where dependencies may not naturally align with a L2R order. TagInsert lifts the L2R constraint by removing the decoder-side attention mask and enabling the model to decide a preferred order of tag generation at inference time. We experiment on the common Part-of-Speech tagging and the more complex Combinatory Categorial Grammar (CCG) supertagging tasks on multiple languages and low-resource data settings. We show that, although encoder-only models like BERT remain strong performers for tagging tasks, TagInsert frequently outperforms a L2R encoder-decoder model, achieves results comparable to encoder-only models under data-scarce conditions, and shows better results on the long-tail of rare categories. Our analyses show that TagInsert alleviates error propagation typical of autoregressive models by learning to defer difficult predictions to later stages in the decoding process.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.21862 [cs.CL] |
| (or arXiv:2610.21862v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21862 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Giacomo Bais [view email]
[v1]
Thu, 15 Oct 2026 13:36:46 UTC (301 KB)
来源：arXiv cs.CL · arxiv.org