arXiv cs.AI· Sina Heydari, Amirreza Abbasi, Mohsen Hooshmand, Majid Ramezani·· 12 小时前AI 评分24
CDAE：用对比去噪提升预训练语言模型的扰动鲁棒性
CDAE: Enhancing Perturbation Robustness in Pretrained Language Models with Contrastive Denoising
AI 导读
研究提出轻量级对比去噪自编码器 CDAE，通过联合优化对比目标与重建目标来精炼 BERT 嵌入，学习对同义词替换、掩码和词丢弃等扰动不敏感的表示。在多种扰动策略和强度下，CDAE 的嵌入相似度保持能力持续优于原始 BERT 嵌入和 SimCSE，且扰动越强差距越大。源代码已公开。
正文 · 原文
Abstract:Pre-trained language models have significantly improved sentence representation learning, yet their embeddings remain sensitive to textual perturbations such as synonym substitution, masking, and word dropout. This work proposes a lightweight Contrastive Denoising Autoencoder (CDAE) that refines pre-trained BERT embeddings by jointly optimizing contrastive and reconstruction objectives to learn perturbation-invariant representations. We evaluate the proposed framework under multiple perturbation strategies and strengths, and compare it against the original BERT embeddings and SimCSE. Experimental results show that CDAE consistently preserves higher embedding similarity under perturbation, with the gap widening as the perturbation strength increases, highlighting perturbation-invariant learning as a promising direction for improving the stability of sentence embeddings. The source code is publicly available at: this https URL
| Comments: | Preprint |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| ACM classes: | I.2.4; I.2.6 |
| Cite as: | arXiv:2607.28236 [cs.AI] |
| (or arXiv:2607.28236v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.28236 arXiv-issued DOI via DataCite |
Submission history
From: Sina Heydari [view email]
[v1]
Thu, 30 Jul 2026 14:04:08 UTC (741 KB)
[v2]
Fri, 16 Oct 2026 11:50:15 UTC (745 KB)
来源：arXiv cs.AI · arxiv.org