arXiv cs.CL· Vinko Sabol\v{c}ec, Angelos Katharopoulos, David Grangier·· 9 天前AI 评分24
有损压缩文本自编码器：Lossy Compressive Text Autoencoders
Lossy Compressive Text Autoencoders
AI 导读
研究提出一种自编码器架构，沿时间轴对隐藏表示做残差降采样与升采样，并配以残差低维离散瓶颈，实现文本的压缩潜在表示。在 web 文本数据上，其压缩表示达到 2.24 bits per byte，与无损文本压缩算法相当，同时保持较好的重建与下游任务表现。研究还分析了不同量化方法、训练目标与数据集，并用 BLEU 和 LLM 评判评估重建文本的表面与语义相似度，另在问答和语义文本相似度基准上做了评测。
正文 · 原文
Abstract:Our work explores learning a compressed latent representation of text, at the intersection of data compression and representation learning. We propose an autoencoder architecture that performs residual downscaling and upscaling of hidden representations along the time axis, with a residual low-dimension discrete bottleneck. We analyze our approach for different quantization methods, training objectives, and datasets. For different levels of compression, we evaluate the similarity between the original and reconstructed text both at the surface-level (BLEU) and at the semantic-level (LLM-based judge). Additionally, we evaluate our models on downstream question-answering and semantic text similarity benchmarks. Our approach results in compressed representations which are on par with lossless text compression algorithms at 2.24 bits per byte on web text data, while having good reconstruction and downstream task performance.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.10738 [cs.CL] |
| (or arXiv:2610.10738v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.10738 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: David Grangier [view email]
[v1]
Wed, 7 Oct 2026 18:08:40 UTC (14,137 KB)
来源：arXiv cs.CL · arxiv.org