arXiv cs.CL· Gregor Kornhardt, Moritz Piening, Jannis Chemseddine, Gabriele Steidl·· 9 天前AI 评分24
SOL：用双切片 Wasserstein 距离度量文本分布之间的差距
SOL: Measuring Gaps between Text Distributions by Double Sliced Wasserstein Metrics
AI 导读
研究者提出 SOL，一种基于双切片 Wasserstein 距离的文本分布度量方法，将每条序列表示为固定 Transformer 下隐状态的经验测度，再比较这些测度的分布，并证明当 Transformer 为单射时 SOL 是度量。实验显示 SOL 能检测分布性失败、还原预期模型趋势并提供稳定的基于样本的估计，研究还用它重新评估了多个在 OpenWebText 上训练的模型。
正文 · 原文
Abstract:Evaluating text generation requires measuring how well the generated distribution matches the data distribution. For autoregressive models, this is done by the perplexity. Diffusion and flow-based language models can only provide a likelihood bound, whose tightness differs between model families. Sample-based substitutes such as generative perplexity with entropy do not consider the distribution fit. We propose SOL,
a distance between text distributions. Each sequence is represented by the empirical measure of its hidden states under a fixed transformer and the distributions of these measures are compared by the double sliced Wasserstein distance. We prove that SOL is a metric if the transformer is injective. Experiments show that SOL detects distributional failures, recovers expected model trends, and provides stable sample-based estimates. We put forward SOL to fill the gap in the current evaluation protocol used for non auto-regressive models. As a first step we use SOL to re-evaluate a variety of models trained on OpenWebText.
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.06513 [cs.CL] |
| (or arXiv:2610.06513v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.06513 arXiv-issued DOI via DataCite |
Submission history
From: Gregor Kornhardt [view email]
[v1]
Mon, 5 Oct 2026 15:29:55 UTC (438 KB)
[v2]
Thu, 8 Oct 2026 15:22:16 UTC (438 KB)
来源：arXiv cs.CL · arxiv.org