arXiv cs.CL· Antonio Franca, Alexander Tong·· 13 小时前AI 评分31
破解生成困惑度：为何无条件文本评估需要新指标
Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs New Metrics
AI 导读
扩散语言模型（DLM）的生成困惑度（gen-PPL）被证明不可靠：它会被"总是输出打分器最可能序列"的采样器刷低，而这类文本本身不连贯。研究构造了四个免训练采样器，在 LM1B 和 OpenWebText 上以非退化 unigram 熵达到或超过训练模型的 gen-PPL。作者据此用分布差异与多样性诊断重新评测近期 DLM，并提出一种基于压缩的差异指标来衡量局部覆盖与保真度。
正文 · 原文
Abstract:Diffusion language models (DLMs) have emerged as the leading non-autoregressive alternative to language modeling. Because their likelihoods are intractable, generative perplexity (gen-PPL) has become the de facto standard for reporting unconditional generation quality. We analyze why gen-PPL is unreliable as a measure of generation quality. Across samples, it is minimized by always emitting the scorer's most likely sequence. Per-sample unigram entropy does not measure this loss of diversity. Within a sample, gen-PPL rewards predictability under the scorer, which need not correspond to text quality. To make this failure explicit, we construct four training-free samplers that reach or beat trained-model gen-PPL on LM1B and OpenWebText at non-degenerate unigram entropy, while producing text that is incoherent by construction. We then rebenchmark recent DLMs using distributional discrepancies and diversity diagnostics, and compare their rankings with LLM judgments. As part of this evaluation, we introduce a compression-based discrepancy that measures local coverage and fidelity, the two directions of the KL divergence between real and generated text. We argue that DLM progress should be evaluated with a set of metrics that expose different properties of generated text.
| Comments: | Presented at the ICML 2026 Workshop on Structured Probabilistic Inference & Generative Modeling (SPIGM), Seoul, South Korea |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.08417 [cs.CL] |
| (or arXiv:2606.08417v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.08417 arXiv-issued DOI via DataCite |
Submission history
From: Antonio Franca [view email]
[v1]
Sun, 7 Jun 2026 02:35:56 UTC (57 KB)
[v2]
Sat, 1 Aug 2026 17:13:06 UTC (58 KB)
[v3]
Thu, 22 Oct 2026 10:36:49 UTC (567 KB)
来源：arXiv cs.CL · arxiv.org