arXiv cs.CL· Zixuan Liu, Gustavo Didier, Fangzheng Wu, Brian Summa, Zizhan Zheng·· 3 天前AI 评分26
Weighted Ensemble：面向语言模型稀有事件估计的免训练采样框架
Weighted Ensemble Sampling for Rare-Event Estimation in Language Models
AI 导读
Weighted Ensemble（WE）是一个免训练框架，用于估计语言模型稀有但严重失败事件的概率。它只从原始语言模型采样，通过周期性将采样预算重新分配给更可能到达目标稀有事件的前缀，并利用 LLM 内部表示作为前缀级信号来指导分配，无需额外学习提议分布。在覆盖小规模与大规模 LLM 的安全相关任务上，WE 提升了稀有事件概率估计的效率与可靠性。
正文 · 原文
Abstract:Rare but severe failures of large language models can become practically significant at deployment scale. Estimating their probabilities with direct Monte Carlo sampling is highly inefficient, while proposal-based methods typically require auxiliary learning of an additional proposal distribution. We introduce \textbf{Weighted Ensemble (WE)}, a training-free framework for rare-event estimation in language models. WE samples exclusively from the original language model and improves upon direct Monte Carlo by periodically reallocating the sampling budget toward prefixes that appear more likely to reach the target rare event. To guide this reallocation without learning a separate proposal distribution, we exploit internal LLM representations as prefix-level signals and empirically validate their effectiveness. We provide a theoretical analysis of the proposed algorithm. Experiments on safety-related tasks spanning both small-scale and modern large-scale LLMs demonstrate that WE improves the efficiency and reliability of rare-event probability estimation.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19202 [cs.CL] |
| (or arXiv:2610.19202v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19202 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zixuan Liu [view email]
[v1]
Tue, 13 Oct 2026 21:11:48 UTC (111 KB)
来源：arXiv cs.CL · arxiv.org