arXiv cs.CL· Avinash Baidya, Xinran Liang, Ruocheng Guo, Xiang Gao, Kamalika Das·· 12 小时前AI 评分29
稀疏证据下的弱监督早期失败预警：面向对话与 LLM 智能体轨迹的轻量监控器
When Evidence is Sparse: Weakly Supervised Early Failure Alerting in Dialogs and LLM-Agent Trajectories
AI 导读
研究者提出一种仅用轨迹级结果训练的轻量外部监控器，用于在对话或 LLM 智能体轨迹展开过程中预警失败。其基于注意力的多实例学习（MIL）组件从轨迹结果中学习各轮贡献，在五个基准上使 Pareto 前沿质量（hypervolume）较朴素前缀监督提升 1-10%，完整系统配合 α-STOP 策略较 SOTA 触发策略提升 3-42%。
正文 · 原文
Abstract:Early failure alerting requires deciding, while a dialog or agent trajectory unfolds, whether it is likely to fail. Existing approaches broadly follow two directions. LLM-based uncertainty monitoring either elicits confidence from the acting model, tying reliability to its own self-assessment, or queries an external frontier judge at each prefix, incurring costly repeated inference. Trained monitors instead either rely on per-turn annotations, which are rarely available, or train every prefix to predict the terminal outcome without explicitly modeling where failure-indicating evidence appears. We address these limitations with a lightweight external monitor trained only from trajectory-level outcomes, motivated by a pattern across five benchmarks: turns rated highly relevant to failure are sparse and often late. Its attention-based multiple instance learning (MIL) component learns turn-level contributions from trajectory outcomes and distills them into online evidence features that complement prefix predictions. We pair the resulting failure estimates with $\alpha$-STOP, a single preference-conditioned policy that selects accuracy-earliness operating points at inference time. The predictor improves Pareto-frontier quality (hypervolume) by 1-10% over naive prefix supervision, while the full system improves it by 3-42% over state-of-the-art trigger policies without per-preference retraining. By avoiding repeated frontier-model calls, our lightweight monitor reduces measured per-decision serving cost by 1,103-3,749$\times$ and latency by 31-106$\times$ relative to the LLM-based uncertainty monitor (GPT-5.2).
| Comments: | 9 pages, 11 figures, and appendix |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC); Machine Learning (cs.LG) |
| Cite as: | arXiv:2606.05414 [cs.CL] |
| (or arXiv:2606.05414v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.05414 arXiv-issued DOI via DataCite |
Submission history
From: Avinash Baidya [view email]
[v1]
Wed, 3 Jun 2026 20:28:27 UTC (2,655 KB)
[v2]
Fri, 30 Oct 2026 23:03:01 UTC (2,857 KB)
来源：arXiv cs.CL · arxiv.org