arXiv cs.CL· Zhenshuai Yin, Jianshu Liang, Xuyue Xiong, Yongxin Guo, Mingda Zhang, Zhongze Luo, Haoyue Liu, Lianmin Zhang, Xiaoying Tang·· 14 小时前AI 评分31
LADDER：诊断噪声反馈下的领域级自我评估
LADDER: Diagnosing Domain-Level Self-Assessment Under Noisy Feedback
AI 导读
LADDER 用于研究语言模型在不可靠反馈下如何评估自身能力。研究发现，模型能准确复述给定的表现信息，却难以从交互经验中形成可信的自我评估；看似准确的评估会掩盖能力差异，也不一定能预测后续任务表现。研究据此提出，可信的自我评估应以实际任务表现及其预测价值为衡量标准。
正文 · 原文
Abstract:Knowing when they are reliable is essential for language models to support human-AI collaboration. Yet interaction feedback is not always complete or correct, and may mislead models' understanding of their own capabilities. We introduce LADDER to study how language models assess themselves under unreliable feedback. Our findings show that models can accurately report supplied performance information while still struggling to form trustworthy self-assessments from interaction experience. Apparently accurate assessments can obscure capability differences and do not necessarily predict future task performance. These findings suggest that trustworthy self-assessment should be evaluated against observed task performance and its predictive value. Our study provides empirical evidence of the limits of model self-knowledge and a basis for evaluating capability awareness in reliable human-AI collaboration.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.34737 [cs.CL] |
| (or arXiv:2610.34737v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34737 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhenshuai Yin [view email]
[v1]
Mon, 26 Oct 2026 02:38:34 UTC (2,512 KB)
来源：arXiv cs.CL · arxiv.org