arXiv cs.CL· Stefan Krsteski, Charlotte Meyer·· 12 小时前AI 评分38
无需 Rollout 即可预测任务难度：跨 17 个智能体基准的事前难度预测研究
Predicting Task Difficulty Without Rollouts
AI 导读
一项 arXiv 研究在覆盖编程、数学、机器学习、网页导航、函数调用等领域的 17 个智能体基准上研究事前任务难度预测，指出 AUC 可能掩盖较差的难度估计。研究发现 token 级熵是有用的预测信号，并展示预期难度与实际难度之间的残差可暴露数据污染、任务不可行等环境缺陷。该工作面向 LLM 智能体长周期任务中试错成本过高的瓶颈，可用于合成数据、基准设计与训练课程构建。
正文 · 原文
Abstract:A fundamental challenge in evaluating and training autonomous agents is measuring the intrinsic difficulty of the tasks they attempt. Estimating this quantity can be useful for environment designers creating synthetic data or benchmarks, as well as for constructing training curricula, thereby offering a way to reduce compute costs. Such an estimation becomes increasingly important as agents (particularly LLM-based) move into longer-horizon domains, where empirical trial-and-error becomes a severe computational bottleneck. However existing work is largely confined to static tasks and relies on evaluation metrics that, as we show, can give an incomplete picture of predictive performance. In this paper we study ex ante difficulty prediction across 17 agentic benchmarks spanning coding, mathematics, machine learning, web navigation, function calling, and other domains. We find that AUC can mask poor difficulty estimates, identify token-level entropy as a useful predictive signal, and demonstrate how residuals between expected and observed difficulty can expose environment flaws such as contamination and infeasibility.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2608.05797 [cs.LG] |
| (or arXiv:2608.05797v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2608.05797 arXiv-issued DOI via DataCite |
Submission history
From: Stefan Krsteski [view email]
[v1]
Thu, 6 Aug 2026 09:33:09 UTC (121 KB)
[v2]
Sun, 27 Sep 2026 07:09:00 UTC (121 KB)
[v3]
Thu, 15 Oct 2026 21:40:07 UTC (121 KB)
来源：arXiv cs.CL · arxiv.org