arXiv cs.AI· Drew T. Nguyen, William Fithian·· 9 天前AI 评分32
对 AI 时间跨度的估计与效度：对 METR 图的统计学审视
On the estimation and validity of AI time horizons---a statistical look at the METR plot
AI 导读
研究基于 228 个任务和 26 个 AI，用样条与项目反应理论重新计算 METR 的 50% 时间跨度，放宽了任务 AI 难度与人类耗时对数呈线性关系的假设。
正文 · 原文
Abstract:METR's 50\% time horizon measures the human completion time of software tasks that an AI solves with 50\% probability, allowing AI capabilities to be expressed in interpretable units. On 228 tasks and 26 AIs, we recompute the time horizons using splines and item-response theory to relax the assumption that the AI difficulty of a task depends linearly on the log of human time. Our fitted spline can be interpreted as a function that \emph{converts} human time to AI difficulty; it is nearly flat in a region from 2--30 min but close to linear elsewhere. Hence, a time-horizon jump from 3 min to 30 min is much easier than one from 30 min to 5 hours despite the same multiplier of $10 \times$. Overall, we contribute time-horizon point estimates that perform better under a cross-validated suite of proper scoring rules, as well as diagnostic plots for assessing time horizons' construct validity. We suggest that time horizons be interpreted together with the diagnostic plots, especially as new time-horizon-based benchmarks are proposed or existing ones grow to include longer tasks.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.12466 [cs.AI] |
| (or arXiv:2610.12466v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12466 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Drew Nguyen [view email]
[v1]
Thu, 8 Oct 2026 17:59:50 UTC (330 KB)
来源：arXiv cs.AI · arxiv.org