arXiv cs.AI· Mohammad Reza Rezaei, Dingxian Wang, Brett Levert, Celeste Fejzo, Andrew Rabinovich, Zhao Chen·· 12 小时前AI 评分40
PAYWORTHY：LLM 能否预测 AI 智能体将完成哪些自由职业工作？
PAYWORTHY: Can LLMs Predict Which Freelance Jobs an AI Agent Will Complete?
AI 导读
研究提出 PAYWORTHY 基准，含 1,494 条真实自由职业招聘帖（从 2020 至 2026 年间 18.8M 条中采样），测试 18 个模型在智能体尝试前预测任务成功率的排序能力。
正文 · 原文
Abstract:Before an AI agent takes on a paid job, someone has to judge whether it is likely to succeed. We test whether frontier language models can make this call before any attempt. We propose PAYWORTHY which contains 1{,}494 real freelance job postings, sampled from 18.8M postings made between 2020 and 2026 on a freelancing platform. An agent attempts each job from the posting alone in a single run, and an LLM judge acting as the client decides whether to accept the work. We ask 18 models to forecast how likely the agent is to succeed and measure how well each model's forecasts rank the accepted jobs above the rejected ones (AUC, where 0.5 is a random ranking). The answer depends on how the agent is described. Told what the agent cannot do, every model ranks the jobs well (AUC 0.66--0.86). Told instead that it is a generic agent with a browser and code execution, the same models rank the same jobs about as well as a random ordering (0.45--0.56). These results hold under judges from four other model families (Claude, GLM, Kimi and Qwen), and the informed forecasts also rank the verdicts of two human raters.
Code is available at this https URL
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.05759 [cs.AI] |
| (or arXiv:2611.05759v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.05759 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mohammad R. Rezaei [view email]
[v1]
Tue, 3 Nov 2026 23:59:53 UTC (1,440 KB)
来源：arXiv cs.AI · arxiv.org