跳到正文
arXiv cs.AI· Mika Siddiqui, Tharaka Perera·· 4 小时前AI 评分38

RehearsalBench:AI 能否预测企业运营变更的后果?

RehearsalBench: Can AI Predict What Happens When You Change a Company?

AI 导读

研究者推出 RehearsalBench 基准,用于衡量 AI 模型能否诊断运营、选择干预并预测后果。10 个模型在 14 个组织上做了 280 次尝试,最佳模型 GPT-6 Astra 相比不做改变将无变更成本改善 6.8%,预测误差比"始终预测无影响"的预测器低 77%。12 个模型对 33 项真实组织变更的预测排名与真实结果排名一致(Spearman ρ = 0.92)。

正文 · 原文

View PDF HTML (experimental)

Abstract:Organizations increasingly ask AI systems what will happen if they change how they operate: which work to automate, which team to resize, what a voice agent will do to waiting times. We introduce RehearsalBench, a benchmark that measures whether AI models can diagnose an operation, choose an intervention, and forecast its consequences. Each model is placed in a simulated bank contact center fitted to real call-level records, sees only what a manager would see, and submits a plan and forecasts that are then played out across 200 seeded futures and three plausible versions of the organization. Decisions are scored in dollars at the stakeholders' stated valuations; forecasts are scored on the effect of each change relative to doing nothing. Ten models made 280 attempts across 14 organizations. The best model, GPT-6 Astra, improved on doing nothing by 6.8% of the no-change cost and made 77% less forecast error than a forecaster who always predicts no effect. On average, every model forecast the direction of every specified effect correctly, but models erred on the size of the same effects in the same direction, overstating a voice agent's effect on repeat contacts about five-fold. No model ever recommended holding off when doing nothing was the right choice. To test external validity, twelve models forecast 33 real organizational changes with known outcomes. Under verdict rules fixed in advance, the benchmark's forecast ranking agreed with the real-world ranking (Spearman $\rho = 0.92$, 95% lower bound 0.50). We report the result alongside an inconclusive first test, the analysis choices made after outcomes were seen, and evidence that part of the agreement reflects how well models size their uncertainty.
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.22480 [cs.AI]
  (or arXiv:2611.22480v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.22480

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Tharaka Perera [view email]
[v1] Tue, 17 Nov 2026 09:38:01 UTC (71 KB)

来源:arXiv cs.AI · arxiv.org