arXiv cs.AI· Henrik Marklund, Alex Infanger, Benjamin Van Roy·· 11 小时前AI 评分31
研究者提出“灾难性表现”的正式定义，用于分析 AI 失准风险
A Definition of Catastrophic Performance
AI 导读
研究者提出“灾难性表现”的正式定义，以区分仅次优与灾难性结果，用于严格分析 AI 失准带来的灾难性风险。该定义基于两个参考值：baseline value 表示未部署任何智能体时的价值，orthogonal value 表示与人类偏好无关的轨迹价值，后者通常远差于前者。灾难性表现即智能体抵消了人类活动已带来的收益。
正文 · 原文
Abstract:A central challenge in AI is specifying objectives that align with human preferences. While existing theoretical results elucidate how misspecification impacts performance, they typically do not distinguish between merely suboptimal and catastrophic outcomes. To enable rigorous analysis of catastrophic risk from AI misalignment, we introduce a formal definition of catastrophic performance. Our definition is based on two reference values. The first, baseline value, captures the value attained if no agent is deployed. The second, orthogonal value, captures the value of a trajectory that is unrelated to human preferences, in a sense we describe. Orthogonal value is typically far worse than baseline value. We interpret these reference values through examples in which past human activity has altered the state of the environment to human benefit. Starting at the resulting state, an AI agent is deployed with the goal of accelerating further progress. Catastrophic performance represents this agent's undoing of benefits afforded by human activity.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.06219 [cs.AI] |
| (or arXiv:2611.06219v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06219 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Alex Infanger [view email]
[v1]
Wed, 4 Nov 2026 09:30:13 UTC (1,043 KB)
来源：arXiv cs.AI · arxiv.org