Hacker News· yusufozkan·· 7 天前AI 评分22
AI 安全中的具体问题：机器学习系统的五类事故风险研究
Concrete Problems in AI Safety
AI 导读
论文将机器学习系统的"事故"定义为由设计缺陷导致的无意有害行为，并按成因归为三类：目标函数错误（避免副作用、避免奖励黑客）、目标函数评估成本过高（可扩展监督）、学习过程中的不良行为（安全探索、分布偏移），共列出五个实践性研究问题。作者梳理了各方向已有工作并提出面向前沿 AI 系统的研究建议，最后讨论了如何更有效地思考前瞻性 AI 应用的安全问题。
正文 · 原文
Abstract:Rapid progress in machine learning and artificial intelligence (AI) has brought increasing attention to the potential impacts of AI technologies on society. In this paper we discuss one such potential impact: the problem of accidents in machine learning systems, defined as unintended and harmful behavior that may emerge from poor design of real-world AI systems. We present a list of five practical research problems related to accident risk, categorized according to whether the problem originates from having the wrong objective function ("avoiding side effects" and "avoiding reward hacking"), an objective function that is too expensive to evaluate frequently ("scalable supervision"), or undesirable behavior during the learning process ("safe exploration" and "distributional shift"). We review previous work in these areas as well as suggesting research directions with a focus on relevance to cutting-edge AI systems. Finally, we consider the high-level question of how to think most productively about the safety of forward-looking applications of AI.
| Comments: | 29 pages |
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:1606.06565 [cs.AI] |
| (or arXiv:1606.06565v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.1606.06565 arXiv-issued DOI via DataCite |
Submission history
From: Jacob Steinhardt [view email]
[v1]
Tue, 21 Jun 2016 13:37:05 UTC (51 KB)
[v2]
Mon, 25 Jul 2016 17:23:29 UTC (52 KB)
来源：Hacker News · arxiv.org