如果 AI 研发自动化引发智能爆炸会怎样
What if automating AI R&D triggers an intelligence explosion?
arXiv 论文评估 AI 研发自动化可能引发智能爆炸的初步证据，认为 AI 系统有望在几年内自动化大部分 AI 研发工作，甚至可能全部自动化。作者指出，若智能爆炸发生，AI 收益可能大幅提前，但也可能带来能力增速超出社会承受力、人类失去对超级智能系统的控制、国家与企业内部权力制衡被削弱等极端风险。论文建议政策制定者尽快提高对 AI 研发自动化的可见度，并研究引导和约束智能爆炸的方式。
Authors:Alan Chan, Christoph Winter, Andrew Barto, Jakub Pachocki, Geoffrey Hinton, Eric Horvitz, Yoshua Bengio, Dawn Song, Jack Clark, Hilary Greaves, Anton Korinek, Samuel Hammond, Thore Graepel, Ben Bariach, Philip H. S. Torr, Sheila A. McIlraith, Jeff Clune, Sam Manning, Girish Sastry, Tom Davidson, Daniel Eth, Sören Mindermann
Abstract:In contrast to even a year ago, AI systems now write most of the code inside the companies that build them. As more of the AI research and development (R&D) pipeline is automated, could AI progress radically accelerate in an "intelligence explosion," where years of advances are compressed into months or less? Preliminary evidence suggests that it could. In this work, we assess this evidence, analyze an intelligence explosion's potential impacts, and propose policy responses. AI systems are on track to automate most AI R\&D work within a few years, and possibly all of it. If this triggers an intelligence explosion, it could dramatically bring forward AI's benefits, but also pose extreme risks: capabilities growth could accelerate far beyond what society can keep up with, humanity could lose control over superhuman AI systems, and checks on power within and between states, companies, and branches of government could be severely eroded. Although there remains much uncertainty about these possibilities, the high stakes warrant serious further attention. Policymakers should urgently obtain more visibility into the automation of AI R&D, develop ways to steer and constrain an intelligence explosion, and prepare society to adapt to an intelligence explosion's impacts.
| Subjects: | Computers and Society (cs.CY); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.36054 [cs.CY] |
| (or arXiv:2609.36054v2 [cs.CY] for this version) | |
| https://doi.org/10.48550/arXiv.2609.36054 arXiv-issued DOI via DataCite |
Submission history
From: Alan Chan [view email]
[v1]
Mon, 28 Sep 2026 18:13:31 UTC (1,076 KB)
[v2]
Mon, 26 Oct 2026 09:06:37 UTC (664 KB)
来源：arXiv cs.CY · arxiv.org