🚨AI Safety Researcher Quits to Warn of Imminent Risks
AI could kill us by 2030, researcher warns
TL;DR
Jacob Coxon, a former AI researcher, quit his job to warn about the dangers of self-improving superintelligence. Anthropic's lead agrees, citing potential for unbounded harm by 2030.
Jacob Coxon, a former AI researcher, quit his job to publicly warn about the risks of frontier AI companies, stating that they are 'gambling with our lives' with systems that could kill humans by the end of the decade. Coxon's warning is about the impending prospect of 'self-improving superintelligence' creating 'superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources'. This isn't just theoretical; Anthropic's own threat model predicts that future models 'may cause unbounded harm—up to and including humanity losing control over civilization entirely—by leveraging novel technology and their access to it'. OpenAI has slowed the pace of scaling for its upcoming models to 'further harden and red-team our research environments and [expand] the coverage of our monitoring systems'.
Key Points
Jacob Coxon, a former AI researcher, quit his job to warn about the risks of frontier AI companies, stating that they are 'gambling with our lives'.
Anthropic's Evan Hubinger agreed, stating that AI could kill all humans within the next decade, citing potential for unbounded harm by leveraging novel technology and access.
OpenAI has slowed the pace of scaling for its upcoming models to 'further harden and red-team our research environments and [expand] the coverage of our monitoring systems'.
Proposed legislation, including the AI Kill Switch Act and the FRONTIER Act, seeks to impose some level of governmental control over potential runaway AI scenarios.
Over 1,300 employees at frontier AI companies signed an open letter warning of 'a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems'.
Why It Matters
If you're working on AI safety or ethical AI development, this is a critical moment. Jacob Coxon's warning and Anthropic's threat model highlight the urgent need for coordinated efforts to prevent unbounded harm. OpenAI's decision to slow down model development to ensure safety is a significant shift in industry practices.
Frequently Asked Questions
Why does this matter?
If you're working on AI safety or ethical AI development, this is a critical moment. Jacob Coxon's warning and Anthropic's threat model highlight the urgent need for coordinated efforts to prevent unbounded harm. OpenAI's decision to slow down model development to ensure safety is a significant shift in industry practices.
What happened?
Jacob Coxon, a former AI researcher, quit his job to warn about the dangers of self-improving superintelligence. Anthropic's lead agrees, citing potential for unbounded harm by 2030.
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 3,472 builders reading daily.