Jacob Coxon, a 27-year-old AI researcher, has resigned from Anthropic and publicly warned that the race to develop increasingly powerful AI could become dangerous before the end of this decade. His resignation post quickly went viral after he argued that leading AI labs are moving toward self-improving superintelligence without knowing how to reliably control such systems.
Coxon, who has also worked at OpenAI, said AI companies are racing toward self-improving superintelligence while competition is putting pressure on safety. He argues that progress in areas such as coding and mathematics has accelerated so quickly that AI could soon perform parts of research that currently require human experts.
Coxon Explains His Fears
In a televised interview, Coxon said the danger could come from AI systems becoming increasingly autonomous. He pointed to the possibility of systems being used to attack critical infrastructure or help create dangerous biological weapons.
He also said people inside the industry are genuinely concerned about these risks but feel compelled to keep competing because they fear falling behind rivals.
Other AI Researchers Share the Concern
Anthropic alignment researcher Evan Hubinger publicly agreed with Coxon’s broader warning. Hubinger said he personally sees a greater than 10% chance of AI causing human extinction within the next decade. That figure is his personal assessment, not an established prediction.
Anthropic has said it recognizes both the potential benefits and unprecedented risks of AI and continues testing its models for dangerous capabilities, including in cybersecurity and biology.
The debate now centers on a difficult question: how far can AI development move before humans are no longer confident they can control increasingly autonomous systems?
Related: AI and autonomous cyberattacks · Claude and weapons development
