Jacob Cokson Leaves Anthropic Over Superintelligence Risks
The rapid pace of artificial intelligence development is raising concerns that autonomous systems could eventually escape human control. Cokson expressed these concerns in an X post on September 8 announcing his departure from Anthropic. He had spent the previous three years working on the pre-training of foundational models and had previously worked at OpenAI. He said his decision was driven entirely by the long-term risks associated with creating self-improving superintelligence.
The AI industry is currently caught in an intense technological race among major developers. Cokson argues that OpenAI, Anthropic, and other technology companies are competing to be the first to develop systems capable of outperforming humans in most cognitive tasks. Meanwhile, these companies lack reliable methods for ensuring the safety of such AI systems. Concern about falling behind competitors is pushing companies to accelerate their research despite recognizing the potential risks.
Evan Hubinger, an Anthropic researcher specializing in AI alignment, publicly supported Cokson’s position. Hubinger estimated the probability of AI causing human extinction within the next decade at more than 10%. This figure reflects his personal forecast rather than an official company position or established scientific consensus. There is currently no evidence that existing AI systems pose an immediate threat to society.
Cokson’s departure highlights the ongoing debate within the technology industry over the pace of AI development. Researchers continue to debate whether AI development should be slowed until effective control mechanisms for autonomous systems are established. Cokson sees greater coordination between technology companies and governments as the key to preventing a potential catastrophe.
