Anthropic researcher backs ex-employee’s AI warning, says human extinction could come within years

 Anthropic researcher Jacob Coxon resigned from the company.

Hours after Anthropic researcher Jacob Coxon resigned from the company warning that the race to develop increasingly powerful AI systems could pose an existential threat, another researcher at the company has backed his concerns, saying many AI developers believe the technology could potentially cause human extinction within the next few years.Samuel Marks, Anthropic’s scalable-oversight lead, said Coxon’s thread was “very worth reading” while stressing that he was sharing his views in a personal capacity and not speaking for the company.

‘AI could cause human extinction’

Offering what he described as a “bird’s-eye view” of AI risks, Marks said developers believe advanced AI could result in human extinction or similarly severe outcomes. He added that the level of concern is generally higher among more senior employees in the industry.Marks said AI companies continue developing increasingly capable systems despite these concerns because of commercial incentives and competition. Developers, he said, fear that other companies could build or deploy the technology less safely or potentially misuse it.

‘No robust way to align AI yet’

Marks also raised concerns about AI alignment, the effort to ensure advanced systems behave in line with human intentions and constraints.“We have methods that can nudge AIs towards better behavior, but nothing that can robustly align them,” Marks said. He suggested that one possible path being considered is developing AI systems capable of helping researchers align their successors more effectively.He further said many AI developers want the industry to slow down and spend more time understanding how to build advanced systems safely. Marks said he had signed an open letter calling for greater caution in frontier AI development.

What Evan Hubinger said

Another Anthropic researcher, Evan Hubinger, also backed Coxon’s warning, saying the industry does not yet have a clear plan to deal with the risks posed by superintelligence.Hubinger, who works in AI alignment, said he personally estimates there is a greater than 10 per cent chance that AI could “kill all humans” within the next decade.He said he believes Anthropic is trying its best, but argued that researchers are “not yet” on track to solve alignment for superintelligence.Hubinger’s comments came in response to Coxon’s resignation and his criticism of Anthropic and OpenAI. Coxon had accused both companies of racing towards self-improving superintelligence despite the potential risks.The comments from the Anthropic researchers come amid growing debate over whether the rapid development of increasingly capable AI systems is moving faster than researchers’ ability to understand and control them.Marks said he works on AI safety because he hopes his research can reduce the possibility of what he described as “extinction-level” outcomes.