Anthropic urges AI development ‘pause’ and conversation about risks

Anthropic has floated the idea of a worldwide “temporary pause” on AI development – and said it was going to convene “policymakers” to discuss the dangers of advanced AI – in its latest release touting the capabilities of its products. In a long post on Thursday, Anthropic detailed the progress of its AI model, Claude, towards “recursive self-improvement” – that is, being able to make better and more powerful versions of itself. Recursive self-improvement is a bugbear of AI safety researchers, viewed as the key step for AI to become superintelligent and therefore unleash widespread consequences on humanity. The idea features heavily in the widely read AI 2027 doomsday scenario of last year, which imagines AI agents designing more and more intelligent versions of themselves, one of which eventually kills all of humanity with a bioweapon in order to make room for more datacentres and solar panels. Anthropic’s post notes a “trend” of increasing capability in Claude which, “taken far enough and given enough compute … points to an AI system capable of fully autonomously designing and developing its own successor”. This, Anthropic said, may increase the risk of “humans losing control over AI systems”. To deal with this, Anthropic proposed to organise conversations where “policymakers, researchers, civil society and other AI companies can help answer some of the questions this piece raises”. …