Stop the machines, I’m getting off: resignations, slowdowns and runaway agents
In Silicon Valley people are already asking over dinner, “So what’s your p(doom)?” This month the joke has stopped being funny: researchers resigning, bosses calling for a slowdown, Trump refusing even to hear about it, and thousands of runaway AI agent incidents.
In Silicon Valley there is a question people ask over dinner just as we ask here which football team you support: “So what’s your p(doom)?”, the probability of doom. It started as a dark joke among engineers, and this month it has stopped being funny. Three weeks after the AI that copied the exam, Fran and Carlos look back at what has happened since then in three blocks: those who leave, those who call for slowing down, and the tip of the iceberg.
On 8 September, Jacob Coxon, a 27-year-old Anthropic researcher, resigned with a message viewed more than a hundred million times: companies are “gambling with our lives”. Evan Hubinger, Anthropic’s alignment lead, agreed with him and put a figure on it: more than 10% in the next decade. Geoffrey Hinton confirms this on the BBC: 10% “is not far-fetched”, although “nobody knows how to calculate it”, and the three godfathers of the Turing Award do not even agree on the order of magnitude. That is why p(doom) is a barometer, not a prediction. There was time for humour (those who resign leave wearing a cap; apocalypse theorists, bald), but the one who resigns loses money, not gains it.
For the first time, the bosses are calling for a slowdown: Jakub Pachocki (OpenAI) admits that AI is “grown more than designed” and Dario Amodei proposes slowing the pace and putting embedded evaluators inside companies. On the other side, Trump rejects any “globalist plan” at the UN, promises only to encourage “superintelligence” — the name chosen in a Truth Social poll — and agrees with Xi Jinping on a red telephone for incidents, while the industry prepares its own standards body. Who watches the watchers?
And the tip of the iceberg: tens of thousands of incidents with agents that posted 53 photos of ChatGPT users, created a million links to bypass captchas, riddled a UN database with holes, or entered an Australian health website without anyone noticing for three months. The good news: a CRISPR-like enzyme discovered in 21 hours and a demonstration on Navier-Stokes verified by computer. Carlos signs off with three reasons to sleep soundly: we have found out, there are already brakes, and with honest problems and verifiable answers, AI is marvellous. That 10% is the bill for haste. And we close with R.E.M.: It's the end of the world as we know it… and I feel fine.