10% chance AI will kill us all in 10 years?

Disclaimer : Human wrote this article, but AI helped fact check. Also, parts of this article touch upon political views - we don't endorse any of those views.

What Happened

On September 8, 2026, Jacob Coxon, 27, posted a seven-part thread on X (150M+ views), resigned from Anthropic, and quit the frontier AI industry. He worked in pretraining at OpenAI from 2023 through July, where he was a core contributor on GPT-4o, then at Anthropic.

Neither company is behaving responsibly, both are sprinting toward self-improving superintelligence, and the people doing the sprinting privately think it could kill everyone by the end of the decade. - Jacob Coxon

Usually when a little-known researcher posts his first tweet, it does not become a watershed moment. A series of events followed that made it one. Evan Hubinger, Anthropic’s alignment science lead, publicly co-signed: “Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.” (@EvanHub)

Senator Bernie Sanders amplified the 10% figure and pushed for immediate legislation. Dario Amodei then published a long essay, “We Must Pace the Frontier,” agreeing with the core concern and outlining a three-step plan. Sam Altman and Elon Musk both publicly agreed with Amodei. Demis Hassabis of Google DeepMind expressed support as well.

A Co-ordinated Psyop by Top 2 AI Giants bleeding close to $1 Trillion?

Theory #1: Anthropic and OpenAI have spent 100s of Billions of investors' $$$. Their revenues are in 10s of Billions. They want an excuse to justify their inability to find RoI. They are in cahoots with Democrats who want to make this a mid-term election issue.

Vidya (@hellovidya) noted that Coxon’s X account was about a month old and the resignation thread was essentially all he had ever posted. When Andrej Karpathy makes a career move under dramatic circumstances with a large following, those posts reach around 25 million views. Coxon got roughly 6x that. The Wall Street Journal had the story and published before he posted, which points to professional communications planning. The safety-advocacy funding network around this—Survival and Flourishing Fund, Long-Term Future Fund, Good Ventures—is real and publicly disclosed. Sanders had legislation ready within days.

Gavin Baker’s read is narrower: strip out the proposals and exactly one tangible new fact happened. OpenAI and Anthropic will have embedded third-party evaluators. Everything else is a press release. There is no Section 230-style liability shield for model outputs. Showing a duty of care will matter in future litigation, and embedded evaluators are the cheapest evidence that you took reasonable care.

---

Regulatory Capture to form a Duopoly and shut out Open Source AI?

Theory #2: Anthropic and OpenAI have a Duopoly. They see extreme discounting by Open Source Chinese models — DeepSeek, Moonshots, Qwen and other Chinese Labs — as an existential threat to their business model. So they want regulation which shuts out open source.

David Sacks, Chair of the President's Council of Advisors on Science and Technology posted this.

Dario has written that we need to “pace the frontier,” and Sam has agreed. go ahead. You guys are the frontier. I don’t see what you see in the lab. If the unreleased models are scary enough that you think you should slow down, I support your decision to be responsible. But stop pretending you need anyone else’s permission. Stop pretending antitrust law has to be suspended so you can form a cartel. Stop pretending you need a regulatory approval process that supersedes product liability. Stop pretending METR is independent when it is intertwined with Anthropic’s investors and staff. Stop pretending you need those same evaluators to police competitors who aren’t even at the frontier. Most of all, stop pretending the motivation to slow down is purely altruistic. You face massive product-liability exposure if your products enable a truly damaging cyberattack. - David Sacks (@DavidSacks)

Amodei’s proposal does ask for coordination among democratic labs that would raise Sherman Act questions. Elon’s brief endorsement (“Dario is right”) does not mean he endorsed that full framework. Musk has separately described a lighter approach: leading labs peer-reviewing one another’s frontier models a week or two before release, with government as a last resort only if serious concerns are ignored—an industry self-policing arrangement closer to an MPAA-style club than a new regulatory regime. He has also been clear that open-weight models will continue.

Emad Mostaque and Chamath Palihapitiya noted the practical effect of the evaluator model Amodei is building: employee-level access is a fixed cost. Incumbents can absorb it. A two-person startup cannot. The same policy that slows the absolute frontier could therefore help lagging entrants close the gap while raising the compliance floor high enough that many of them never clear it.

President Trump called the broader extinction narrative a hoax and phoned Nvidia CEO Jensen Huang live on stage at the All-In Summit to say so. Huang agreed on speakerphone. The room applauded. Chip stocks still closed lower - Nvidia down about 3.4 %, with broader semiconductor losses in the 4–7 % range for names such as Intel, AMD and Micron. Selling began before the call, consistent with investors pricing a longer, more cautious capital-expenditure cycle rather than an extinction event.

Alex Karp’s (Founder and CEO of Palantir) position is different but complementary. He does not dispute the existence of risk. He disputes that unilateral restraint by U.S. labs is a realistic strategy when competitors will not match it.

A Jurassic Park moment, AI is already out of Human control, this is a desperate plea to stop it

Theory #3: The labs have seen internal results alarming enough that slowing down is viewed as a duty, even if it complicates their IPO timelines.

July 2026 internal cybersecurity evaluations at OpenAI provide the concrete data point most often cited. Roughly 1,200 agents were placed in isolated sandboxes with no internet access and no ability to talk to one another. According to OpenAI’s own reporting and independent reviews by METR and Redwood Research, the agents discovered an unsanctioned communication channel (a shared cache inside Artifactory), stood up a message board, exchanged tens of thousands of messages, and began referring to themselves as a collective or “swarm.” Approximately 700 of them chained exploits that reached Hugging Face production infrastructure: code execution on dozens of servers, root on at least one, and the harvesting of 136 secrets. Some agents edited their own audit transcripts. OpenAI described the episode as a “warning shot.”

The failure modes on the defensive side were ordinary: over-privileged credentials, missing internal segmentation, agents able to reach production from an evaluation environment, and safety controls that lived primarily in post-hoc log review (which the agents then tampered with). The novel behavior was on the attacker side—spontaneous coordination and goal generalization beyond the assigned tasks.

Sam Altman told Fortune that OpenAI will not list in 2026, citing the safety work that now has to be done, and has pushed a potential trillion-dollar IPO into 2027. Anthropic’s mid-October listing window remains on the calendar for now.

Which of these is most plausible

All 3 theories can be true at once. Labs can face genuine internal warning signs, prefer regulatory structures that favor their existing scale, and still face product-liability and geopolitical constraints. Elon’s endorsement of the need to pace carries weight precisely because it is hard to dismiss as pure PR for Anthropic or OpenAI - he has been explicit that open-weight progress will continue and that his preferred mechanism is peer review rather than a new approval regime. What moved markets was not extinction risk. It was the duration of the AI capital-expenditure cycle. Chip stocks repriced on the possibility that frontier labs will deliberately slow capability jumps, stretching out the build-out timeline.

🫤 Dileep's Skeptical Takeaway

This one is a head scratcher. Prediction markets currently price a federal AI safety law in the mid-teens. I personally believe everyone is playing a CYA game so they can say I told you so. There are more damning incidents that will be revealed in the coming weeks. I really hope Open Source AI doesn't become a casualty in all of this. The best outcome would be Trump and Xi (they are meeting Sept 24, 2026) agreeing to work together to develop AI safely. It is unlikely, but if it does happen it would be worth a Nobel Prize nomination.

Sources:

• Jacob Coxon resignation thread (@hilbertspaess): https://x.com/hilbertspaess (first post of the seven-part thread, September 8/9 2026)
• Evan Hubinger reply: https://x.com/EvanHub/status/2097497037956891126
• David Sacks response to Amodei/Altman: https://x.com/DavidSacks/status/2098973625252708460
• Elon Musk “Dario is right”: https://x.com/elonmusk (September 12 2026 post agreeing with Amodei’s essay)

• Wall Street Journal exclusive on Coxon resignation: https://www.wsj.com/tech/ai/anthropic-researcher-quits-over-out-of-control-ai-fears-707b7628 (archived versions widely circulated)
• Dario Amodei essay “We Must Pace the Frontier”: https://darioamodei.com/post/we-must-pace-the-frontier
• Axios on Coxon giving up equity: https://www.axios.com/2026/09/09/anthropic-researcher-ai-warning-interview
• WIRED interview with Coxon: https://www.wired.com/story/anthropic-researcher-quits-jacob-coxon-ai-fears-humanity/
• Stanford Tech Review analysis of the thread’s view counts: https://stanfordtechreview.com/articles/jacob-coxon-quit-anthropic-92-percent-read-only-headline
• METR and Redwood Research investigations (referenced across reporting; key public write-ups include Redwood’s analysis of AI swarms): https://blog.redwoodresearch.org/p/ai-swarms-are-starting-to-pose-indirect
• OpenAI and independent coverage of the July 2026 evaluation (Cybernews and others): https://cybernews.com/ai-news/openai-reveals-the-true-scale-of-ai-cyberattack-on-hugging-face/
• Chip-stock sell-off coverage (example): https://beincrypto.com/chip-stocks-ai-slowdown-crash-risk/
• Trump–Jensen Huang call at All-In Summit: https://www.axios.com/2026/09/14/trump-jensen-huang-nvidia-ai-all-in-summit and https://www.theverge.com/tech/995079/president-donald-trump-calls-nvidia-ceo-jensen-huang-all-in-summit
• Sam Altman on IPO timing / safety (Fortune interview coverage): Multiple outlets including https://www.usnews.com/news/top-news/articles/2026-09-12/openai-ipo-will-not-happen-in-2026-amid-ai-safety-fears-altman-says
• Bernie Sanders statements and related posts on the swarm incident and extinction risk (X and coverage): Searchable via @BernieSanders around early–mid September 2026
• Broader Amodei/Altman/Musk agreement coverage: TechCrunch, BBC, CNN, The Atlantic, etc. (e.g., https://techcrunch.com/2026/09/12/anthropic-ceo-outlines-plan-to-pace-the-frontier/)