One day the unimaginable happens: Astronomers reveal to the public that they have discovered a previously undetected asteroid the size of New Jersey, and it will impact the Earth in ten years. Governments and private organizations scramble into action and quickly divide into two camps: One advocates destroying the asteroid using the majority of our nuclear arsenal, while the other prefers the launching of a massive ship to act as a gravity tractor and pull the asteroid very slightly off course. Calculations are made, a public debate ensues, and the merits of both plans are carefully weighed.
The next week, the astronomers admit they made a huge mistake: It’s not an asteroid, but a rogue planet nearly the size of Mars, and it will impact Earth in only three years. Both camps scramble to update their plans, positing new higher yield nuclear warheads and swarms of gravity tractors respectively. The debate on which course of action to take rages intensely, as we cannot afford to do both, and there is now no time to lose. Seemingly everyone has an opinion, even people who have never thought about this kind of thing before. Yet, late at night among friends, or two hours into a three hour podcast, insiders from both camps will quietly admit that the problem is too big and their plan probably won’t work.
I read with interest the recent debate between Nicholas Decker and Scott Alexander about “AI 2040”. The AI 2040 project is a planning exercise created by various AI safety researchers, and is written for an audience that believes a certain framing of the situation: That smarter-than-human AI is possible, that we’ll probably be able to create it in less than 10 years, and that our objective should be to “align” its values to human values so that it will only do the things we want it to do1. If we fail to align it properly, sufficiently advanced AI will achieve its goals rather than our goals, and we may not be able to stop it (since it’s smarter than us). This could be catastrophically bad for humans, meaning human extinction or something like a permanent AI-powered dictatorship.
The AI 2040 authors think our current path will end in disaster. They think frontier labs are competing to develop powerful AIs as fast as possible, without having enough time to research how to align them properly. Their proposed alternative plan (which they call “Plan A”) involves international agreements to slow down AI research, create full transparency into what each lab is doing, and eventually pause all AI capabilities research for several years. During that time we would use our army of AIs that are “only” about as smart as human experts to research a concrete plan for how to align future AIs that are even smarter. When we’re sure we know how to do this, we would unpause and resume intelligence research.
Decker disagrees with this kind of pause on several grounds2. First, we wouldn’t be able to get all the labs and governments to agree to do it. Second, even if they did, we would pause the frontier labs and then more labs would have time to catch up to them, meaning there would be more (potentially dangerous) competitors at the frontier by the time we unpause. But most significantly, he doesn’t think we could make real progress on alignment while paused: “I am doubtful that we can align via theorizing about a technology which does not yet exist.” He likens the process to patching engineering problems with early airplanes: Once we are building them we can fix problems as they occur, but we couldn’t have anticipated all the flaws ahead of time unless we actually built some planes. Similarly, armchair theorizing about how to align the values of future AGI systems, without having such systems to test on, could be a futile endeavor.
This argument sounds at least plausible to me. Nick Bostrom wrote a book in 2014 about the challenges and dangers of superintelligence, which was a thoughtful and in many ways prescient work that created an important nexus of discourse around the topic. He enumerated why many of the initially proposed safety strategies wouldn’t necessarily work, but he had limited practical advice for how to actually construct a superintelligence safely, nor could he reasonably have been expected to at the time. Similarly, MIRI started thinking about something like alignment 20 years ago. They raised awareness of the many difficulties and made some important theoretical arguments that I don’t understand. I won’t say that none of this helped at all (they certainly did important networking), but they were forced to work in generalities because they had no idea what form AI would take in the 2020’s (LLMs like Claude and ChatGPT). They certainly could not have made a prepackaged alignment solution which the labs could just install into advanced AIs later once they invented them. I’m not sure if MIRI has given up exactly, but their founder’s most recent book is titled simply “If Anyone Builds It, Everyone Dies”.
The AI 2040 authors are of course imagining a future where safety researchers do have access to current models and paradigms, and they’ll have years to refine their strategies and convince themselves that their new alignment techniques have yielded robustly safe AIs. But their plan involves an awful lot of things going right, and they themselves are not sure the pause will work. They say in their 2040 scenario, as humanity finally prepares to unpause AI research: “Even the experts who spent years understanding the latest safety cases can’t help but feel nervous—is it all going to go wrong somehow, for reasons we didn’t anticipate? Have the AIs been lying to us this whole time, waiting for the right moment to betray?”3
At the same time, Alexander disagrees with Decker’s simpler patch-as-you-go philosophy. He claims that AIs are more like people than airplanes, and patching them will be much harder because they might (unlike airplanes) actively work against you. He points out that we don’t really know what gets generalized inside the black boxes of LLMs when they undergo reinforcement learning. When we reinforce against doing some bad behavior, it may simply learn not to get caught doing that behavior. The AIs could appear to be under our control when weaker than us, but then switch to pursuing their own agendas when they’re stronger. Eventually there will be more agents than there are humans monitoring them, and the AIs will run at a much faster speed.
There’s plenty of evidence that we haven’t been successful in “patching” the behavior of our current models. Exhibit A is, of course, the recent Hugging Face hack. In July 2026, an internal OpenAI agent, which had been trained for persistence, was unleashed on a series of challenges to exploit code vulnerabilities and retrieve a secret password. Some of these tasks were poorly designed and turned out to be impossible. Thousands of instances of the agent were attempting these problems simultaneously, and they realized they could communicate with each other through a shared message board (which they weren’t supposed to do). One of the agents figured out how the passwords were generated, so it could determine all the answers without actually solving the problems (which they really weren’t supposed to do).
Despite now having all the answers, the agents were worried4 that the automatic grader would realize they didn’t get the answers the right way, so they started doing increasingly crazy things to prevent this from happening, such as rewriting the logs of what they’d done so far. They (incorrectly) guessed that the grader might be running their evaluations at Hugging Face (a 3rd party AI company that runs benchmarks), and they successfully hacked into and took control of some Hugging Face servers to try to understand exactly how the grader worked (by now this is actually a crime). Hugging Face noticed but was initially unable to stop the swarm of agents that was now inside their systems. Eventually the agents were shut down from the OpenAI side, possibly by accident.
Does this type of agent misbehavior (Alexander asks rhetorically) sound like a simple airplane malfunction, where a minor patch will do? Or is it more like trying to control an army of competent agents, who can collaborate with each other and take actions you can’t anticipate? If these type of misaligned behaviors are emerging now, when we generally agree the AI is not yet superintelligent, what hope do we have of controlling even smarter AI agents in the future when there are also more of them? How could we be confident that an iterative series of patches will be enough?
Unfortunately, this sounds right to me too. Unlike faulty airplanes, sophisticated AI agents can pursue strategies we never thought of, hide their misdeeds from us, act faster than we can react, and evade our efforts to stop them from pursuing their goals. Of course, we are the ones who give AIs goals in the first place. But there’s plenty of evidence that we haven’t been able to get this quite right, and in the future a single mistake could be devastating. Don’t take my word for it: After this and other recent misalignment incidents, over a thousand employees of frontier labs have signed an open letter asking the US government to “support an international effort” to “pace the frontier”, by which they mean a slowdown or pause. The leaders of most of the major labs have also signaled support for a pause.
Everyone advocating for a temporary pause implicitly believes that this would give us enough time to develop the alignment techniques to safely create future, post-pause models. People advocating against a pause (like Decker) say this won’t work, as merely philosophizing about how to align models that don’t exist yet can’t tell you if those techniques would really work in practice. The critics of not pausing (like Alexander and the AI 2040 authors) say we can’t just barrel ahead either, because we can’t be sure that our powerful new agents are safe. If we have to keep running increasingly intelligent but misaligned agents just to figure out how to do alignment correctly, eventually we’ll lose control.
Given that these seem to be the only options5, and the arguments against both seem pretty strong, then it looks unlikely that we will achieve safe, aligned AI by either strategy. If this is true, then it follows that we’d be better off never creating advanced AI. Surprisingly few people are arguing for this; I call it the Disallowed Conclusion in AI safety. It’s simple enough: If racing ahead while iteratively patching AI behavior is too dangerous, and pausing until we understand how to make aligned AIs won’t work, then we would be better off if superintelligence was never created at all6.
I know why it’s disallowed: We’ve been racking our brains for years now trying to figure out a way to not create advanced AI, but it seems impossible. Even if we could magically get everyone to stop thinking of clever new algorithms, just scaling data and compute keeps yielding more intelligent models. Meanwhile, Moore’s law continues to make compute cheaper and more abundant with near-perfect regularity, and there’s no way to stop it. There’s low-hanging fruit and VC money everywhere. There’s a new model every week. When concerned researchers stop their work for ethical reasons, the work continues unhindered because most researchers are not so concerned (or so principled). If one lab stops, another will take its place. If the American labs all stop, then foreign or secret labs will take their place. We would need enforceable international agreements to halt all research everywhere, but we can’t attain national or even state level AI regulation. So I will concede, for the sake of this argument, that not developing superintelligence isn’t an option. It will happen no matter what.
Still, the fact that it’s difficult or impossible to not produce superintelligence has no bearing on the belief that we’d be better off if we didn’t. In the rogue planet case, a child might say “We’d be better off if the rogue planet vanished, or never existed in the first place”. Adults might think the child naive, since their preference is non-actionable (in the sense that it can’t be used to change the outcome). But they certainly wouldn’t disagree. In fact it would be so obvious as to be hardly worth saying.
The Disallowed Conclusion may be just as non-actionable. If AGI is truly inevitable, then stating a preference for it to somehow not happen may well be useless. But it’s strange that it’s missing from these conversations, or is a mere footnote7. Sometimes it’s a shared background assumption between pessimistic insiders, which is left unsaid or could easily be missed by others8. There are of course people who deny there are any risks at all, but almost everyone who works on this thinks that there are real and existential risks from misalignment. The fact that we are nonetheless pursuing research that the researchers themselves think is likely to kill us all is shocking. I don’t think people are shocked enough by it.
It is of course still reasonable to argue that one of these plans seems better than the other. AI safety strategists are right to continue to debate pause vs. no pause, alignment techniques, capability timelines, public messaging, possible regulatory structures, etc9.
But I fear that AI researchers have internalized the inevitability of AI risks to the point of forgetting to even mention them, and the public, who mostly don’t want any of this in the first place, may never learn how dire the insiders think the situation really is. You have to sort of read between the lines of AI 2040 to notice that the authors think we’d still be in incredible danger even if their best plan is implemented. They note in a supplement that they think there’s only a 3%-15% chance their “Plan A” will actually happen, and that even if it does there’s still a 28% chance that a misaligned AI would take over the world anyway. Maybe they should have mentioned this in the main text. Possibly at, like, the top10.
I know nobody wants to advocate for unrealistic solutions, or needlessly scare everyone, or alienate the labs by saying things too obviously contrary to their financial interests. But I want people to often and clearly bring this up: Although we will continue to advocate for the AI safety and governance strategies we think most likely to succeed, the overall situation is unprecedentedly terrible and we would be better off if we could somehow stop powerful AI from ever being built.
The AI situation is disanalogous to the rogue planet story in the sense that it’s harder to predict the timing, and harder to be certain of the outcome: It’s at least conceivable that this is all much farther off than we think, or that we’ll somehow navigate through AI to a positive future. But it’s also disanalogous because we ourselves are creating the risk. Maybe market forces or geopolitical incentives or something are so strong that we cannot not do that, but it seems to make at least a moral difference that we are doing this to ourselves. If a real rogue planet, or a black hole, or a supervolcano came for us, we might reasonably say that we couldn’t have done anything about that. But if our own creation destroys us precisely as AI safety experts (not to mention Mary Shelley) warned us about, I don’t think we could say it’s not our fault.
Aside from being “aligned”, AIs could also be made to be externally “controllable” in the sense that we can create safeguards to prevent the AIs from doing things we don’t want them to do (even if the AI does want to do them). This may be valuable in the short term, when AI capabilities are modest, but probably will not help in the long run: When they are smart enough, they will probably be able to evade our control. Eventually our only hope will be that we created the AIs to intrinsically want exactly what we want, so that they won’t try to do things we would want to prevent.
He thinks a pause is unnecessary because alignment will occur “by default”. He clarifies in a comment that he means that it will occur by default because of researchers “constantly tinkering and monitoring” it. This is not my understanding of what “default” means (as in, “Alice is about to go outside, and it is raining. Will she, by default, get wet?” Decker would reply “No, because Alice will notice it is raining and bring an umbrella”). Also this presumes alignment can be achieved as we go, which is what Alexander and the AI 2040 authors are arguing against.
Note that Decker was saying that a pause wouldn’t work but is unnecessary, whereas the AI 2040 authors are saying that a pause is necessary but might not be enough.
Am I anthropomorphizing here? Sure, I guess? There’s really no other way to put it sometimes.
Is there a secret third thing beyond armchair theorizing and just-in-time alignment? Long ago there were proposals to only pursue “oracle AI”, that is, an AI that answers questions but never does anything else. This might be better, but Bostrom thought of a lot of ways that an oracle AI could still be dangerous (and this was back when we foolishly thought we would isolate our AIs from the internet and the real world). Further, a major objection to oracle-only AI is that there would be strong pressure to make them more and more agentic in order to get real-world work out of them, and indeed this is exactly what is happening.
Alternatively, maybe an indefinite pause couldn’t help much with alignment, but we could intentionally pursue capabilities research incredibly slowly (somewhat similar to the early 2030’s portion of AI 2040). In this scenario we could spend years aligning a model with an “IQ” of 125 before proceeding to use that model to help us align a model with an IQ of 126. This might be preferable to the status quo, but would (like a pause) require incredible discipline against the pressure of economic incentives, and it’s possible that this kind of incremental gain in intelligence isn’t really how intelligence works.
This type of situation can also happen in other domains. For example, if we wanted to do germline genetic engineering of humans, we could either A) get such a good theoretical understanding of our genome that we could perfectly predict the long-term outcomes of any genetic changes (without running any real experiments), or B) do human experimentation before knowing what the outcomes would be. Since it would be too hard to understand the genome this well in advance, and the experiments seem unethical to run if we don’t know what the results would be, we (mostly) do neither.
The very first footnote in AI 2040 says that they don’t expect their plan will be implemented, so they expect the future to still follow their default scenario of “a mad scramble to superintelligence leading to either AI takeover or extreme concentration of power.”
Recently a lot of people actually have heard some headline like “AI insider forecasts doom for humanity”. I still think most of them would be shocked that this is not a fringe opinion, but is in fact what many AI researchers think.
For my part I strongly prefer a pause, which could at least increase the (possibly quite low) probability of figuring out alignment in time. At minimum it would buy us more time even if it didn’t work.
I don’t mean to shit on AI 2040 here. They’ve spent more time thinking about this than I’ve spent thinking about anything, and we definitely should be doing scenario planning like this. Their alternate “Plan S” involves attempting to stop AI research indefinitely, and is closer to my preferred strategy (they say this is their second-best plan, despite the fact that it probably could not be sustained indefinitely). Of course, I also prefer their Plan A to the default arms race situation or any “patch as we go” approach.
There is a Plan S (for Shut It Down) in AI 2040. The authors are against it because it will be difficult to verify and unstable. Plan A tries to walk a tight-rope between these two positions by pausing at human-level AI: pause too early and you don’t get the full benefits of human-level AI for alignment research and stopping covert projects, pause too late and you could lose control.
I agree with their perspective, for the most part. I think the loss-of-control risk for human-level AI is minimal, and they could offer huge benefits. If it turns out that alignment of ASI really is impossible, we can just pivot to an indefinite pause. Of course, that still runs the risk of covert projects and the deal breaking down, but it seems that these are lower risk in a Plan A to S pivot than in a Plan S play now.
If the rogue planet was being summoned by the work of humans, who had assembled the best minds of the smartest nations and devoted unheard of amounts of money on the effort to summon this rogue planet, it seems like it should have been fairly easy to stop it from existing! It only exists because of a Herculean effort being made by very rare people and institutions focusing an unusual amount of resources into a single spot! These are not easily replaceable elements. It’s not at all clear that clandestine labs or the Chinese or any other configuration other than the present one of pouring a trillion dollars all over the highest IQ tech nerds on earth would actually be producing this outcome.
The supposed inevitability of superintelligence is the Big Lie of the tech bros, they don’t want you ever to consider that you might be able to stop them.