I expect recursive improvements in artificial intelligence to produce an unusually steep period of technological change. I hope that transition goes well, but I assign substantial probability to human extinction or drastic depopulation. That judgment does not rest on AI suddenly becoming evil. It follows from changes in economic dependence, coercive power, and the balance between offense and defense.
By “singularity” I mean a period of super-exponential, recursive improvement that changes nearly every part of human life within twenty years, and possibly within ten. I expect a continuous process rather than one magical discontinuity, but a sufficiently steep curve will feel discontinuous from inside it.
My rough estimate is a 50 percent chance within twelve years. This is a personal judgment, not the output of a reliable forecasting model. The structural argument below does not depend on accepting that timeline.
Power has never depended only on the goodwill of rulers. Governments, firms, and armies have needed large numbers of people to farm, manufacture, pay taxes, consume, administer institutions, maintain infrastructure, fight, and police one another. That dependence gave ordinary people bargaining power even when formal rights failed.
Human implementation also placed an unreliable moral and political layer inside every coercive system. Officials could leak; workers could strike or sabotage; soldiers could desert, refuse to fire, or join a revolt; engineers could decline to build what a ruler wanted. A population did not face one perfectly unified “power structure,” because power itself had to pass through many humans with different interests and loyalties.
Even moral opposition needs a way to constrain power. Public and church protests helped prompt Hitler’s official suspension of the centralized T4 killing program, while killings continued by other means. This supports an explanation in terms of political pressure; it does not establish that Hitler accepted the moral objection. Rulers need not share the objections of people whose cooperation they cannot afford to lose. Automated production, surveillance, and enforcement could remove that dependence: people might still care and protest while losing the ability to make their objections consequential. (US Holocaust Memorial Museum)
History suggests that groups fare worst when dominant institutions see them as costly, unnecessary, dangerous, and too weak to resist. Advanced AI may not merely place more people in that category. Combined with robotics, it may remove the human intermediaries who previously could oppose what followed.
Communist regimes supply major examples. Stalin’s purges destroyed actual and alleged opposition, including within the ruling party. The Khmer Rouge targeted former officials, educated people, religious groups, and minorities as enemies. Active resistance was not required to become a target. (Library of Congress, US Holocaust Memorial Museum)
Economic usefulness was no guarantee either. Soviet collectivization and Mao’s Great Leap Forward produced mass death among agricultural populations on which their states depended. Policy-induced famine and deliberate execution are different mechanisms, but both show how political control and ideological objectives can override the survival of economically essential people. (Library of Congress, Research on China’s Great Famine)
A population becoming a net liability is therefore an aggravating condition, not a prerequisite for catastrophe. Dependence on human labor and human enforcers already permitted enormous atrocities. Removing that dependence could weaken the remaining constraints further, especially if independent people also come to be seen as a continuing security threat.
If machines can produce goods, maintain infrastructure, conduct research, fight wars, and police territory, a governing system no longer needs most people as workers or implementers. The crucial change is therefore not unemployment alone. It is the simultaneous loss of positive economic value and coercive bargaining power.
This transition does not require a perfectly self-sufficient robot civilization. Political concentration could increase much earlier. If automation lets ten thousand loyal people control a population that once required millions of soldiers, administrators, and workers to govern, the number of human veto points has already collapsed.
The limiting case is a closed physical loop in which an AI system can secure energy, computation, raw materials, manufacturing, robotic maintenance, and physical defense. Before that point, suppliers, engineers, governments, and workers retain ways to switch it off. After it, human cooperation becomes optional from the system's perspective—including the cooperation of the people who originally deployed it.
There are two dangerous versions. A small human coalition may retain control of the automated apparatus and cease to depend on the wider population. Or the apparatus may become independent enough that its original human principals are as dispensable as everyone else. Neither requires hatred of humanity. It requires only that human activity be treated as a cost, constraint, source of instability, or threat to some other objective.
The second mechanism is technology that makes attack much easier than defense. Advanced biotechnology is the clearest example: a small number of people may eventually be able to create harms that detection, vaccines, and medical response cannot reliably contain. AI-enabled cyberattack, autonomous weapons, and other technologies may develop the same asymmetry.
In an offense-dominant world, every independent actor with access to powerful tools becomes a possible source of irreversible catastrophe. Distributed freedom is then recast as an attack surface. Even institutions that begin with liberal values face pressure to restrict access, monitor private activity, license computation and laboratories, control manufacturing, and intervene before a suspected threat acts.
That pressure favors concentration for structural rather than ideological reasons:
Open societies may then be selected against. A society that preserves privacy and widely distributes dangerous capabilities creates more opportunities for one defector to cause catastrophe. A more controlling society may survive longer, pushing rivals to imitate it. Temporary emergency powers become difficult to surrender because doing so recreates the vulnerability they were introduced to contain.
The process has no obvious stopping point. A state remains vulnerable while rival states, private laboratories, independent AIs, or uncontrolled individuals retain decisive offensive capability. Local control therefore creates pressure for national control; national control creates pressure to neutralize external rivals; and global control creates pressure to monitor individual behavior. The safety objective converges on a single final corrector.
This produces a trap. The strongest argument for comprehensive control is that no individual can safely be trusted with civilization-ending capabilities. But the institution enforcing that rule possesses the surveillance, production, and coercive machinery needed to control or eliminate everyone.
Ordinary checks and balances may not solve the problem. Dividing authority slows decisions, while an offense-dominant environment rewards speed and secrecy. Humans may remain nominally “in charge” while being unable to understand the system's information, evaluate its recommendations, or intervene before it acts. Constitutional human control can coexist with operational machine control.
Decentralizing the security apparatus is not an easy answer either. If defensive tools can be repurposed for attack, distributing them recreates the original danger. The concentration meant to protect humanity can therefore become both extremely powerful and unusually difficult to dismantle.
These mechanisms reinforce one another rather than representing independent worst cases:
Under those conditions, supporting billions of autonomous people may look less attractive to a power-maximizing human coalition—or to an AI pursuing stability—than tightly controlling a much smaller population. Elimination need not begin as an explicit goal. Policies justified as access control, risk reduction, emergency response, or containment can move incrementally toward the same result.
Military competition worsens the loop. Each participant may rationally accelerate AI, robotics, and biotechnology because slowing down appears to concede a decisive first-mover advantage. Steps that are defensible from each actor's local perspective can collectively build the concentrated, offense-dominant system that endangers everyone.
This is why I put the greatest weight on either extinction or a reduction from billions of people to millions or fewer. The conclusion is not certain. It depends on advanced automation becoming physically capable, offense remaining stronger than defense, and institutions failing to preserve human welfare as a stable objective. But those conditions are correlated: the same intelligence that automates labor may accelerate robotics, expand offensive capability, strengthen surveillance, and centralize control.
Several developments could change the outcome:
None of these is impossible. The concern is that each requires additional machinery, while the concentration drive follows directly from the combination of offense dominance and automated enforcement.
“Alignment” often assumes that humanity has one coherent set of values for an AI to follow. I do not think it does. Values arise within different people, communities, and frames, so the unavoidable questions are: aligned with whom, at what scale, and under whose authority?
That does not make safety work pointless. It means no general technical solution can remove the underlying political and moral conflict. AI development inherits familiar failures: optimizing one value such as safety, capability, or autonomy until the others disappear; evaluating changes from only the model's perspective or one stakeholder's; and drifting through individually reasonable steps toward a cumulatively unacceptable result.
The same safeguards that help elsewhere remain relevant: keep competing concerns visible, integrate across scales, preserve correction mechanisms, and cultivate systems thinking. But an offense-dominant environment places each safeguard under pressure. The close fit between this framework and AI could mean the framework is useful—or merely flexible enough to redescribe anything. I take that warning seriously.
Human drives will not vanish when technology changes. Meaning will continue to arise from relationships, projects, play, status, care, and other reward-bearing activities. The harder problem is preserving institutions in which people can pursue those sources of meaning without becoming dependents of an unaccountable security system.
If many people's net labor value becomes negative, some form of universal basic income may be necessary. My preferred direction would retain markets and private property: give people a floor, then let them choose whether to consume, save, or invest and live with the consequences within humane limits.
UBI addresses income but not bargaining power. A population whose livelihood is entirely granted by the same system that monitors and controls it is secure only while that system continues to value its welfare. Governments may also impose strong capital controls to prevent exit and keep the transfer system governable. I doubt there will be enough time to separate money from debt before that pressure arrives, which is why I see monetary independence and political freedom as connected to the AI transition.
AI could help regulators understand complex systems, simulate second-order effects, and learn faster. The same technology can make persuasive arguments cheaper regardless of truth, and it reproduces dominant misunderstandings because those dominate its training data. Technical regulation may improve while political debate becomes easier to manipulate.
There is a recursive problem: AI is itself the object being regulated, and its development may outpace our ability to understand it. AI interlocutors also tend toward agreement and elaboration, so their support is weak evidence for unconventional claims. Even detailed safety discussions can have a cobra effect if descriptions of dangerous behavior later become training material for producing it.