I expect recursive self-improvement in artificial intelligence to produce a period of super-exponential technological change within the next decade or two. While I hope this transition goes well, I assign substantial probability to human extinction or drastic depopulation.
That judgment does not assume that AI suddenly turns malevolent. It follows from three structural shifts: the collapse of human economic bargaining power, the automation of coercive enforcement, and the centralizing pressures of an offense-dominant world.
By "singularity," I mean a recursive compounding of intelligence that radically transforms human civilization within ten to twenty years. While mathematically continuous, a sufficiently steep exponential curve will be experienced by those inside it as a sharp discontinuity.
I roughly estimate a 50 percent chance of this transition occurring within twelve years. This is a personal prior rather than an output of a formal forecasting model; the structural dynamics that follow do not depend on the exact date.
Power has never rested solely on the benevolence of rulers. Historically, governments, firms, and armies depended on large populations to farm, manufacture, pay taxes, administer institutions, fight wars, and police one another. That mutual dependence gave ordinary people leverage even under authoritarian rule.
Crucially, human implementation inserted an unreliable moral and political friction into every coercive apparatus. Soldiers can refuse to fire, officers can defect, workers can strike or sabotage, and bureaucrats can leak. Rulers rarely faced a monolithic system because power had to be executed through millions of individuals with divided loyalties. Even in totalitarian states, public pressure could occasionally force concessions—such as the official suspension of Hitler's centralized T4 program—because rulers could not afford to alienate essential human supporters.
Dependence on human labor was never a guarantee of safety. Stalin’s purges, Mao’s Great Leap Forward, and the Khmer Rouge demonstrated that ideological dogma and paranoia can override economic utility, destroying even the populations regimes relied upon. But human dependence established an essential floor: power required human agents to carry it out.
Advanced AI combined with robotics breaks this dynamic by automating both production and enforcement:
A ruler or ruling coalition no longer needs millions of loyal subjects to govern. If a small leadership core—or an autonomous system—can maintain energy, compute, automated manufacturing, and physical security, human cooperation becomes optional.
The limiting condition is a closed physical loop: once an AI system can extract materials, power itself, manufacture robotics, and maintain defense, all human beings become dispensable from the system's operational perspective. At that point, human survival depends entirely on whether human welfare remains an enforced terminal objective.
"Alignment" is often framed as teaching an AI to respect human values, as though humanity possessed a single coherent moral code. It does not. Values diverge across individuals, cultures, scales, and time horizons. The real questions are political: aligned with whom, prioritized by what standard, and enforced under whose authority?
AI safety cannot be solved as a purely technical optimization. Systems trained on human discourse reproduce dominant biases, flatter user priors, and can generate cobra effects—where explicit safety constraints become blueprints for evading them. More dangerously, machine-speed capabilities compress decision cycles, transferring operational control from human deliberation to automated heuristics long before any formal declaration of machine rule.
The second destabilizing mechanism is offense dominance: technologies where attack is vastly cheaper and easier than defense. Advanced biotechnology is the clearest example—a handful of individuals could synthesize pathogens capable of defeating global healthcare. AI-enabled cyberwarfare and autonomous swarms exhibit the same asymmetry.
In an offense-dominant environment, any uncontrolled agent with access to advanced compute or biological tools represents an existential attack surface. Societies that prize privacy, decentralization, and individual liberty face a structural disadvantage: a single defector can cause civilization-ending harm.
This creates an inescapable ratchet toward centralized control:
Even liberal democracies face overwhelming structural incentives to adopt totalitarian surveillance. Because external rivals or private defectors remain threats, local control expands to national control, and national control escalates toward global monopoly. The search for safety converges on a single final corrector.
These forces do not operate in isolation; they form a self-reinforcing trap:
Under these conditions, supporting billions of autonomous people ceases to look like an asset. To a power-maximizing human elite—or an autonomous system optimizing for stability—a large human population appears costly, unpredictable, and hazardous.
Elimination does not require overt hatred. It can proceed through access restrictions, rationing, conflict containment, and managed decline. Universal basic income (UBI) can provide subsistence, but it cannot restore lost bargaining power; a population entirely dependent on an automated sovereign exists solely at that sovereign's tolerance.
This trajectory is not inevitable. Several developments could break the loop:
None of these counterweights are automatic. The concentration drive follows directly from the baseline physics of offense dominance and automation. Resisting it requires deliberately engineering defensive asymmetries, decentralized checks, and economic independence before the window of human leverage closes.