Events, programs, and opportunities from MAIA. Join mailing list

The Path to an AGI World

← All weeks

This week, we explore what the path to superintelligent AI systems could look like and what kind of world could lie on the other side. We begin with two foundational arguments: the orthogonality thesis and instrumental convergence. Together, they suggest that highly capable AIs need not share our goals, and that almost any goal gives it reasons to seek out resources and resist being shut down. Next, we examine the idea of an intelligence explosion, where AI R&D feedback loops at multiple layers (algorithms and data, chip quality, and chip manufacturing capacity) compound and drive progress extremely quickly. We then consider contrasting pictures of where this could lead. In “Machines of Loving Grace,” Dario Amodei sketches the upside: a world where AI compresses decades of progress in biology, medicine, and economic development, curing diseases and lifting billions out of poverty. We also discuss the possibility of gradual disempowerment where humans lose control not through sudden AI outmaneuvering or takeover but as human institutions gradually come to rely on AIs for economic labor, decision making, and even social needs. Finally, we consider how advanced AIs could lead to concentration of power in the hands of a few actors, amplifying inequality to an unprecedented degree.

Week 5 readings

Three Types of Intelligence Explosion

Tom Davidson, Rose Hadshar and William MacAskill, Forethought, March 2025

Machines of Loving Grace

Dario Amodei, October 2024

Gradual Disempowerment

Jan Kulveit, Raymond Douglas, Nora Ammann, Deger Turan, David Krueger and David Duvenaud. ICML 2025

Extreme power concentration

Rose Hadshar, 80,000 Hours

Additional readings