David Duvenaud

The Lighthaven Post-AGI Workshop talks are out! From galaxy-brained theories of convergent morality, to hard-headed discussions of political power, here's a 🧵 of the talks: post-agi.org/talks-berkeley

The Post-AGI Workshop: Economics, Culture and Governance | San Diego 2025 post-agi.org

@michael.muthukrishna.com on cultural evolution: human brains shrank by about a tennis ball after agriculture, once knowledge was split among specialists. We got collectively smarter but individually dependent: the first gradual disempowerment. www.youtube.com/watch?v=AOyd...

Cultural Evolution and Alignment by Michael Muthukrishna youtube.com

@habryka.bsky.social argued that Coherent Extrapolated Volition should be required for building ASI. He'd accept a 99% chance of an empty universe for a 1% shot at a human-guided CEV future, over the default trajectory. www.youtube.com/watch?v=2MUF...

Go CEV or Go Home by Oliver Habryka youtube.com

Owen Cotton-Barratt discussed "Leviathan risk", a third risk besides misaligned AI and human power grabs. With cheap intelligence on tap, corporations, states and ideologies may "wake up" and act as coherent agents, with less slack for the humans inside. www.youtube.com/watch?…

Leviathan Risk by Owen Cotton Barratt youtube.com

@raymonddouglas.bsky.social proposed "vertical alignment": collectives like states and firms can have their own goals, not just a weighted average of their members' preferences. This unifies AI takeover with emergent dynamics where no one seizes power. www.youtube.com/watch?v=…

Vertical Alignment by Raymond Douglas youtube.com

@hamandcheese.bsky.social says AGI will be an organizational shock since 20th-century regulation is brittle. We need "differentiable regulation" with loss functions instead of statutes, reward models instead of rulemaking, benchmarks instead of standards. www.youtube.com/watch…

Post AGI Statecraft by Samuel Hammond youtube.com

@critch.bsky.social proposed how to negotiate moral standards with alien minds. "Schelling Goodness" suggests that even without agreeing on what "good" means, agents trying to give the same answer as everyone else converge on some moral judgments: www.youtube.com/watch?v=YAh-...

Schelling Goodness by Andrew Critch youtube.com

@davidskrueger.bsky.social argued we should stop building advanced AI entirely, by dismantling the compute supply chain. Stopping production is simpler than regulation, more robust to cheating, and gives everyone years to react if someone defects. www.youtube.com/watch?v=IN0Y...

Stop AI: How and Why by David Krueger youtube.com

@antonleicht.bsky.social on the "Permanent Periphery": Countries without frontier AI get the risks without the benefits, and become irrelevant. Backstops against a domestic permanent underclass (votes, redistribution) barely exist btwn countries. www.youtube.com/watch?v=ezyE...

Permanent Periphery by Anton Leicht youtube.com

@erikbryn.bsky.social explained that markets beat central planning when knowledge is local and planners lack compute. AI changes this, making centralization of economic and political power more competitive: www.youtube.com/watch?v=G1YT...

AGI based Central Planning by Erik Brynjolfsson youtube.com

Fearing "loss of meaning" from AI job loss sounds trivial vs. fearing death. But I think desire for meaning evolved to prevent being ostracized, maybe fatally. Most sources of meaning (e.g. helping others, being skilled) are also reasons to support you. So it's a similar fear.