Reading List (blogroll)

A Is For America

It’s increasingly clear that nobody has a plan for if this AI thing turns out to be real. Some people have suggestions, but they’re all things like “regulate a little more” or “regulate a little less” or “react to things as they come up”. This won’t be enough. Not just because things may move too quickly - although they will - but because in order to regulate or react, you need to know what you’re aiming for, and it’s increasingly clear that people can’t even visualize what AI going well could look like. What would it take to honestly tell our children that we rose to the occasion, to make the AI transition go down alongside the American Revolution and D-Day as one of our country’s finest hours? If your brain sputters and throws an error message at the question, isn’t that a problem?

It’s a total coincidence that Plan A comes out the week after America’s 250th birthday. It was supposed to come out earlier, but got delayed. Then it was supposed to come out later, but got pushed forward.

Still, the saying goes “A wizard is never late, nor is he early; he arrives exactly when he means to.” And if anyone qualifies as wizards, it’s Daniel Kokotajlo and his team of forecasters at the AI Futures Project. I previously wrote about Daniel’s eerie accuracy over the 2021 - 2025 period. Since then, they’ve gained worldwide fame for their AI 2027 scenario, which predicted the rise and quick takeover of coding agents in early 2026, plus something like the fight over Fable1.

Plan A isn’t another prediction. It’s a wish list, a positive vision, a road map for navigating the future. It describes the best course of action that Daniel and the AI Futures Project can come up with, and what would happen if we took it.

“Really? You got America a policy paper for its 250th birthday? Doesn’t America already have enough policy papers?” Sort of, but it’s not exactly a policy paper. It starts in a timeline similar to that of AI 2027, on track for a poorly-controlled intelligence explosion that either ends the world or dooms it to permanent techno-oligarchy. But this time, America is blessed with some extra foresight and determination, and makes only good choices (all non-Americans behave naturally, including trying to thwart America when incentivized to do so). It gives a year-by-year description of this best-of-all-possible-worlds, from now through 2040, as predicted by the best AI forecasters alive, with over a dozen supplements explaining all the implementation details.

This is a crazy thing to try releasing. Daniel gave me several justifications for doing it anyway, but the one I remember most is that it’s supposed to be a floor. When some politician proposes a data center ban, or says that we have to gut safety regulation to compete with China, or promises a job retraining program, think to yourself: does this person have a vision for where all of this ends up? If so, is it as good as Plan A? If not, consider demanding that they do better.

I did a lot of writing for AI 2027 and was listed as a co-author. Some of my writing made it into Plan A too, but it was a bit less. The difference is of degree rather than kind, but because of this - and to give me more latitude to discuss it the way I like with less PR blowback - we decided not to put me as a co-author this time. I continue to be proud of having a part in this, small as it may be.

(related: everything in this post is my opinion only, and not officially endorsed by the AI Futures Project)

A Is For Agreement

The linchpin of Plan A is a joint AI regulatory regime with China.

In the late 2020s, as the world shoots toward an intelligence explosion, the US government realizes that it has no control over the situation as long as race dynamics continue to hold. Multiple “Mythos moments” leave them convinced that the situation is spiraling out of control, but analysts continue to insist that if we unilaterally slow or regulate AI, China will continue its own research and gain a dangerous strategic advantage over us.

So the hypothetical wise statesman president proposes a joint regulatory regime to China, and China agrees. This is one of the sections AIFP spent the most time thinking about and trying to justify - the conceit was that America is wise and foresightful by narrative fiat, but our rivals still have to behave in believable ways. Still, they think China’s agreement is plausible. They have concerns similar to ours (things are moving too fast, society is being disrupted, they can’t rule out existential risk), plus the additional concern that they’re currently losing the race to America and so an enforced tie would be in their favor.

The US and China don’t trust each other, so any agreement would have to be trustless, ie impossible to cheat, win-win even if you expect the other side is trying as hard as it can to defect against you. This is another area that AIFP expected to be a major sticking point for most people, and that they put in lots of work to justify. Their plan is:

1. Establish joint control over the supply of new chips

2. Establish common knowledge of the location of all existing chips.

3. Ensure that all new and existing chips go to mutually-transparent, mutually-audited secure data centers.

Establishing control over the chip supply is easy. Only a few companies can design AI chips (eg NVIDIA in the US, Huawei in China), and only a few factories in the world can produce them (eg TSMC in Taiwan, Intel in the US, SMIC in China). All of these companies and factories are in the US, China, or client states kept on very short leashes. The US and China simply tell these companies and factories to send chips only to licensed customers, and send auditors and inspectors to ensure compliance. Something like this is already in place; this deal merely tightens the restrictions and lets both countries participate in the auditing process.

Hunting down all existing chips is only slightly harder. Most of these chips are in giant data centers the size of small cities using entire power plants’ worth of electricity - so hard to hide. The rest can be traced to their final locations using customer records from NVIDIA, TSMC, etc. AIFP explain their methods in more detail in the Covert Project Supplement, but estimate that they can track down 98.5% of existing chips (we’ll come back to the remaining 1.5% later).

Once the two countries are confident they’ve established control over almost all chips, they relocate them to licensed data centers (“whitesites”), which the other side is allowed to send auditors to inspect. The Verification Supplement describes very-near-future technology (would take 1-2 years to have ready; I recently recommended grants to organizations working on creating it) that can monitor these data centers and provide mutual transparency into what they’re doing. The auditors can ensure the technology is installed and uncompromised, and then both sides will know if the other is defecting against the deal. Other auditors in the chip factories ensure that any new chips produced are being sent to the whitesites too.

The end result is that the US can be confident that at least 98.5% of China’s chips are in white sites where the American government can audit their activities, and so China can’t defect on the deal by training dangerous AI without America knowing. The same is true on the other side; China knows where almost all of America’s chips are, and can be confident we aren’t defecting.

A Is For Aristotelian

Now that the US and China know what’s going on in each other’s data centers, and agree in principle to coordinate their AI research, what do they do?

In an earlier draft of Plan A, I called this the “Golden Path”, before the more dignified team members took out the reference. The idea is that we need a sort of golden mean in the speed of AI progress. Too fast, and we get misaligned AI or some kind of social upheaval (you’ve already this case a thousand times; read If Anyone Builds It, Everyone Dies for the details). But why worry about going too slow?

In Plan A, the biggest risk of going too slow is that the deal falls apart. The clearest precedent here is arms control regimes; START + New START held on for a good few decades, but were suspended over tensions around Ukraine in 2023, then expired fully in 2026. The JCPOA nuclear treaty with Iran barely made it two years. If we dilly-dally and do nothing for fifty years, probably the agreement falls apart and we’re right back where we started. So we should have some plan to do what we want to do with the agreement within a decade or two.

The second-biggest risk comes from that 1.5% of compute we discussed earlier. Again, we’re assuming that we can’t trust China (and vice versa) - so they’ve done their maximum possible defection, hidden 1.5% of their compute, and are using it to train some maximally dangerous military AI. How long before that AI could undergo an intelligence explosion and give them a decisive strategic advantage? AIFP calculates that this would take somewhere north of a decade - so again, we should have some plan to get what we want out of the deal before that period is up.

The third-biggest risk comes from the steady march of technology. Chips and algorithms get better every year; AIs that took city-sized data centers to train today may take only academic-grade hardware tomorrow. Once a dangerous AI can be trained on academic-grade hardware, trustless deals become impossible, because anyone could be hiding a few scattered supercomputers. There are some cheap things we can do to slow this process down, but stopping it entirely could require authoritarian-seeming interventions or a generalized slowdown of economic progress. Rather than go that route, we should have some plan to get what we want out of the deal before these considerations become pressing - which, once again, means before a few decades are up.

And although it didn’t make it into the main text, there’s an additional consideration around balancing the risk of AI vs. the risk of all the things AI could save us from. Nuclear war, bioengineered pandemics, mirror life, collapsing fertility, decreasingly functional politics, “the polycrisis” - I don’t worry much about this stuff compared to dangerous AI in five years, but the longer we delay AI to work on alignment, the more the risk from AI goes down, and the risk from all these other things goes up, until eventually they meet in the middle and we delayed too long. How long should we delay? I think this one alone suggests slightly longer timelines than the others, but it’s still decades and not centuries.

So in Plan A, the US and China agree to go as fast as possible without compromising on safety.

Starting in the early 2030s, they make a push to train more AI, quicker. These AIs will still be trained by existing companies (Anthropic, Alibaba, DeepSeek, OpenAI, etc) and can still be used for business purposes, but they’ll be trained and hosted in regulated monitored data centers, there will be a moving capabilities ceiling, both countries will raise the ceiling at the same rate at the same time, and the resulting AIs will be available to everyone in the world.

(at some point the US and China will loop all the other countries into this regulatory regime as some kind of sort-of-but-not-really-voting observers; they agree to follow the rules in exchange for shared benefits, including data centers on their territory, access to the AIs, and a share of future AI-generated wealth. This isn’t strictly necessary, because no other country really has the ability to do much with AI, but it’s a nice gesture for our utopian scenario)

Then, in the mid-2030s, they pause at AIs around the level of top human geniuses. These AIs are close enough to current AIs that the same alignment techniques that mostly-sort-of-work for ours might mostly-sort-of-work for them too. But if they don’t, it’s fine - the safety case rests primarily on “control”, ie keeping the AIs in a box they can’t get out of (in this case, highly regulated data centers that have been secured from the inside). This wouldn’t work for superintelligence, but with enough care, it will work for these “merely” top-human-level models. The plan is to spend the next ~10 years using this “country of geniuses in a data center” to solve AI alignment, along with approximately all other problems.

A Is For Abundance

The middle of Plan A is AIFP’s pleasant fantasy about all the problems they solve easily by deploying millions to billions of top-human-genius level AIs.

I previously said the conceit of this exercise is that America only makes good decisions. Even so, you could be forgiven for some skepticism here; even if the genius-level AIs give us the technological capacity to solve our problems, what provides the political will? Their excuse is the AI superforecasters and things like them. The AI For Epistemics supplement provides the details, but they dream of a world where today’s forecasting tools blossom into a wide ecosystem of advisors that voters, politicians, and the media use to build models of the future and guide their decisions. They hope that trustworthy AIs will be able to assert convincingly that the policies they recommend will go well, and that the alternative is techno-feudalism, mass immiseration, or (in some cases) doomsday. This breathes new life into our usually sclerotic political system and allows it to dream big.

The top-human-genius AIs soon become capable of taking most white-collar jobs. The electorate solves this with a “citizen’s dividend” (I was warned against using the term “UBI”) which is very easy to afford, since the AIs are causing double-digit and even triple-digit yearly GDP growth (think that’s crazy? Read the Economics Supplement). Because of extreme deflation in most other goods, the dividend is pegged to compute; as compute production rises, it goes from $25,000 per person per year at its inception in 2033 to $1.6 million in 2035 (all dollar amounts are extreme-deflation-adjusted; for more, read the supplement). Various diseases get cured. Various social ills get ended. Probably there are flying cars or some similarly cool inventions.

All this growth brings new risks of its own. If uncontrolled, it could spark another arms race - eg if the US grows at 150% per year, but China only at 100% per year, then after ten years America is 9x bigger and could easily crush its rival. If unmonitored, someone could use the cancer-cure technology to create something extremely bad, from bioweapons to mirror life to things I won’t mention because you’ll accuse them of being implausible. This gets solved basically the same way as the data centers - the US, China, and the various sort-of-but-not-really-voting observer nations agree to restrict AI-assisted economic growth to special economic zones which are heavily monitored by all parties. This has the bonus of ensuring that the 200-story-tall nano-assembly plant won’t be in your personal backyard.

For more pleasant fantasies, read the 2033 - 2037 sections.

A Is For Alignment

Meanwhile, billions of other top-human-genius-level AIs work on questions around AI alignment.

This suffers from a sort of bootstraps problem; how do we know that these AIs will be safe?

First, they might be safe simply because they haven’t gotten to the regime where they’re unsafe. Today, AIs are slightly misaligned but still easy enough to get useful work out of. As AIs progress from here toward superintelligence, they may get gradually less aligned for several reasons: they’ll be more agentic, they’ll be better at scheming coherently, they’ll have more “out-of-distribution” options, and they’ll be smart enough to generalize in weird ways2. Top-human-genius AIs will be some part of the way along that distribution, but maybe not so far as to behave 100% adversarially.

But Plan A doesn’t rely on this; its safety case hinges on control. AIs are kept in heavily-guarded data centers that they can’t escape even if they wanted to (the bandwidth is low enough that exfiltrating their weights over the network would take decades, and people entering or leaving are searched for USB drives, etc). All of their actions are watched by AIs from other companies and countries; anything suspicious is escalated to human monitors. Smart humans and faux-misaligned AIs constantly red-team the security. Since safety incentives now outweigh race incentives, there is no reason to rush towards unreadable chains-of-thought, and so chains of thought are monitored carefully and successfully. Some genuinely misaligned AIs are created as “model organisms” in extremely controlled conditions and observed to see what they do; if they are capable of hacking or social engineering at a certain level, everyone ensures that this level is insufficient to escape the data centers. If any AIs do escape or even make progress towards escaping, we prepare to trade with them rather than treat them as fully adversarial. None of these mechanisms would work to control a true superintelligence, but that’s why we aren’t building one of those yet.

Since these AIs may not be fully-aligned, a key part of early research is using whatever useful work we can get out of them to bring them to a point where we can trust them further, eg where they’ll do good alignment research and not try to fake it (“You’re absolutely right, I did propose a scheme that would result in the destruction of humanity - and that’s my bad”). Once we’re at that point, we can invest tens of billions of researcher-years into the alignment problem and come out with something actually trustworthy; AIFP speculates about what that might look like in the Alignment Supplement and the Insider Perspective. In a best-case scenario, we replace the current generation of weird machine-learning-kludge-based AIs with some sort of more mathematical AI that it’s possible to prove things about, and prove it to be aligned or at least corrigible.

Along with this technical research, we’re also doing philosophical . . . something between “research” and “debate”, deciding what values we want the aligned AIs to have, or how much we want the AIs to follow government orders vs. follow the general will of humanity vs. search for moral truths on their own. The end of this stage probably looks like every country with AIs either implanting its own values or leaving it to individual companies/people to do this, and the AIs doing incomprehensible economic bargaining among themselves to work out any conflicts.

Once we’re very certain that AIs are fully aligned - AIFP speculates this could be around 2040 - we hand over the sovereignty layer of government to them. Why? All of the problems we talked about in the “risks of going too slow” section - deal breakdown, defection, nuclear war, etc - are problems with misaligned humans doing bad things. Once we have fully-aligned AIs, we want to give them final say over those levers so the risks don’t happen. Because the AIs are fully-aligned, they continue to let humans run the policy layer of government. The idea is that the US President or the paramount leader of China still controls day-to-day decisions, but if someone tries to pull off a coup and launch the nukes for no reason, then an aligned AI controls the nuclear missiles and it says no.

Even after we’ve handed over the nukes, the AIs still don’t race to superintelligence - having an alignment plan that will provably work for superintelligence is a higher bar than having one that provably works for the sort of top-human-genius AIs we trust to lead us through the 2030s. In our scenario, the AIs start an intelligence explosion pretty soon after the handover in 2040. But this is just for narrative flow; at this point, the risks of going too slow are greatly diminished, and if the AIs say it will take until 2100 to be really sure they’ve solved everything, we should give them until 2100 (we’ve probably conquered aging by this point anyway).

After we have superintelligence in 2040, the superintelligences solve whichever philosophical problems are solvable, predict whichever aspects of the future are predictable, and then, in consultation with human governments, help chart the best possible future for humanity among the stars. AIFP took this part out of the main text, because they worried that fancy Washington policymakers reading our scenario would get weirded out by descriptions of what type of auction you use to determine who gets how many galaxies, but it’s in the Epilogue and the Space Governance Supplement and I’ll try to blog about it more later this week.

A Is For All Of Us

Why care about this weird sci-fi story?

For the past year or two, I’ve kind of been in despair. If AI 2027 is right, then we could get an uncontrolled intelligence explosion sometime in the late 2020s or early 2030s. Over the course of a year or two, we could go from a basically normal world where we’re mostly talking about shoplifting and health care costs and Trump, to a world completely under the control of some sort of incomprehensible superintelligence. Even a nimble and competent government would have trouble responding on such a timescale (let alone our government) and the likely outcome would be as AI 2027 portrays it: oligarchy or extinction.

So it seems like we should try to prevent AI progress somehow. But preventing technological progress - “degrowth”, as the kids are calling it these days - is historically one of the worst possible bets. Done in small doses, it merely leads to preventable poverty, misery, and mass death. Done too much, it breaks the engine of social advance entirely, trapping whole civilizations in a morass of pessimism, paranoia, and zero-sum thinking which is near-impossible to escape after they’ve fallen in.

But surely the heuristic that technology is usually good can’t be extended into a command to never worry about any technology, no matter how apparently catastrophic? On the other hand, everyone who stands in the way of technology thinks their specific case is justified; the maxim “don’t do it unless it’s justified” does zero work. These are the sorts of considerations that have been haunting me. As Woody Allen put it, “Mankind is facing a crossroad. One road leads to despair and hopelessness, and the other to total extinction. Let us pray that we have the wisdom to choose correctly.”

Plan A feels like our best hope to do something actually good. The key insight is that if powerful AI is really as close and transformative as we think, then there’s a massive surplus that can satisfy everyone. Tyler Cowen and Marc Andreessen and the accelerationists will scream about how we’re slowing down, but we imagine our slow world as going much faster than they imagine their fast one. If Tyler wants double-digit GDP growth in the 2030s, we’ll give him triple-digit. If Marc Andreessen wants a cancer cure by 2050, we’ll give it to him by 2035. The choice isn’t business-as-usual versus a world with some gee-whiz AI innovations. It’s nanobots eating the solar system in 2033 vs. cancer cures in 2035. When we say “we must slow down AI”, we mean we want the cancer-cures-in-2035 one. If we’re merely on track for a few cool gee-whiz AI innovations in the 2040s, then I’m wrong about everything and none of this really matters one way or the other.

Milton Friedman said that true political change only happens during crises, and that the future belongs to whoever has a plausible plan ready when the crisis happens. AIFP releases Plan A in this spirit. If you’re against it, you should think long and hard about what alternative course of action you expect the government to take once the crisis becomes evident, and whether it will go better for your interests than Plan A will. If you think that there will never be a crisis, that the public will never challenge AI, that and you can just keep reacting to things as they arise - then good luck with that. I really think we’re the good cop here.

Except that I shouldn’t say “we”. It’s wholly a coincidence that Plan A solves this problem. Nobody else on the team seemed that interested in it; they were just trying to calculate what actually has the best chance of keeping us alive. In fact, they asked me to stress that this is all provisional: if we get to 2030-something and the calculus of the Golden Path demands we move slower than this, they’ll recommend going slower; if it demands we move faster, they’ll recommend that too.

But in fact, their current calculations do suggest a Golden Path which ends poverty and disease within a decade, and gives us a glorious interplanetary future within our lifetimes. This wasn’t a requirement of their model. It’s just how things coincidentally seem to have worked out. The optimal regime for alleviating doomers’ concerns happens to be one which should satisfy accelerationists, and a trajectory well within accelerationists’ Overton Window happens to have the best possible properties for safety research.

Like in previous advances in AI, I can only attribute it, as all else, to divine benevolence.

You can read Plan A here. Don’t miss the supplements hidden in the top right corner.