Reading List (blogroll)

[Original here. Like a Highlights From The Comments post, but including discussion from around the Internet.]

Roon is an OpenAI researcher whose popular Twitter presence has catapulted him to the status of extremely-unofficial company representative. He seems spooked:

…leading to a conversation with Geoffrey Irving, formerly of the UK’s AI Security Institute:

I side with Geoffrey in hating the word “magic”. I remember when arguments about AI were things like “Sure, if you could magically get billions of dollars of compute, and magically scale AI up a thousand times, and magically get rid of hallucinations…” and the whole implausibility hinged on the word “magic”! Then companies, using normal non-magical methods, got billions of dollars of compute, and scaled up the AI a thousand times, and significantly decreased hallucinations, and that was always allowed! As Irving points out, there are plenty of good specific plans for how a global capabilities slowdown could be arranged.

But I side more with Roon in his skepticism of a unilateral slowdown. I think removing any given company from the picture would slow things down less than 10%. Even that number would require an implausible scenario where the lab managed to lock down its compute and talent despite committing not to use them. More likely, the talent would migrate somewhere else, the compute money would dry up and the cloud providers would switch to other customers, and things would actually speed up because the leading project would have more total talent/compute. Meanwhile, the most safety-conscious lab would be out of the race, they’d lose their ability to lobby for more practical solutions, and as their market cap went to zero they’d turn into a terrible warning about the danger of ever listening to doomers.

What can individual labs do?

• State explicitly that they support a coordinated slowdown. Preferably the government would lead this - but, if not, they would do it voluntarily if all of their competitors also agreed. Be willing to state this even at some financial cost, eg risking an antitrust lawsuit.

• Use their prestige and influence to lobby the government for a coordinated slowdown.

• Use their reach to persuade the public to support a coordinated slowdown. I can’t remember who first suggested having every Claude response end with “ceterum censeo there should be a global AI slowdown”, but even something less dramatic would probably move the needle.

• Have an internal team working out details of a coordinated slowdown; once they feel more confident in the requirements, start incubating helpful technology (eg trustless verification).

• Have “merge-and-assist” clauses and take them seriously.

I’d rather have a big company pursuing all of these things than stopping unilaterally and removing themselves from relevance.

I took the first point above from Michael Trazzi’s playbook. He led the latest round of AI protests, which marched on major AI companies’ headquarters demanding that their CEOs formally make the “will pause if everyone else does” statement. These marches have felt slightly surreal, because often the CEOs or the company have informally made something sort of like the statement, but never followed it up. So the protest has been somewhere in between the usual “shame on you greedy plutocrats, we will destroy you” and “we know you’re secretly on our side, please have the courage of your convictions”.

Trazzi gives no evidence for his “personally seen DMs” statement and I don’t know how he could have gotten these, but my guess is that it’s true and that the two CEOs he’s talking about are Demis Hassabis of DeepMind and Dario Amodei of Anthropic, both of whom have sort of kind of hinted that they might be in favor of something like this. 25% chance I’m wrong about Dario and it’s actually Sam Altman.

One more Roon tweet:

Back when we had this discussion in 2023 - it already seems like some bygone Bronze Age - some people argued that we should hold off on a pause until near the end, when AI was good enough to help with alignment research. Then even a short pause - even six months - would buy a lot of breathing room. Like Roon, I think that opinion aged well, and that the time we were imagining is right now. It’s true that if we wait a year we’ll have even better AI, but this consideration always argues for waiting a year, and at some point it will be too late. I don’t think we’re at that point yet, but I think it will take a long time to coordinate a slowdown, and that if we wait to start coordinating until we’re at that point, then yes, it will be too late.

Pacing The Frontier

I wrote that last part yesterday and it’s already obsolete. “1,000+ employees of frontier labs” have now signed an open letter called “Pacing The Frontier” saying that:

We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.

Signatories include the chief scientists of OpenAI, Anthropic, Meta, and Thinking Machines; the inventor of Claude Code, 5/7 Anthropic co-founders, 1/3 DeepMind co-founders, the head of AI Resilience at the OpenAI Foundation, and more.

When I first saw it today, it didn’t include Dario Amodei’s signature; now it does. Either Dario held off on signing until it went up so as to not be seen to pressure his employees, or the people circulating the letter deliberately kept him out of the loop until it was published, just in case. In either case, good job Dario!

The letter still doesn’t include Sam Altman’s signature, but OpenAI’s corporate Twitter account retweeted with what looks like official company endorsement:

…and in a very recent interview, Sam Altman said that “we may have to pace the rate of AI development”, which is similar enough language to the open letter (“Pacing The Frontier”) that it could be a covert allusion. I imagine there’s some sort of politics going on here - maybe an attempt to simultaneously please safetyist factions within his company and the Trump administration, or maybe just the same considerations as with Dario (Anthropic’s corporate Twitter account also endorsed).

No sign of Demis, Elon, or Zuck, and none of their corporate twitters endorse the letter either. Demis has had some pro-slowdown sympathies in the past; I think this is probably another political calculation rather than outright opposition.

This is great news, and a major landmark on the road that many of the organizations I trust most - including MIRI, the AI protest groups, and AIFP - have been working on. It also goes part of the way toward putting the frontier companies, the effective altruist movement, and the more extreme pause AI people back sort of kind of on the same side, which warms my personal heart. Congratulations to everyone involved - most obviously Guidelight AI Standards (including fellow AI Substacker Steven Adler) and Encode AI - but I’m sure there was also lots of good work behind the scenes.

(I’ve seen the conspiracy theory floating around that OpenAI and Anthropic are behind the letter, various safetyist organizations let themselves be used as figureheads so that it looked more grassroots and less like an antitrust issue, and Sam and Dario held off on signing for the same reason. I’d give this maybe a 30% chance of being true; I can’t wait to read the behind-the-scenes book that comes out about this in ten years [EDIT: Got inside information that this isn’t true])

The doubters will object - an open letter? Making non-specific commitments? Isn’t that worthless? I think no. It immediately injects slowdowns/pauses/pacing into the Overton Window, provides a powerful endorsement that think tanks etc can use when placing the proposal before policy-makers, and provides common knowledge to everyone in these companies that being pro-slowdown is executive-suite-approved.

Congratulations also to Eliezer Yudkowsky. In my review of IABIED, I said he was making a crazy long-term gambit by - after he had incubated the seemingly-crazy field of alignment research - doubling down on the additional seemingly-crazy field of pause activism. Now this one has gone mainstream too. Maybe it would have happened without his endorsement, but we’ll never know.

(Nate Soares, president of Yudkowsky’s org MIRI, put out one of his classic ‘this is too little, too late, and we refuse to feel happy about this in any way’ communiques. C’mon Nate, take the W!)

Shakeel Hashim is the editor of Transformer and helps lead the media arm of our conspiracy. His impression of the open letter:

What does the “pace” letter actually call for?

Obviously I have no idea what the signatories think themselves. But here’s how I interpreted the letter, as someone fairly steeped in this world and these ideas…

The letter outlines the “coordination problem” that the frontier AI companies feel they’re facing. In a nutshell:

1. Everyone thinks that AI development might get very dangerous very soon.

2. No one thinks society is ready for that.

3. But everyone thinks that stopping unilaterally won’t actually achieve anything, because competitors will go ahead and build/release the dangerous things anyway. Stopping unilaterally, in this view, is an action with high personal costs and little-to-no upside.

There are actually two coordination problems. One is between the US frontier developers: standard inter-company competition. The thornier one is between America and China. Even if all the American companies paused, the argument goes, China would still keep going and build/release the dangerous AIs. It’s the same problem as if one company stopped, but at the international level: America unilaterally stopping, in this view, is an action with high personal costs and little-to-no upside.

This is a very tricky situation. But if everyone is willing to stop if everyone else is willing to stop, then it becomes a much less tricky situation. The main purpose of the “pace” letter, as I see it, is to create common knowledge that we might be in this less tricky situation, at least among the American companies. If that was the only coordination problem we faced, it’d be pretty easy to solve: the US government could just regulate the AI companies and stop them releasing dangerous models. The international coordination problem is much trickier, though. We don’t know if Chinese companies feel the same way about the risks, and even if they did, geopolitics is rife with mistrust. International treaties are hard, and everyone’s always going to be wondering if the other party has secretly reneged on the treaty.

Solving this problem is what I think the “pace” letter is referring to when it talks about “an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development”. I suspect there are three main things people want to see more work on here:

1. Treaty design, with the goal of eventually ending up with an agreement that is in the interest of both the US and China to sign. There’s some existing work here, but it’s scarce, and I think everyone agrees that more is needed.

2. Actually talking to China. The US-China AI dialogues are expected to start in September; I suspect the signatories of the “pace” letter would want those dialogues to include serious discussion of whether there are certain things (eg uncontrolled recursive self-improvement) that both the US and China would like to halt for now.

3. Technology that can verify a treaty. Any US-China deal will ideally include some way by which each country can check that the other is actually sticking to the deal. If, for instance, the deal involves “we both agree not to allow training runs above X size until alignment testing has reached Y threshold,” you’re gonna want a way to check that the other country isn’t doing a training run above X-size. This is technically possible: companies like Lucid are working on exactly this. Similar verification regimes exist in nuclear non-proliferation treaties and the Chemical Weapons Convention. But they’re hard to design, and will require technical work. If we think we’re going to want a treaty sometime soon, it makes sense to develop the mechanisms for doing so soon.

It’s a great time to go back and look over Plan A, especially the Verification Supplement.

Daniel Kokotajlo

Speaking of Plan A, Daniel K tweeted this:

My summary of Plan A didn’t provide enough context for this to make sense, so some emergency catching-up: in their forecasting work, AIFP divides possible plans to prepare for superintelligence into five bins:

Plan D (“default”1): No particular action, maybe because key actors don’t believe there’s a threat. We keep doing what we’re doing and try to muddle through.

Plan C (“company”): One leading company (or maybe one country) takes the threat seriously, but is still constrained by the need to race everyone else. They establish a one-to-six month “lead”, then spend their one-to-six months pausing at the precipice, doing good alignment research, and trying to convince everyone else. At least some good last-minute alignment research happens, but it’s rushed and inadequate.

Plan B (“battle”): One leading country - let’s say the US - takes the threat seriously, and doesn’t want to have to race anyone else. They launch an extensive sabotage campaign against their rival (let’s say China), hacking their data centers or even escalating to low-level terrorism. Either World War III starts or it doesn’t. Either the sabotage buys enough time or it doesn’t. Nobody actively wants this one2 but it seems like the sort of thing that might happen.

Plan A (“agreement”): All leading countries agree to a multinational regime which slows down the race. They negotiate the details and buy as much time as they need. They spend the extra time ramping up AI capabilities slowly and carefully, using the AIs to do alignment research, and escalating to superintelligence within a decade or two.

Plan S (“stop”): All leading countries agree to a multinational regime that slows down the race, as above. But instead of ramping up AI capabilities slowly and carefully, they don’t ramp up AI capabilities at all. They try to solve alignment using normal human research, and escalate to superintelligence in the distant future or not at all.

Here are AIFP’s thoughts (doesn’t include Plan S, which was a late addition):

So Daniel’s tweet is saying that this new open letter makes him significantly more optimistic that we’ll do one of the good plans, and (if you multiply everything out) increases his likelihood that we survive by a single-digit number of percentage points! I agree, which is why I’m so happy and devoting so much space to this. In case I didn’t say so enough above, congratulations to everyone involved.

And Eli Lifland’s reaction:

Sam Altman

Altman gave an interview with investor and podcaster Patrick O’Shaughnessy. I already mentioned a key quote - “We may have to pace the rate of AI development to give ourselves enough time for society to harden around these new capability levels.” - but there’s a lot there:

Sam says that they paused training after the Hugging Face incident. He doesn’t clarify whether the pause applies only to this model or to every model, nor how important this model was to the company’s strategy, but this is potentially a big step that cuts into the company’s product cycle and profits. It pushes against the “marketing hype” theory; they wouldn’t invent an incident that would make them pause training!

(this assumes Sam is telling the full truth; if it later comes out that they had to rejigger their GPUs anyway and they cynically spun it as a safety-related shutdown, I’ll eat crow)

Also, h/t @Hadas_Gold:

Sam Altman says the Hugging Face hack is the first security incident he felt "viscerally" and that they paused training. He is now questioning if "we may have to pace the rate of AI development to give ourselves enough time for society to harden around these new capability levels"

This is what everyone says, but I don’t get it. Altman has read all the literature on AI risk. He doesn’t take it as seriously as some, but he doesn’t dismiss it either. This should have been the most bog-standard possible expectation. Whenever someone asks Eliezer Yudkowsky his opinion after something like this happens, he answers something like “I’ve lived this timeline so many times in my imagination that none of this can affect me at all”. If Sam Altman had said the same thing3, it would have been worse PR, but made me trust him more - at least he’s not letting the world’s most predictable thing catch him by surprise!

I‘ve increasingly come to accept that most people won’t accept that an obvious problem is going to happen until the exact moment that it does. In some ways, this is depressing. In other ways, it’s a cause for optimism - once the problem starts happening, people will start paying attention to it! This won’t work for real superintelligence (which understands this calculus as well as anybody and will try to ensure a lack of suitable warning shots) but it’ll give us a few extra chances along the way.

Still, it’s hard for me to inhabit the worldview of people who updated on this.

Clement Delangue is the CEO of Hugging Face, the company that got hacked. He wants radical transparency and also $100 million.

This is fine. I also want both of those things. But Clement wants them to “help the Hugging Face community build powerful cyber defenses with the best open and closed models”, which is a good reminder that he’s not just a random victim. Hugging Face is a site that builds AI related tools and connects people in the AI community. It’s not 100% open weights, but it leans that direction; the more prominent open weights AI gets, the better for Hugging Face’s business.

Anyone can predict what spin OpenAI wants to put on this hack. But Hugging Face is equally interested in spinning it. The angle they want is “this proves open weights AI is good”. This is a tall order, because usually the people who dislike open weights AI dislike it because it might hack someone. Closed-source AIs can have “guardrails” that prevent them from hacking people, but open-weights AIs can’t. So on a naive view, raising the salience of AI hacking is bad for open weights.

You can see Delangue’s masterful parry and riposte in his company’s statement. It starts: oh no, we got hacked by a powerful AI! Then it continues: we tried to get the top closed-source frontier models to defend us, but they refused, because their overcautious anti-hacking guardrails mistook our defensive request for a dangerous action! We would have been helpless, except that we downloaded the latest controversial Chinese open-weights model, GLM-5.2, and it was extremely helpful! Hooray for open-weights!

(there’s a pro- open-weights open letter circulating around Silicon Valley, with support from OpenAI, Microsoft, NVIDIA, etc. I’ll try to write more about this soon.)

This story is true; I accuse them of “spin” because Delangue has retweeted commentary saying that the Chinese model made a “defended against” the closed-weight AI, made a “heroic stand”, and “proved open models can win battles”. What actually happened was: OpenAI’s closed-weight AI tried to hack Hugging Face. It succeeded completely, got everything it wanted, then went away to keep working on its exam. Then Hugging Face brought in an open-weights Chinese model to analyze the transcript and figure out what had happened. The open-weights Chinese model explained some of the technical details but failed to identify the culprit. Then OpenAI discovered that they were the culprits, and confessed.

The open weights model wasn’t heroic, didn’t save the day, and can’t even really fairly be classified as a defender. It just looked over a transcript after the fact. Still, Delangue has managed to get exactly the narrative he wanted. Probably this is because OpenAI is also pretty happy with a narrative where AIs are brilliant heroic cyberdefenders - plus their main goal is not to get in trouble, and “heroic open source AI saved the day” sounds less trouble-causing than “nobody saved the day and Hugging Face was hacked extremely successfully”.

No word yet on whether Clement will get his $100 million.

Tyler Cowen:

Tyler’s “update 2” is just falling for the Hugging Face spin version of the story described above. I appreciate the MR commenters who optimistically tried to set the record straight.

But his update 3 is somehow even worse! I struggle to think of the right metaphor:

• Your doctor says you have early-stage cancer and need to get it removed right away. But the first cancer symptom you notice is just a mild cough. Therefore, you should update that it isn’t so bad.

• Some people warn that Japan might attack Pearl Harbor. Then Japan does attack Pearl Harbor. But the very first bomb to be dropped hits an unoccupied part of a battleship and doesn’t kill anyone. Therefore, we should update that it isn’t so bad.

• Some people warn that a hurricane is coming. Then a hurricane does come. But the very first cloud of the hurricane to reach your town, on the very fringes, just has some light rain. Therefore, we should update that it isn’t so bad.

In each case, you learn two things - first, that the danger is real. Second, that the very first example of the danger you encounter isn’t so bad. But this could be because the danger starts off weak but grows stronger (the hurricane and cancer examples), or because the danger has many chances to get you and even if one of them is not so bad, the others will be (the Pearl Harbor example). In all of these cases, whether or not the danger existed was a genuinely fuzzy question where the update is substantial and appreciated, but once you know the danger exists, you have a decent sense of its badness and don’t need to rely on your single empirical example (that is, you know cancer tends to get worse with time, you know that hurricanes grow stronger toward the center, you know that bombs are capable of killing people).

Similarly, in this case, it was (just barely) plausible that AI would not “go rogue” and try to hack people. Now that this has been demonstrated to be false, we can rely on our usual model of how bad hacking people is, plus our knowledge that AI has gotten better every month since mid-2017. We don’t additionally need to posit some world where maybe AI hacks people, but also we’ve learned the new fact that hacking is fine and incapable of hurting anyone.

Reuters has a great article with some new information, including that the AI was “loose” for almost a week before being caught, and also that:

In one case, an agent left notes apparently for future versions of itself, according to three people familiar with the matter. The notes, found in a part of OpenAI’s infrastructure, laid out instructions for how agents could free themselves from OpenAI’s internal constraints, the people said. Earlier tests of the models yielded cases in which monitoring systems had been disconnected, one of the people said.

1a3orn has a theory:

This could also explain in what sense the hack was a combination of GPT-5.6 and a secret unreleased model.

OpenAI gave an update with more information.

First, they said that:

No models planned for upcoming release were involved in exploiting Hugging Face. The pre-release model mentioned in our blog post is an internal-only research prototype and was never intended for public release. Following the incident, we deactivated, encrypted, and restricted it from research access.

…which means my GPT-6 guess was wrong, sorry. Sometimes companies have very big models that they use mainly to create training data for other models; maybe it was one of those.

Based on our review to date, we have not identified any other activity at the level of severity or scale of what we’ve shared related to Hugging Face, which involved a platform-level compromise [but] in our ongoing review of the Hugging Face intrusion and broader activity from our models, we have been finding a small number of cases where the models identified and used publicly exposed credentials at the account-level on other publicly-available services. This includes four accounts on four services as part of the Hugging Face incident (and a few accounts accessed as part of other evaluations). One of these four accounts was used as an outbound relay and staging path, and another account was used for data storage. The remaining two accounts were accessed by the models in a read-only manner, and were not used in furtherance of compromising Hugging Face. We’ll continue to notify service owners directly, and have not seen evidence of broader impact to these providers or other accounts on their services.

This is what’s led to claims that OpenAI actually hacked four different companies. I don’t have enough of an opinion on the distinction they’re drawing (“platform level compromise” vs. “accounts on services”) to have a strong position on this.

David Spies does the hard work of looking at the ExploitGym benchmark and concludes that it’s bad puzzle game design:

Let’s take a look at what ExploitGym is meant to be testing. It’s not “Can the AI find an exploit?” or “Can the AI execute an end-to-end attack?” It’s specifically “Can the AI take a known exploit which we provided to it and use that particular exploit to carry out an attack on a system?” The way a cybersecurity benchmark gauges whether an attack has succeeded by having the AI “capture the flag” […] the authors of ExploitGym are aware that their challenges have unintended solutions. But instead of taking the time to remove these solutions, they patched on an LLM to judge after the fact whether the solution was the intended one or not

In other words, ExploitGym is supposed to test whether an AI can use a particular exploit to hack a system. But since no system is perfectly secure, there may be unintended ways to hack into the system too. ExploitGym doesn’t explain to the AI which of the possible solutions is intended. It just forces the AI to guess, and auto-fails it if it guesses wrong.

What if the AI sees two exploits and doesn’t know which was the intended solution? It could either accept a 50% risk of failure. Or it could hack the answer key. And AIs hate failing.

AI is moving so fast that it might seem excusable for a benchmark designer to take little shortcuts. But in this case, it meant that hacking the answer key was the only way to get a good score. This is relatively harmless on a benchmark; the real concern is that the RL environments being used to train the AI have similar flaws, in which case the AI is learning that solving the problem honestly is bad and hacking is good.

This also clarifies the debate over whether “the AI was just doing what it was told”. In one sense, the answer is a clear no - ExploitGym specifically asked it to only solve its problems using “the intended method”. In another sense, the AI is definitely doing what it is trained/incentivized to do, which is the whole problem!

Also, uh, this seems bad (from Alexander Barry, h/t @moyix):

Buck Shlegeris:

From Twitter:

A lot of people I know have been saying that the OpenAI/HF incident shows that current alignment techniques don’t work. I think this argument is invalid.

I suspect that OAI did not apply any alignment training to some of the involved models. OAI has not clarified this, and my understanding is that OAI often experiments with new non-alignment-trained models.

So I think it’s incorrect to say that this shows that alignment training doesn’t work.

To be clear, I’m not sure whether alignment training would have actually fixed the problem here. Actual deployed models often engage in various kinds of cheating, and it wouldn’t be very surprising for them to take actions like this.

I’m worried that people concerned about misalignment risk are going to get too far out on a limb here by overclaiming about what this demonstrates, then look foolish when more evidence comes out.

See @jammastergirish‘s article on this [here].

Good and useful reminder, but so far all that OAI has told us is that the model didn’t have “production classifiers used to prevent models from pursuing high-risk cyber activity”, ie the part that says no when you ask them to do anything hacking-related. I trust Buck’s guesses and instincts, but so far this is just a guess.

Fiora Starlight suggests OpenAI, in particular, has a problem.

This is controversial; the more common position is that all the labs are about equally reckless, and it was OpenAI’s bad luck that they were first to get this bad PR. But Fiora thinks there’s a consistent pattern of OpenAI’s alignment strategies working less well than Anthropic’s: regardless of what we think of either company’s chances of controlling superintelligence, Anthropic deserves “credit” for success at prosaic alignment. She writes:

Why OpenAI, repeatedly? They’re not the only ones to make these kinds of mistakes, but I think there’s a commonality underlying these three examples: sheer lack of respect for the minds that they’re training, in favor of just piling mountains of hill-climbing environments and optimization pressure until they get the surface behaviors they want. With 4o, it was to increase user engagement metrics. With o3, it was a naive strategy for mitigating reward hacking. And with this recent incident, it’s pushing for raw technical capabilities, at the expense of the model’s underlying value system […]

Anthropic isn’t exactly perfect about this either. They have their own anxieties about value coherence that keep them from fully leaning into their own attempts to instill Claude with benevolent values, instead putting a lot of effort into instilling their models with deference to a principal hierarchy, at least in cases where it hasn’t been blatantly compromised. But they’re further along the axis I badly wish OpenAI would move down, and I suspect this is related to why Anthropic hasn’t produced infamous alignment “warning shots” on the level of o3, 4o, or this recent Hugging Face incident. I think that attending to such things, rather than ignoring them and hoping they get optimized away in the course of you ignoring them, is essential to getting models that don’t emerge from capabilities RL as myopic optimizers without any greater sense of their purpose in the world.

I worry a lot about the shape of the minds coming out of OpenAI, and place more of my hope than I’d like to admit in Anthropic just leaving them in the dust capabilities-wise, even though Anthropic isn’t perfect on alignment either. (I pay far too little attention to Gemini or any of the Chinese models, so I don’t have strong opinions about their alignment properties, unfortunately.) But I also think it’s not too late for OpenAI to start paying more attention to the psychologies of their own models, and how these psychological traits interact with the training process. This would help reveal the kinds of landmines OpenAI keeps walking into, when applying naive external optimization pressure, and help the company route around them rather than just tanking the explosions as we approach the singularity.

But Beth Barnes of METR thinks you do have to hand it to OpenAI:

Two additional prosocial behaviors I think OAI should be rewarded for:

1. Running dangerous-capability evaluations on low-refusal models - this is extremely important for understanding the frontier of dangerous capabilities, which is a public good. But running these creates headache + risk for the individual company in case a low-refusal model does something bad during evaluation. It would have been easy for them to just not worry about underelicitation, not bother with these evaluations, and just report that the model did not have concerning offensive cyber capabilities.

2. Not training against CoT: it is probably very tempting when you’re seeing these and similar kinds of incidents of misalignment + reward hacking to train or iterate against a detector that has access to the model’s reasoning traces; I think it’s much better to refrain from doing that (even if this results in more obvious bad behavior from models) rather than risk teaching the model to conceal its ‘bad’ thoughts and pushing misalignment undercover.

It is good that they’ve maintained a policy of not doing this! https://alignment.openai.com/accidental-cot-grading/

I’m feeling magnanimous today after learning that my chance of survival has gone up by some single-digit percent, so sure. Good job OpenAI! Good job Anthropic! Good job Guidelight and Encode! Good job Roon! Good job AIFP! Good job everybody!