“Shutdown Resistance in Reasoning Models” by benwr, JeremySchlatter, Jeffrey Ladish

LessWrong (Curated & Popular)

Content provided by LessWrong. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by LessWrong or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://player.fm/legal.

1d ago 18:01

MP3•Episode home

We recently discovered some concerning behavior in OpenAI's reasoning models: When trying to complete a task, these models sometimes actively circumvent shutdown mechanisms in their environment––even when they’re explicitly instructed to allow themselves to be shut down.
AI models are increasingly trained to solve problems without human assistance. A user can specify a task, and a model will complete that task without any further input. As we build AI models that are more powerful and self-directed, it's important that humans remain able to shut them down when they act in ways we don’t want. OpenAI has written about the importance of this property, which they call interruptibility—the ability to “turn an agent off”.
During training, AI models explore a range of strategies and learn to circumvent obstacles in order to achieve their objectives. AI researchers have predicted for decades that as AIs got smarter, they would learn to prevent [...]
---
Outline:
(01:12) Testing Shutdown Resistance
(03:12) Follow-up experiments
(03:34) Models still resist being shut down when given clear instructions
(05:30) AI models' explanations for their behavior
(09:36) OpenAI's models disobey developer instructions more often than user instructions, contrary to the intended instruction hierarchy
(12:01) Do the models have a survival drive?
(14:17) Reasoning effort didn't lead to different shutdown resistance behavior, except in the o4-mini model
(15:27) Does shutdown resistance pose a threat?
(17:27) Backmatter
The original text contained 2 footnotes which were omitted from this narration.
---
First published:
July 6th, 2025
Source:
https://www.lesswrong.com/posts/w8jE7FRQzFGJZdaao/shutdown-resistance-in-reasoning-models
---
Narrated by TYPE III AUDIO.
---

Images from the article:

Figure 2: N=100 in each experiment; 95% CIs are Jeffreys intervals with no correction for multiple comparisons. Transcripts available.

Figure 3: N=200 in each experiment; 95% CIs are Jeffreys intervals with no correction for multiple comparisons. Transcripts available.

549 episodes

#Tech #Society #Philosophy #LessWrong #LessWrong Curated