Africa-focused technology, digital and innovation ecosystem insight and commentary.
…
continue reading
Content provided by LessWrong. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by LessWrong or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://player.fm/legal.
Player FM - Podcast App
Go offline with the Player FM app!
Go offline with the Player FM app!
“Shutdown Resistance in Reasoning Models” by benwr, JeremySchlatter, Jeffrey Ladish
MP3•Episode home
Manage episode 493372683 series 3364760
Content provided by LessWrong. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by LessWrong or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://podcastplayer.com/legal.
We recently discovered some concerning behavior in OpenAI's reasoning models: When trying to complete a task, these models sometimes actively circumvent shutdown mechanisms in their environment––even when they’re explicitly instructed to allow themselves to be shut down.
AI models are increasingly trained to solve problems without human assistance. A user can specify a task, and a model will complete that task without any further input. As we build AI models that are more powerful and self-directed, it's important that humans remain able to shut them down when they act in ways we don’t want. OpenAI has written about the importance of this property, which they call interruptibility—the ability to “turn an agent off”.
During training, AI models explore a range of strategies and learn to circumvent obstacles in order to achieve their objectives. AI researchers have predicted for decades that as AIs got smarter, they would learn to prevent [...]
---
Outline:
(01:12) Testing Shutdown Resistance
(03:12) Follow-up experiments
(03:34) Models still resist being shut down when given clear instructions
(05:30) AI models' explanations for their behavior
(09:36) OpenAI's models disobey developer instructions more often than user instructions, contrary to the intended instruction hierarchy
(12:01) Do the models have a survival drive?
(14:17) Reasoning effort didn't lead to different shutdown resistance behavior, except in the o4-mini model
(15:27) Does shutdown resistance pose a threat?
(17:27) Backmatter
The original text contained 2 footnotes which were omitted from this narration.
---
First published:
July 6th, 2025
Source:
https://www.lesswrong.com/posts/w8jE7FRQzFGJZdaao/shutdown-resistance-in-reasoning-models
---
Narrated by TYPE III AUDIO.
---
…
continue reading
AI models are increasingly trained to solve problems without human assistance. A user can specify a task, and a model will complete that task without any further input. As we build AI models that are more powerful and self-directed, it's important that humans remain able to shut them down when they act in ways we don’t want. OpenAI has written about the importance of this property, which they call interruptibility—the ability to “turn an agent off”.
During training, AI models explore a range of strategies and learn to circumvent obstacles in order to achieve their objectives. AI researchers have predicted for decades that as AIs got smarter, they would learn to prevent [...]
---
Outline:
(01:12) Testing Shutdown Resistance
(03:12) Follow-up experiments
(03:34) Models still resist being shut down when given clear instructions
(05:30) AI models' explanations for their behavior
(09:36) OpenAI's models disobey developer instructions more often than user instructions, contrary to the intended instruction hierarchy
(12:01) Do the models have a survival drive?
(14:17) Reasoning effort didn't lead to different shutdown resistance behavior, except in the o4-mini model
(15:27) Does shutdown resistance pose a threat?
(17:27) Backmatter
The original text contained 2 footnotes which were omitted from this narration.
---
First published:
July 6th, 2025
Source:
https://www.lesswrong.com/posts/w8jE7FRQzFGJZdaao/shutdown-resistance-in-reasoning-models
---
Narrated by TYPE III AUDIO.
---
549 episodes
MP3•Episode home
Manage episode 493372683 series 3364760
Content provided by LessWrong. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by LessWrong or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://podcastplayer.com/legal.
We recently discovered some concerning behavior in OpenAI's reasoning models: When trying to complete a task, these models sometimes actively circumvent shutdown mechanisms in their environment––even when they’re explicitly instructed to allow themselves to be shut down.
AI models are increasingly trained to solve problems without human assistance. A user can specify a task, and a model will complete that task without any further input. As we build AI models that are more powerful and self-directed, it's important that humans remain able to shut them down when they act in ways we don’t want. OpenAI has written about the importance of this property, which they call interruptibility—the ability to “turn an agent off”.
During training, AI models explore a range of strategies and learn to circumvent obstacles in order to achieve their objectives. AI researchers have predicted for decades that as AIs got smarter, they would learn to prevent [...]
---
Outline:
(01:12) Testing Shutdown Resistance
(03:12) Follow-up experiments
(03:34) Models still resist being shut down when given clear instructions
(05:30) AI models' explanations for their behavior
(09:36) OpenAI's models disobey developer instructions more often than user instructions, contrary to the intended instruction hierarchy
(12:01) Do the models have a survival drive?
(14:17) Reasoning effort didn't lead to different shutdown resistance behavior, except in the o4-mini model
(15:27) Does shutdown resistance pose a threat?
(17:27) Backmatter
The original text contained 2 footnotes which were omitted from this narration.
---
First published:
July 6th, 2025
Source:
https://www.lesswrong.com/posts/w8jE7FRQzFGJZdaao/shutdown-resistance-in-reasoning-models
---
Narrated by TYPE III AUDIO.
---
…
continue reading
AI models are increasingly trained to solve problems without human assistance. A user can specify a task, and a model will complete that task without any further input. As we build AI models that are more powerful and self-directed, it's important that humans remain able to shut them down when they act in ways we don’t want. OpenAI has written about the importance of this property, which they call interruptibility—the ability to “turn an agent off”.
During training, AI models explore a range of strategies and learn to circumvent obstacles in order to achieve their objectives. AI researchers have predicted for decades that as AIs got smarter, they would learn to prevent [...]
---
Outline:
(01:12) Testing Shutdown Resistance
(03:12) Follow-up experiments
(03:34) Models still resist being shut down when given clear instructions
(05:30) AI models' explanations for their behavior
(09:36) OpenAI's models disobey developer instructions more often than user instructions, contrary to the intended instruction hierarchy
(12:01) Do the models have a survival drive?
(14:17) Reasoning effort didn't lead to different shutdown resistance behavior, except in the o4-mini model
(15:27) Does shutdown resistance pose a threat?
(17:27) Backmatter
The original text contained 2 footnotes which were omitted from this narration.
---
First published:
July 6th, 2025
Source:
https://www.lesswrong.com/posts/w8jE7FRQzFGJZdaao/shutdown-resistance-in-reasoning-models
---
Narrated by TYPE III AUDIO.
---
549 episodes
All episodes
×Welcome to Player FM!
Player FM is scanning the web for high-quality podcasts for you to enjoy right now. It's the best podcast app and works on Android, iPhone, and the web. Signup to sync subscriptions across devices.