Search a title or topic

Over 20 million podcasts, powered by 

Player FM logo
Artwork

Content provided by Kieran Gilmurray. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Kieran Gilmurray or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://podcastplayer.com/legal.
Player FM - Podcast App
Go offline with the Player FM app!

The $3 Trillion Question: Can AI Match Human Experts?

14:36
 
Share
 

Manage episode 509276480 series 3535718
Content provided by Kieran Gilmurray. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Kieran Gilmurray or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://podcastplayer.com/legal.

What happens when AI attempts the same complex work as human experts with 14 years of experience? The answer might reshape our understanding of the economic future.

TL;DR:

  • GDP Val tests AI on complex, multimodal tasks requiring handling of CAD designs, spreadsheets, and presentations
  • Tasks are created from actual professional work products that take humans an average of 7 hours to complete
  • Claude Opus performed best with 47.6% of its deliverables rated as good as or better than human experts
  • AI shows potential to make workflows 40% faster and 63% cheaper when paired with human oversight
  • 3% of AI failures were classified as "catastrophic," including incorrect medical diagnoses and suggestions of financial fraud
  • Simple prompt improvements like asking models to self-check their work significantly reduced formatting errors
  • Current models still struggle with ambiguity and tasks requiring tacit knowledge or complex human interaction

GDP Val represents a fundamental shift in how we evaluate artificial intelligence. Rather than abstract academic metrics, this new benchmark from OpenAI measures how well frontier AI models handle real-world economic tasks across nine major sectors worth $3 trillion annually.

The methodology is ruthlessly practical—AI models must complete complex assignments that typically take human experts seven hours, handling everything from CAD designs to financial spreadsheets while synthesizing information from up to 38 reference documents.
The results are both promising and sobering. Claude Opus led the evaluation with 47.6% of its outputs rated equal to or better than work from professionals at organizations like Apple, Goldman Sachs, and Boeing. When integrated into realistic workflows with human oversight, these models demonstrated potential to make knowledge work 40% faster and 63% cheaper.

Yet failures remain significant—3% were classified as "catastrophic," including incorrect medical diagnoses and recommendations of financial fraud.
Perhaps most valuable is GDP Val's illumination of where AI currently excels (document formatting, data analysis) and where it falters (following complex instructions, handling ambiguity).

This economic lens offers businesses and policymakers unprecedented clarity about AI's near-term impact on knowledge work, while highlighting that the highest-value human skills—tacit knowledge, real-time collaboration, and complex communication—remain beyond current AI capabilities.

How quickly will that gap close? That's the trillion-dollar question worth pondering.

Listen into a audio version of this report created using Google Notebook LM for your listening pleasure.

Link to research: GDPval.pdf

Support the show

𝗖𝗼𝗻𝘁𝗮𝗰𝘁 my team and I to get business results, not excuses.
☎️ https://calendly.com/kierangilmurray/results-not-excuses
✉️ [email protected]
🌍 www.KieranGilmurray.com
📘 Kieran Gilmurray | LinkedIn
🦉 X / Twitter: https://twitter.com/KieranGilmurray
📽 YouTube: https://www.youtube.com/@KieranGilmurray
📕 Want to learn more about agentic AI then read my new book on Agentic AI and the Future of Work https://tinyurl.com/MyBooksOnAmazonUK

  continue reading

Chapters

1. Introducing GDP Val Benchmark (00:00:00)

2. Methodology and Task Complexity (00:01:36)

3. Evaluation Process and AI Grading (00:04:55)

4. Model Performance and Economic Impact (00:06:28)

5. Limitations and Catastrophic Failures (00:10:05)

6. Future Improvements and Big Picture (00:12:56)

147 episodes

Artwork
iconShare
 
Manage episode 509276480 series 3535718
Content provided by Kieran Gilmurray. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Kieran Gilmurray or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://podcastplayer.com/legal.

What happens when AI attempts the same complex work as human experts with 14 years of experience? The answer might reshape our understanding of the economic future.

TL;DR:

  • GDP Val tests AI on complex, multimodal tasks requiring handling of CAD designs, spreadsheets, and presentations
  • Tasks are created from actual professional work products that take humans an average of 7 hours to complete
  • Claude Opus performed best with 47.6% of its deliverables rated as good as or better than human experts
  • AI shows potential to make workflows 40% faster and 63% cheaper when paired with human oversight
  • 3% of AI failures were classified as "catastrophic," including incorrect medical diagnoses and suggestions of financial fraud
  • Simple prompt improvements like asking models to self-check their work significantly reduced formatting errors
  • Current models still struggle with ambiguity and tasks requiring tacit knowledge or complex human interaction

GDP Val represents a fundamental shift in how we evaluate artificial intelligence. Rather than abstract academic metrics, this new benchmark from OpenAI measures how well frontier AI models handle real-world economic tasks across nine major sectors worth $3 trillion annually.

The methodology is ruthlessly practical—AI models must complete complex assignments that typically take human experts seven hours, handling everything from CAD designs to financial spreadsheets while synthesizing information from up to 38 reference documents.
The results are both promising and sobering. Claude Opus led the evaluation with 47.6% of its outputs rated equal to or better than work from professionals at organizations like Apple, Goldman Sachs, and Boeing. When integrated into realistic workflows with human oversight, these models demonstrated potential to make knowledge work 40% faster and 63% cheaper.

Yet failures remain significant—3% were classified as "catastrophic," including incorrect medical diagnoses and recommendations of financial fraud.
Perhaps most valuable is GDP Val's illumination of where AI currently excels (document formatting, data analysis) and where it falters (following complex instructions, handling ambiguity).

This economic lens offers businesses and policymakers unprecedented clarity about AI's near-term impact on knowledge work, while highlighting that the highest-value human skills—tacit knowledge, real-time collaboration, and complex communication—remain beyond current AI capabilities.

How quickly will that gap close? That's the trillion-dollar question worth pondering.

Listen into a audio version of this report created using Google Notebook LM for your listening pleasure.

Link to research: GDPval.pdf

Support the show

𝗖𝗼𝗻𝘁𝗮𝗰𝘁 my team and I to get business results, not excuses.
☎️ https://calendly.com/kierangilmurray/results-not-excuses
✉️ [email protected]
🌍 www.KieranGilmurray.com
📘 Kieran Gilmurray | LinkedIn
🦉 X / Twitter: https://twitter.com/KieranGilmurray
📽 YouTube: https://www.youtube.com/@KieranGilmurray
📕 Want to learn more about agentic AI then read my new book on Agentic AI and the Future of Work https://tinyurl.com/MyBooksOnAmazonUK

  continue reading

Chapters

1. Introducing GDP Val Benchmark (00:00:00)

2. Methodology and Task Complexity (00:01:36)

3. Evaluation Process and AI Grading (00:04:55)

4. Model Performance and Economic Impact (00:06:28)

5. Limitations and Catastrophic Failures (00:10:05)

6. Future Improvements and Big Picture (00:12:56)

147 episodes

All episodes

×
 
Loading …

Welcome to Player FM!

Player FM is scanning the web for high-quality podcasts for you to enjoy right now. It's the best podcast app and works on Android, iPhone, and the web. Signup to sync subscriptions across devices.

 

Copyright 2025 | Privacy Policy | Terms of Service | | Copyright
Listen to this show while you explore
Play