Search a title or topic

Over 20 million podcasts, powered by 

Player FM logo
Artwork

Content provided by Dev and Doc. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Dev and Doc or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://podcastplayer.com/legal.
Player FM - Podcast App
Go offline with the Player FM app!

Everything you need to know about LLM benchmarks- Turing Test, OpenAI's Healthbench, ARC prize, LM arena

55:19
 
Share
 

Manage episode 501751128 series 3585389
Content provided by Dev and Doc. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Dev and Doc or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://podcastplayer.com/legal.

Whenever there was AI, there were benchmarks- from the turing test, to society-changing benchmarks like MNIST and ImageNet to modern problems like the ARC prize, benchmarked served a vital purpose to measure the performance of AI models. But something has shifted in modern times, in the LLM era have benchmarks lost their utility, becoming mere advertisement for big tech?

Even seemingly more sophisticated benchmarks like LM Arena can be gamed by tech giants. We also deep dive into healthcare benchmarks like OpenAI's Healthbench (deeply problematic) and Microsoft's AI-DXO orchestrator agent for diagnosis. Where is this all going? How do we make the perfect benchmark? Or is the real work to be done afterwards in the real world?

๐Ÿ‘‹ Hey! If you are enjoying our conversations, reach out, share your thoughts and journey with us. Don't forget to subscribe whilst you're here :)

---

Timestamps
00:00 Intro - The OG benchmarks - Turing test, MNIST, ImageNET
06:40 Are large language models benchmarks similar to humans taking tests?
10:05 Are we testing model capability vs production ready?
12:00 LLM era - data contamination
15:30 LM Arena - The leaderboard illusion paper - how big tech games benchmarks
28:35 Goodhart's law - When a measure becomes a target, it ceases to be a good measure
32:05 Some good benchmarks - games - Pokemon, ARC prize, Minecraft
34:35 Medical benchmarks - OpenAI's healthbench has some big problems
46:50 Microsoft AI-DXO orchestrator for case reports

---

Connect with Us

Your Hosts:
๐Ÿ‘จ๐Ÿปโ€โš•๏ธ Doc - Dr. Joshua Au Yeung - LinkedIn
๐Ÿค– Dev - Zeljko Kraljevic - Twitter

Follow & Subscribe:
YT: https://youtube.com/@DevAndDoc
Spotify: Follow us on Spotify
Apple Podcasts: Listen on Apple Podcasts
Substack: https://aiforhealthcare.substack.com/

For enquiries:
๐Ÿ“ง [email protected]

---

Production Credits
๐ŸŽž๏ธ Editor: Dragan Kraljeviฤ‡ - Instagram
๐ŸŽจ Brand & Art: Ana Grigorovici - Behance

  continue reading

30 episodes

Artwork
iconShare
 
Manage episode 501751128 series 3585389
Content provided by Dev and Doc. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Dev and Doc or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://podcastplayer.com/legal.

Whenever there was AI, there were benchmarks- from the turing test, to society-changing benchmarks like MNIST and ImageNet to modern problems like the ARC prize, benchmarked served a vital purpose to measure the performance of AI models. But something has shifted in modern times, in the LLM era have benchmarks lost their utility, becoming mere advertisement for big tech?

Even seemingly more sophisticated benchmarks like LM Arena can be gamed by tech giants. We also deep dive into healthcare benchmarks like OpenAI's Healthbench (deeply problematic) and Microsoft's AI-DXO orchestrator agent for diagnosis. Where is this all going? How do we make the perfect benchmark? Or is the real work to be done afterwards in the real world?

๐Ÿ‘‹ Hey! If you are enjoying our conversations, reach out, share your thoughts and journey with us. Don't forget to subscribe whilst you're here :)

---

Timestamps
00:00 Intro - The OG benchmarks - Turing test, MNIST, ImageNET
06:40 Are large language models benchmarks similar to humans taking tests?
10:05 Are we testing model capability vs production ready?
12:00 LLM era - data contamination
15:30 LM Arena - The leaderboard illusion paper - how big tech games benchmarks
28:35 Goodhart's law - When a measure becomes a target, it ceases to be a good measure
32:05 Some good benchmarks - games - Pokemon, ARC prize, Minecraft
34:35 Medical benchmarks - OpenAI's healthbench has some big problems
46:50 Microsoft AI-DXO orchestrator for case reports

---

Connect with Us

Your Hosts:
๐Ÿ‘จ๐Ÿปโ€โš•๏ธ Doc - Dr. Joshua Au Yeung - LinkedIn
๐Ÿค– Dev - Zeljko Kraljevic - Twitter

Follow & Subscribe:
YT: https://youtube.com/@DevAndDoc
Spotify: Follow us on Spotify
Apple Podcasts: Listen on Apple Podcasts
Substack: https://aiforhealthcare.substack.com/

For enquiries:
๐Ÿ“ง [email protected]

---

Production Credits
๐ŸŽž๏ธ Editor: Dragan Kraljeviฤ‡ - Instagram
๐ŸŽจ Brand & Art: Ana Grigorovici - Behance

  continue reading

30 episodes

๋ชจ๋“  ์—ํ”ผ์†Œ๋“œ

×
 
Loading …

Welcome to Player FM!

Player FM is scanning the web for high-quality podcasts for you to enjoy right now. It's the best podcast app and works on Android, iPhone, and the web. Signup to sync subscriptions across devices.

 

Copyright 2025 | Privacy Policy | Terms of Service | | Copyright
Listen to this show while you explore
Play