Search a title or topic

Over 20 million podcasts, powered by 

Player FM logo
Artwork

Content provided by Hugo Bowne-Anderson. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Hugo Bowne-Anderson or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://podcastplayer.com/legal.
Player FM - Podcast App
Go offline with the Player FM app!

Episode 60: 10 Things I Hate About AI Evals with Hamel Husain

1:13:15
 
Share
 

Manage episode 509237981 series 3317544
Content provided by Hugo Bowne-Anderson. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Hugo Bowne-Anderson or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://podcastplayer.com/legal.

Most AI teams find "evals" frustrating, but ML Engineer Hamel Husain argues they’re just using the wrong playbook. In this episode, he lays out a data-centric approach to systematically measure and improve AI, turning unreliable prototypes into robust, production-ready systems.

Drawing from his experience getting countless teams unstuck, Hamel explains why the solution requires a "revenge of the data scientists." He details the essential mindset shifts, error analysis techniques, and practical steps needed to move beyond guesswork and build AI products you can actually trust.

We talk through:

  • The 10(+1) critical mistakes that cause teams to waste time on evals
  • Why "hallucination scores" are a waste of time (and what to measure instead)
  • The manual review process that finds major issues in hours, not weeks
  • A step-by-step method for building LLM judges you can actually trust
  • How to use domain experts without getting stuck in endless review committees
  • Guest Bryan Bischof's "Failure as a Funnel" for debugging complex AI agents

If you're tired of ambiguous "vibe checks" and want a clear process that delivers real improvement, this episode provides the definitive roadmap.

LINKS

🎓 Learn more:

  continue reading

60 episodes

Artwork
iconShare
 
Manage episode 509237981 series 3317544
Content provided by Hugo Bowne-Anderson. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Hugo Bowne-Anderson or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://podcastplayer.com/legal.

Most AI teams find "evals" frustrating, but ML Engineer Hamel Husain argues they’re just using the wrong playbook. In this episode, he lays out a data-centric approach to systematically measure and improve AI, turning unreliable prototypes into robust, production-ready systems.

Drawing from his experience getting countless teams unstuck, Hamel explains why the solution requires a "revenge of the data scientists." He details the essential mindset shifts, error analysis techniques, and practical steps needed to move beyond guesswork and build AI products you can actually trust.

We talk through:

  • The 10(+1) critical mistakes that cause teams to waste time on evals
  • Why "hallucination scores" are a waste of time (and what to measure instead)
  • The manual review process that finds major issues in hours, not weeks
  • A step-by-step method for building LLM judges you can actually trust
  • How to use domain experts without getting stuck in endless review committees
  • Guest Bryan Bischof's "Failure as a Funnel" for debugging complex AI agents

If you're tired of ambiguous "vibe checks" and want a clear process that delivers real improvement, this episode provides the definitive roadmap.

LINKS

🎓 Learn more:

  continue reading

60 episodes

Alla avsnitt

×
 
Loading …

Welcome to Player FM!

Player FM is scanning the web for high-quality podcasts for you to enjoy right now. It's the best podcast app and works on Android, iPhone, and the web. Signup to sync subscriptions across devices.

 

Copyright 2025 | Privacy Policy | Terms of Service | | Copyright
Listen to this show while you explore
Play