Search a title or topic

Over 20 million podcasts, powered by 

Player FM logo
Artwork

Content provided by Patrick Wheeler and Jason Gauci, Patrick Wheeler, and Jason Gauci. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Patrick Wheeler and Jason Gauci, Patrick Wheeler, and Jason Gauci or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://podcastplayer.com/legal.
Player FM - Podcast App
Go offline with the Player FM app!

185: Workflow Orchestrators

1:32:02
 
Share
 

Manage episode 517701083 series 8393
Content provided by Patrick Wheeler and Jason Gauci, Patrick Wheeler, and Jason Gauci. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Patrick Wheeler and Jason Gauci, Patrick Wheeler, and Jason Gauci or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://podcastplayer.com/legal.

Intro topic: Asymmetric Returns

News/Links:

Book of the Show

Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h

Tool of the Show

Topic: Workflow Orchestrators

  • Why
    • Batch jobs (embarrassingly parallel)
    • Long-running tasks (e.g. transcoding video)
    • Checkpointing/resuming
  • How
    • Message Queues
    • Containerization
    • Worker Pools & Autoscaling
    • History & Backfill
  • Steps to run workflows:
    • Containerize the workflow definition and send to the cloud
    • Containerize all the individual tasks
    • Submit job(s)
  • Examples
    • Airflow
      • Legacy but dominant
    • Dagster
      • Great UX for python developers
    • Temporal: https://temporal.io/
      • The new hotness
    • Ray
      • Low-level but very powerful
    • Kubeflow
      • Designed for ML workflows, integrated dashboard

★ Support this podcast on Patreon ★
  continue reading

186 episodes

Artwork

185: Workflow Orchestrators

Programming Throwdown

7,601 subscribers

published

iconShare
 
Manage episode 517701083 series 8393
Content provided by Patrick Wheeler and Jason Gauci, Patrick Wheeler, and Jason Gauci. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Patrick Wheeler and Jason Gauci, Patrick Wheeler, and Jason Gauci or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://podcastplayer.com/legal.

Intro topic: Asymmetric Returns

News/Links:

Book of the Show

Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h

Tool of the Show

Topic: Workflow Orchestrators

  • Why
    • Batch jobs (embarrassingly parallel)
    • Long-running tasks (e.g. transcoding video)
    • Checkpointing/resuming
  • How
    • Message Queues
    • Containerization
    • Worker Pools & Autoscaling
    • History & Backfill
  • Steps to run workflows:
    • Containerize the workflow definition and send to the cloud
    • Containerize all the individual tasks
    • Submit job(s)
  • Examples
    • Airflow
      • Legacy but dominant
    • Dagster
      • Great UX for python developers
    • Temporal: https://temporal.io/
      • The new hotness
    • Ray
      • Low-level but very powerful
    • Kubeflow
      • Designed for ML workflows, integrated dashboard

★ Support this podcast on Patreon ★
  continue reading

186 episodes

All episodes

×
 
Loading …

Welcome to Player FM!

Player FM is scanning the web for high-quality podcasts for you to enjoy right now. It's the best podcast app and works on Android, iPhone, and the web. Signup to sync subscriptions across devices.

 

Copyright 2025 | Privacy Policy | Terms of Service | | Copyright
Listen to this show while you explore
Play