Search a title or topic

Over 20 million podcasts, powered by 

Player FM logo
Artwork

Content provided by The Data Flowcast. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by The Data Flowcast or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://player.fm/legal.
Player FM - Podcast App
Go offline with the Player FM app!

Building the Future of Airflow Execution at Astronomer with Ian Buss and Piotr Chomiak

22:25
 
Share
 

Manage episode 502869003 series 2053958
Content provided by The Data Flowcast. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by The Data Flowcast or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://podcastplayer.com/legal.

The evolution of orchestration in Airflow continues with innovations that address both scalability and security. From improving executor reliability to enabling remote execution, these advancements reshape how organizations manage data pipelines.

In this episode, we’re joined by Ian Buss, Principal Software Engineer at Astronomer, and Piotr Chomiak, Principal Product Manager at Astronomer, who share insights into the Astro Executor and remote execution.

Key Takeaways:

00:00 Introduction.

04:13 How product leadership drives scalability for enterprise needs.

08:23 Architectural changes that improve reliability and remove bottlenecks.

10:15 Metrics that enhance visibility into system performance.

12:54 The role of remote execution in addressing security requirements.

15:56 Differences between open-source solutions and managed offerings.

19:04 Broad industry adoption and applicability of remote execution.

20:39 Future advancements in language support and multi-tenancy.

Resources Mentioned:

Ian Buss

https://www.linkedin.com/in/ian-buss/

Piotr Chomiak

https://www.linkedin.com/in/piotr-chomiak-b1955624/

Astronomer | Website

https://www.astronomer.io

Apache Airflow

https://airflow.apache.org/

Airflow Slack Community

https://airflow.apache.org/community/

Beyond Analytics conference

https://astronomer.io/beyond/dataflowcast

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

  continue reading

68 episodes

Artwork
iconShare
 
Manage episode 502869003 series 2053958
Content provided by The Data Flowcast. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by The Data Flowcast or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://podcastplayer.com/legal.

The evolution of orchestration in Airflow continues with innovations that address both scalability and security. From improving executor reliability to enabling remote execution, these advancements reshape how organizations manage data pipelines.

In this episode, we’re joined by Ian Buss, Principal Software Engineer at Astronomer, and Piotr Chomiak, Principal Product Manager at Astronomer, who share insights into the Astro Executor and remote execution.

Key Takeaways:

00:00 Introduction.

04:13 How product leadership drives scalability for enterprise needs.

08:23 Architectural changes that improve reliability and remove bottlenecks.

10:15 Metrics that enhance visibility into system performance.

12:54 The role of remote execution in addressing security requirements.

15:56 Differences between open-source solutions and managed offerings.

19:04 Broad industry adoption and applicability of remote execution.

20:39 Future advancements in language support and multi-tenancy.

Resources Mentioned:

Ian Buss

https://www.linkedin.com/in/ian-buss/

Piotr Chomiak

https://www.linkedin.com/in/piotr-chomiak-b1955624/

Astronomer | Website

https://www.astronomer.io

Apache Airflow

https://airflow.apache.org/

Airflow Slack Community

https://airflow.apache.org/community/

Beyond Analytics conference

https://astronomer.io/beyond/dataflowcast

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

  continue reading

68 episodes

All episodes

×
 
Loading …

Welcome to Player FM!

Player FM is scanning the web for high-quality podcasts for you to enjoy right now. It's the best podcast app and works on Android, iPhone, and the web. Signup to sync subscriptions across devices.

 

Copyright 2025 | Privacy Policy | Terms of Service | | Copyright
Listen to this show while you explore
Play