Inside Bosch’s Airflow 3 Revolution: Remote Execution With Jens Scheffler The Data Flowcast: Mastering Apache Airflow ® For Data Engineering And AI podcast

Content provided by The Data Flowcast. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by The Data Flowcast or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://podcastplayer.com/legal.

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI »
Inside Bosch’s Airflow 3 Revolution: Remote Execution with Jens Scheffler

4h ago 28:02

MP3•Episode home

The evolution of Airflow has reached a milestone with the introduction of remote execution in Airflow 3, enabling flexible orchestration across distributed environments.

In this episode, Jens Scheffler, Test Execution Cluster Technical Architect at Bosch, shares insights on how his team’s need for large-scale, cross-environment testing influenced the development of the Edge Executor and shaped this major release.

Key Takeaways:

(02:39) The role of remote execution in supporting large-scale testing needs.

(04:44) How community support contributed to the Edge Executor’s development.

(08:41) Navigating network and infrastructure limitations within secure environments.

(13:25) Transitioning from database-heavy processes to an API-driven model.

(14:16) How the new task SDK in Airflow 3 improves distributed task execution.

(16:54) What is required to set up and configure the Edge Executor.

(19:36) Managing multiple queues to optimize tasks across different environments.

(23:30) Examples of extreme distance use cases for edge execution.

Resources Mentioned:

Jens Scheffler

https://www.linkedin.com/in/jens-scheffler/

Bosch | LinkedIn

https://www.linkedin.com/company/bosch/

Bosch | Website

https://www.bosch.com/

Apache Airflow

https://airflow.apache.org/

Edge Executor (Edge3 Provider Package)

https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html

Astronomer’s Astro Executor

https://www.astronomer.io/docs/astro/astro-executor/

Beyond Analytics Conference

http://astronomer.io/beyond/dataflowcast

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

65 episodes

#Tech #Airflow

Inside Bosch’s Airflow 3 Revolution: Remote Execution with Jens Scheffler

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

published 4h ago