Building a Unified Data Platform at Pattern with William Graham

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Content provided by The Data Flowcast. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by The Data Flowcast or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://player.fm/legal.

3M ago 24:09

MP3•Episode home

The orchestration of data workflows at scale requires both flexibility and security. At Pattern, decoupling scheduling from orchestration has reshaped how data teams manage large-scale pipelines.

In this episode, we are joined by William Graham, Senior Data Engineer at Pattern, who explains how his team leverages Apache Airflow alongside their open-source tool Heimdall to streamline scheduling, orchestration and access management.

Key Takeaways:

00:00 Introduction.

02:44 Structure of Pattern’s data teams across acquisition, engineering and platform.

04:27 How Airflow became the central scheduler for batch jobs.

08:57 Credential management challenges that led to decoupling scheduling and orchestration.

12:21 Heimdall simplifies multi-application access through a unified interface.

13:15 Standardized operators in Airflow using Heimdall integration.

17:13 Open-source contributions and early adoption of Heimdall within Pattern.

21:01 Community support for Airflow and satisfaction with scheduling flexibility.

Resources Mentioned:

William Graham

https://www.linkedin.com/in/willgraham2/

Pattern | LinkedIn

https://www.linkedin.com/company/pattern-hq/

Pattern | Website

https://pattern.com

Apache Airflow

https://airflow.apache.org

Heimdall on GitHub

https://github.com/patterninc/heimdall

Netflix Genie

https://netflix.github.io/genie/

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

82 episodes

#IT Management #Tech #Industries #Airflow