About this episode
The orchestration of data workflows at scale requires both flexibility and security. At Pattern, decoupling scheduling from orchestration has reshaped how data teams manage large-scale pipelines.In this episode, we are joined by William Graham, Senior Data Engineer at Pattern, who explains how his team leverages Apache Airflow alongside their open-source tool Heimdall to streamline scheduling, orchestration and access management.Key Takeaways:00:00 Introduction.02:44 Structure of Pattern’s data teams across acquisition, engineering and platform.04:27 How Airflow became the central scheduler for batch jobs.08:57 Credential management challenges that led to decoupling scheduling and orchestration.12:21 Heimdall simplifies multi-application access through a unified interface.13:15 Standardized operators in Airflow using Heimdall integration.17:13 Open-source contributions and early adoption of Heimdall within Pattern.21:01 Community support for Airflow and satisfaction with scheduling flexibility.Resources Mentioned:William Grahamhttps://www.linkedin.com/in/willgraham2/Pattern | LinkedInhttps://www.linkedin.com/company/pattern-hq/
