About this episode
Modern data orchestration at scale demands reliability, speed and thoughtful adoption of new tooling. As organizations grow, keeping pipelines efficient while supporting more teams becomes a critical challenge.In this episode, we’re joined by Ethan Shalev, Data Engineer at Wix, to discuss how Wix operates Airflow at massive scale, migrates to Airflow 3 and uses AI to accelerate development.Key Takeaways:00:00 Introduction.02:13 Wix structures data engineering across multiple product-focused organizations.03:40 Migrating nearly 8,000 DAGs to Airflow 3 requires careful planning.04:31 Migration creates an opportunity to remove long-standing legacy Airflow code.05:32 Internal playbooks and Cursor rules standardize and speed up DAG migrations.07:39 Airflow 3 introduces backfills, DAG versioning and asset-aware scheduling.09:16 Deferrable operators reduce scheduler congestion in large Airflow environments.12:54 AI-generated code still requires review and strong testing practices.14:52 Moving to managed Airflow reduces operational burden on internal platform teams.15:57 Improving multi-tenancy and UI personalization remains a key Airflow need.Resources Mentioned:Ethan Shalevhttps://www.linkedin.com/in/e
