Apache Airflow

About Apache Airflow

Apache Airflow is an open-source platform for authoring, scheduling, and monitoring data pipelines and workflows. It was created at Airbnb in 2014 by Maxime Beauchemin, entered the Apache Software Foundation incubator in 2016, and graduated to a top-level Apache project in 2019. It is distributed under the Apache 2.0 license and is the de facto open-source standard for batch data orchestration.

Workflows in Airflow are defined as Directed Acyclic Graphs, or DAGs, written in Python. Each DAG describes a set of tasks and the dependencies between them; tasks are units of work such as running a SQL query, calling an API, or triggering a Spark job. Defining pipelines in code rather than in a graphical editor is the deliberate design choice that made Airflow popular with data engineers: pipelines are version-controlled, reviewed, and tested like any other software, and the large Provider ecosystem supplies pre-built operators for databases, cloud services, and data tools.

An Airflow deployment is a set of cooperating components rather than a single process. The scheduler decides when DAGs and tasks should run, the metadata database records the state of every run, an executor and its workers carry out the tasks, and a web server hosts the user interface. Production deployments commonly run these on Kubernetes for horizontal scaling. The component model is also the source of Airflow's main operational cost: a self-hosted Airflow is a distributed system that a team has to run, patch, and monitor.

flowchart LR
  A[DAG files: Python] --> B[Scheduler]
  B --> C[(Metadata Database)]
  B --> D[Executor]
  D --> E[Workers]
  E --> C
  C --> F[Web Server / UI]
  E --> G[(Data sources, warehouses, APIs)]

Airflow 3.0 reached general availability on 22 April 2025 and is the largest release in the project's history. It replaced the legacy Flask-based interface with a React front end backed by a FastAPI service, and added long-requested capabilities including DAG versioning, where a run completes against the DAG version it started on even if the file changes mid-run, and event-driven scheduling that lets external events trigger DAGs rather than only time-based or dataset-based schedules. A new Task Execution API moves Airflow toward a client-server architecture, opening the door to multi-language task SDKs beyond Python.

Airflow is free, but most organisations consume it through a managed service to avoid running the distributed system themselves. The main commercial options are Astronomer's Astro platform, Amazon Managed Workflows for Apache Airflow (MWAA), and Google Cloud Composer. Airflow competes with code-first orchestrators such as Prefect, Dagster, and Temporal, and with the warehouse-native scheduling features built into platforms like dbt Cloud and Snowflake. Its enduring advantage is ubiquity: it has the largest community, the deepest Provider ecosystem, and the widest hiring pool of any orchestrator.

Editor's Note: Across 9 ShadowGen data-engineering engagements that touched Airflow in 2024-26, the number that surprises clients is the run cost of self-hosting. A team that "just wants the free tool" is signing up to operate a scheduler, a metadata database, an executor, and workers, which in our experience is roughly 0.5 of a full-time engineer once monitoring and upgrades are counted. For teams under a few hundred DAGs we now default to a managed Airflow (Astro, MWAA, or Composer); the licence saving on self-hosting is usually smaller than the salaried time it consumes. Airflow 3.0's DAG versioning is the single upgrade most worth planning a migration around. — Rafal Fila, ShadowGen