Airflow - Operators vs. TaskFlow decorators

✅ 1. Operators (Classic style)

Tasks are created explicitly using Operator classes like BashOperator, PythonOperator, ShortCircuitOperator, etc.

Example

from airflow.operators.bash import BashOperator

task = BashOperator(
    task_id="run_bash",
    bash_command="echo hello",
)

Characteristics

Aspect	Operators
Syntax style	Imperative ("define tasks, then set dependencies")
Task type	Anything with a specific operator (Bash, SQL, Kubernetes, EMR, etc.)
XCom	Must be managed manually (xcom_pull, return values explicitly)
UI display	Shows operator class name (e.g., BashOperator)
Dependencies	Set manually (task1 >> task2)
Pros	Very explicit; works with all operator types; stable; more control
Cons	Verbose; Python functions need PythonOperator wrappers

---

✅ 2. Decorators (TaskFlow API)

Introduced in Airflow 2.0 to simplify DAG creation using modern, Pythonic syntax.

Example

from airflow.decorators import task

@task
def run_bash():
    return "hello"

run_bash()

Characteristics

Aspect	Decorators (TaskFlow)
Syntax style	Functional and declarative
Task type	Python tasks only (using your own Python code)
XCom	Automatic (function return value → XCom)
UI display	Shows task as @task Python call
Dependencies	Inferred automatically via function calls
Pros	Very clean; auto XCom; fewer imports; easier testing
Cons	Only for Python tasks; cannot replace BashOperator, SparkSubmitOperator, etc.

---

🔍 Key Behavioral Differences

1. Creation Model

Operators:

You instantiate a task:

task = BashOperator(task_id="do", ...)

Decorators:

You call a function and it becomes a task:

@task
def do():
    ...
do()

---

2. Dependencies

Operators:

Set explicitly:

task1 >> task2

Decorators:

Dependencies come from function calls:

result = task1()
task2(result)

Airflow infers:

task1 >> task2

---

3. XCom Handling

Operators:

Manual:

value = ti.xcom_pull(task_ids='task1')

Decorators:

Automatic:

@task
def t1():
    return 5

@task
def t2(x):
    print(x)

t2(t1())   # passes XCom automatically

---

4. Types of Work You Can Do

Operators:

Use when you need specialized behaviors:

• Bash

• SQL

• Docker

• Kubernetes

• Spark

• EMR

• Sensors

• Transfer operators (S3 → GCS)

Decorators:

Only for Python code.

You still need operators for non-Python tasks.

---

5. UI Appearance

Operators:

Tooltip shows operator class:

BashOperator

Decorators:

Tooltip shows:

Python decorated @task

---

🎯 When to Use Each

Use Operators when:

• You need a specialized operator (Bash, Spark, Docker, etc.)

• You want explicit, more traditional Airflow construction

• You are migrating older DAGs

• You need fine-grained task control

Use TaskFlow decorators when:

• Your logic is pure Python

• You want automatic XCom passing

• You prefer cleaner, modern DAG definitions

• You're building fully Python-based workflows

---

🧩 Combined Use (Most Common)

Most modern DAGs use both together:

@task
def process_data():
    return "result"

run_spark = SparkSubmitOperator(task_id="spark_job", application="app.py")

process_data() >> run_spark