Airflow & Kestra: a Simple Benchmark

Having spent over 5 years as a data engineer, Airflow was my go-to workflow orchestration tool. With its robust ecosystem and widespread adoption, it often stands out as the go-to orchestration solution for many.

I’ve deployed it on-premise, with Google Cloud Composer, and on Kubernetes, all with the help of experienced engineers: maintaining the infrastructure was hard, developing and maintaining the Python codebase was time consuming.

Selecting the right orchestration tool involves in-depth technical evaluation:

• How long does it take to install and configure the tool? Both in development and production.
• How long does it take to write a pipeline? Does it scale easily? How long does it take to onboard developers?
• How long do task executions take? Does it perform well? Do performances scale?
• Can it support critical, customer-facing applications?

Now that I’m working with Kestra, a solution with a distinct approach, I wanted to compare it to Airflow. Many data-engineers and developers ask me how Kestra differs from Airflow? What are the advantages of moving to Kestra?

This post compares Airflow and Kestra, focusing on installation, configuration, pipeline syntax, and performance.

Here, I keep it simple: using getting-started documentation pages and no specific configuration tuning. Tests have been executed on Airflow version 2.7 and Kestra version 0.14.

Benchmarking is inherently difficult due to the multitude of potential options that need to be examined. The figures outlined in the following paragraph could be slightly different in other environments or with in-depth fine-tuning.

Installation & Configuration

Installation and configuration are probably the most boring parts of software. When looking for a new tool we always look for something as simple as possible to set up.

Everyone has infrastructure and system constraints and we often lose time in configuration overhaul. We’re all looking for easy accessibility and ease of customization.

In this advent, let’s look at Kestra and Airflow installation and configuration:

On the Airflow side, the quick start is made of several steps:

• Installing with Python package installer (pip)
• Edit some configuration variables
• Run airflow standalone that will start the instance with an in-memory database and expose the user interface on port 8080.

This installation comes with SequentialExecutor that can’t run tasks in parallel. To use another Executor, we have to setup a specific database backend, which add a bit more of configuration.

Another part of the documentation mentions a docker-compose installation that create several services such as a Redis database, a Postgres database, and Celery as executor engine — allowing task parallelism.

As highlighted in the Kestra getting started documentation, a simple docker-compose up -d allows the start of a Kestra instance alongside a Postgres container. Exposing the web interface on port 8080.

On the configuration side, the Airflow main server configuration setup can be edited with an airflow.cfg file. It’s almost the same idea in Kestra, which makes accessible a YAML environment file, extendable with Micronaut specification (as it’s used as the backend framework of Kestra).

Code Syntax

Having good programming semantics is probably the most important element when developing software. Under the hood, it means easier maintainability, easier onboarding, and easier discussion. Different people with different backgrounds can finally speak the same language. It brings back everyone to the table.

Ease of writing, readability, and scalability are the three parts we’re looking for. Combined, they empower software developers and save considerable time in project development.

Airflow is based on Python and the concept of Directed Acyclic Graph (DAG). Here is a basic hello-world DAG:

Hello World in Airflow

Kestra syntax is based on YAML. Here is a basic hello-world flow:

Hello World in Kestra

As Python is imperative, it’s often less readable and less understandable than YAML. But it has the advantage of being highly customizable.

YAML on the other side can have some pitfalls. Without proper tooling, YAML files in their raw format can be frustrating to work with, including issues such as:

• Inconsistent interpretations by different YAML parsers, e.g. some parsers may interpret 0123 as an octal number, while others will treat it as a string
• Missing whitespaces causing indentation errors
• The need to look up some property names or their types
• Uncertainty whether some property is supposed to be a map or a list of maps, etc.

However, it’s worth looking at those issues in the context of a specific tool. Kestra solved all of the common YAML challenges through an API-based engine and has made the right choice for a declarative definition of orchestration logic built on top of robust schemas with proper validation mechanisms. You can find more in this corresponding blog post.

In the rest of this blog, I will leverage a more complex pipeline to effectively evaluate the performance attributes of the two tools at hand. It will also better uncover how much the syntax can be complicated to grasp.

My objective is to establish an initial pipeline designed to ingest data from a CSV file and initiate multiple runs of a subsequent pipeline. In this process, each run will involve passing a batch of data as inputs, and five child tasks will be responsible for printing out these batches in the logs.

Using this approach, we can find out how orchestration engines perform when they’re running multiple concurrent tasks at once and understand their overall capabilities.

Here are the Kestra flows for such exercises:

This is the main flow:

id: for-each-item
namespace: io.kestra.tests

inputs:
  - name: file
    type: FILE

tasks:
  - id: each
    type: io.kestra.core.tasks.flows.ForEachItem
    items: "{{ inputs.file }}"
    batch:
      partitions: 1000
    namespace: io.kestra.tests
    flowId: for-each-item-subflow
    wait: true
    transmitFailed: true
    inputs:
      items: "{{ taskrun.items }}"

and the corresponding children flow being called by the main flow:

id: for-each-item-subflow
namespace: io.kestra.tests

inputs:
  - name: items
    type: STRING

tasks:
  - id: per-item-1
    type: io.kestra.core.tasks.log.Log
    message: "{{ inputs.items }}"

  - id: per-item-2
    type: io.kestra.core.tasks.log.Log
    message: "{{ inputs.items }}"

  - id: per-item-3
    type: io.kestra.core.tasks.log.Log
    message: "{{ inputs.items }}"

  - id: per-item-4
    type: io.kestra.core.tasks.log.Log
    message: "{{ inputs.items }}"

  - id: per-item-5
    type: io.kestra.core.tasks.log.Log
    message: "{{ inputs.items }}"

Here are the equivalent DAGs in Airflow:

from datetime import datetime, timedelta
import pandas as pd
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from airflow.operators.dagrun_operator import TriggerDagRunOperator

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2023, 12, 19),
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}

def process_csv_and_trigger_child_dag(**kwargs):
    csv_file = '/opt/airflow/dags/full.csv'  # Update with your CSV file path
    row_count = 5000
    desired_batch_count = 1000
    chunk_size = row_count // desired_batch_count
    # Read CSV in chunks
    for chunk in pd.read_csv(csv_file, chunksize=chunk_size):
        # Pass the chunk data as parameters to the child DAG
        params = {'csv_data': chunk.to_dict(orient='records')[0]}
        
        # Trigger child DAG for each chunk
        trigger = TriggerDagRunOperator(
            task_id=f'trigger_child_dag_{chunk.index[0]}',
            trigger_dag_id='child_v2_dag',
            conf=params,
            dag=kwargs['dag'],
        )

        trigger.execute(context=kwargs)

parent_dag = DAG(
    'dynamic_parent_dag_v2',
    default_args=default_args,
    description='Dynamic DAG to process CSV and trigger child DAG for each batch',
    schedule_interval=None,  # Set the schedule interval as needed
    catchup=False  # Set to False if you don't want historical runs to be triggered
)

process_csv_task = PythonOperator(
    task_id='process_csv_and_trigger_child_dag',
    python_callable=process_csv_and_trigger_child_dag,
    provide_context=True,
    dag=parent_dag,
)

and the corresponding sub-DAG

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python_operator import PythonOperator

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2023, 12, 19),
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}

def process_csv_data(**kwargs):
    csv_data = kwargs['dag_run'].conf['csv_data']  # Retrieve CSV data from parameters
    print(f"{csv_data}")

child_dag = DAG(
    'child_v2_dag',
    default_args=default_args,
    description='Child DAG to process CSV data for each batch',
    schedule_interval=None,  # Set the schedule interval as needed
    catchup=False  # Set to False if you don't want historical runs to be triggered
)

for i in range(0, 5):
    process_csv_task = PythonOperator(
        task_id=f'process_csv_data_{i}',
        python_callable=process_csv_data,
        provide_context=True,
        dag=child_dag,
    )

As we can see, there are a lot of differences in syntax.

In Airflow, one has to know about Python programming language and Airflow concepts to understand what’s going on. Kestra with it’s key-value system based on YAML doesn’t require programming skills to be readable.

Performances

Both Airflow and Kestra come with many backend options.

Airflow offers two main setups:

• Standalone version (Quick Start): This is a simple option for getting started, but it has limitations. It uses an in-memory SQLite database and doesn’t allow parallel execution of tasks due to the built-in SequentialExecutor. To leverage parallelism, you’ll need to explore Airflow’s configuration and choose a more suitable executor, potentially requiring additional installation steps.
• Multi-component version: This provides more power and scalability. It uses a separate database (like PostgreSQL) and a queueing system (like Redis) to handle tasks. Additionally, it utilizes the CeleryExecutor, which enables parallel execution of tasks within your pipelines.

Kestra offers several deployment choices:

• Standalone server: This is a basic setup suitable for getting started. It requires a separate database like PostgreSQL or MySQL to manage workflows.
• Multi-component version: Kestra components such as worker or executor can be scaled horizontally or vertically for a more advanced setup — especially for better resiliency and scalability.
• High-availability version: This provides increased reliability and scalability and can used in place of the PostgresSQL backend. It leverages Kafka, a high-throughput messaging system, and Elasticsearch, a powerful search and analytics engine, for handling workflows and data.

All tests, installation, and configuration have been done with the following context:

• run on a single VM on the cloud with 8 cores 32Gb of memory and 100Gb hard disk space.
• no tuning in configuration (servers and framework settings)
• CSV file input is about 800kb and 5000 rows.

Performance benchmark summary. I kept the SQLite setup for Airflow as it’s what used in the documentation quick start. In my opinion, the most interesting comparison can be made when using CeleryExecutor on Postgres database backend.

The key point of using an orchestration engine is having concurrent execution, not just the data processing task (which is often uncoupled through another service).

Kestra, designed for high throughput, performs better than Airflow in micro-batch processing. Kestra’s backend in Java might also contribute to its better performance compared to Airflow’s Python foundation.

Conclusion: What’s the Return On Investment?

When looking at the return on investment when choosing an orchestration tool, there are several points to consider:

• Time of installation/maintenance
• Time to write pipeline
• Time to execute (performance)

Based on the paragraphs above, it’s clear that Kestra offers a higher ROI globally compared to Airflow:

• Installing Kestra is easier than Airflow; it doesn’t require Python dependencies, and it comes with a ready-to-use docker-compose file using few services and without the need to understand what’s an executor to run task in parallel.
• Creating pipelines with Kestra is simple, thanks to its syntax. You don’t need knowledge of a specific programming language because Kestra is designed to be agnostic. The declarative YAML design makes Kestra flows more readable compared to Airflow’s DAG equivalent, allowing developers to significantly reduce development time.
• In this benchmark, Kestra demonstrates better execution time than Airflow under any configuration setup.

If you’re looking for great data resources and thinking, subscribe to my newsletter — 👀 From An Engineer Sight.