Airflow- Introduction

Airflow for Data Engineers

Working with data involves a ton of prerequisites to get up and running with the required set of data, it’s formatting and storage. The first step of a data science process is Data engineering, which plays a crucial role in streamlining every other process of a data science project.

Traditionally, data engineering processes involve three steps: Extract, Transform and Load, which is also known as the ETL process. The ETL process involves a series of actions and manipulations on the data to make it fit for analysis and modeling. Most data science processes require these ETL processes to run almost every day for the purpose of generating daily reports.

Ideally, these processes should be executed automatically in definite time and order. But, it isn’t as easy as it sounds. You might have tried using a time-based scheduler such as Cron by defining the workflows in Crontab. This works fairly well for workflows that are simple. However, when the number of workflows and their dependencies increase, things start getting complicated. It gets difficult to effectively manage as well as monitor these workflows considering they may fail and need to be recovered manually. Apache Airflow is such a tool that can be very helpful for you in that case, whether you are a Data Scientist, Data Engineer, or even a Software Engineer.

What is Apache Airflow?

“Apache Airflow is an open-source workflow management platform. It started at Airbnb in October 2014 as a solution to manage the company’s increasingly complex workflows. “

Apache Airflow (or simply Airflow) is a highly versatile tool that can be used across multiple domains for managing and scheduling workflows. It allows you to perform as well as automate simple to complex processes that are written in Python and SQL. Airflow provides a method to view and create workflows in the form of Direct Acyclic Graphs (DAGs) with the help of intelligent command-line tools as well as GUIs.

Apache Airflow is a revolutionary open-source tool for people working with data and its pipelines. It is easy to use and deploy considering data scientists have basic knowledge of Python. Airflow provides the flexibility to use Python scripts to create workflows along with various ready to use operators for easy integrations with platforms such as Amazon AWS, Google Cloud Platform, Microsoft Azure, etc. Moreover, it ensures that the tasks are ordered correctly based on dependencies with the help of DAGs, and also continuously tracks the state of tasks being executed.

One of the most crucial features of Airflow is its ability to recover from failure and manage the allocation of scarce resources dynamically. This makes the Airflow a great choice for running any kind of data processing or modeling tasks in a fairly scalable and maintainable way.

In this tutorial, we will understand how to install it, create the pipeline and why data scientists should be using it, in detail.

Understanding the workflow and DAG

The set of processes that take place in regular intervals is termed as the ‘workflow’. It can consist of any task ranging from extracting data to manipulating them.

Direct Acyclic Graphs (DAG) are one of the key components of Airflow. They represent the series of tasks that needs to be run as a part of the workflow. Each task is represented as a single node in the graph along with the path it takes for execution, as shown in the figure.

Airflow also provides you with the ability to specify the order, relationship (if any) in between 2 or more tasks and enables you to add any dependencies regarding required data values for the execution of a task. A DAG file is a Python script and is saved with a .py extension.

Basic Components of Apache Airflow

Now that you have a basic idea about workflows and DAG, we’ve listed below some of the commonly used components of Apache Airflow that make up the architecture of the Apache Airflow pipeline.

Web Server: It is the graphical user interface built as a Flask app. It provides details regarding the status of your tasks and gives the ability to read logs from a remote file store.

Scheduler: This component is primarily responsible for scheduling tasks, i.e., the execution of DAGs. It retrieves and updates the status of the task in the database.

Executer: It is the mechanism that initiates the execution of different tasks one by one.

Metadata Database: It is the centralized database where Airflow stores the status of all the tasks. All read/write operations of a workflow are done from this database.

Now that you understand the basic architecture of the Airflow, let us begin by installing Python and Apache Airflow into our system.

Create your first Airflow pipeline

The Apache Airflow pipeline is basically an easy and scalable tool for data engineers to create, monitor and schedule one or multiple workflows simultaneously. The pipeline requires a database backend for running the workflows, which is why we will start by initializing the database using the command:

airflow initdb

Upon initializing the database, you can now start the server using the command

airflow webserver -p 8080

This will start an Airflow webserver at port 8080 on your localhost.
Note: You can specify any port as per your preference.

Now, open up another terminal and start the airflow scheduler using the command:

airflow scheduler

Now, if you have successfully started the airflow scheduler, you access the tool from your browser to monitor as well as manage the status of completed and ongoing tasks in your workflow at localhost:8080/admin.