Overleaf provides a rich-text editor so you don't need to know any code to get started you can just edit the text, add images, and see the resume, cv automatically created for you.
Grammarly improves your writing by checking your grammar, spelling, and punctuation correctness. It improves the clarity of your sentences by making them concise. It can help you with your resume writing.
Working with data involves a ton of prerequisites to get up and running with the required set of data, it’s formatting and storage. The first step of a data science process is Data engineering, which plays a crucial role in streamlining every other process of a data science project.
Traditionally, data engineering processes involve three steps: Extract, Transform and Load, which is also known as the ETL process. The ETL process involves a series of actions and manipulations on the data to make it fit for analysis and modeling. Most data science processes require these ETL processes to run almost every day for the purpose of generating daily reports.
Ideally, these processes should be executed automatically in definite time and order. But, it isn’t as easy as it sounds. You might have tried using a time-based scheduler such as Cron by defining the workflows in Crontab. This works fairly well for workflows that are simple. However, when the number of workflows and their dependencies increase, things start getting complicated. It gets difficult to effectively manage as well as monitor these workflows considering they may fail and need to be recovered manually. Apache Airflow is such a tool that can be very helpful for you in that case, whether you are a Data Scientist, Data Engineer, or even a Software Engineer.
What is Apache Airflow?
“Apache Airflow is an open-source workflow management platform. It started at Airbnb in October 2014 as a solution to manage the company’s increasingly complex workflows. “
Apache Airflow (or simply Airflow) is a highly versatile tool that can be used across multiple domains for managing and scheduling workflows. It allows you to perform as well as automate simple to complex processes that are written in Python and SQL. Airflow provides a method to view and create workflows in the form of Direct Acyclic Graphs (DAGs) with the help of intelligent command-line tools as well as GUIs.
Apache Airflow is a revolutionary open-source tool for people working with data and its pipelines. It is easy to use and deploy considering data scientists have basic knowledge of Python. Airflow provides the flexibility to use Python scripts to create workflows along with various ready to use operators for easy integrations with platforms such as Amazon AWS, Google Cloud Platform, Microsoft Azure, etc. Moreover, it ensures that the tasks are ordered correctly based on dependencies with the help of DAGs, and also continuously tracks the state of tasks being executed.
One of the most crucial features of Airflow is its ability to recover from failure and manage the allocation of scarce resources dynamically. This makes the Airflow a great choice for running any kind of data processing or modeling tasks in a fairly scalable and maintainable way.
In this tutorial, we will understand how to install it, create the pipeline and why data scientists should be using it, in detail.
Understanding the workflow and DAG
The set of processes that take place in regular intervals is termed as the ‘workflow’. It can consist of any task ranging from extracting data to manipulating them.
Direct Acyclic Graphs (DAG) are one of the key components of Airflow. They represent the series of tasks that needs to be run as a part of the workflow. Each task is represented as a single node in the graph along with the path it takes for execution, as shown in the figure.
Airflow also provides you with the ability to specify the order, relationship (if any) in between 2 or more tasks and enables you to add any dependencies regarding required data values for the execution of a task. A DAG file is a Python script and is saved with a .py extension.
Basic Components of Apache Airflow
Now that you have a basic idea about workflows and DAG, we’ve listed below some of the commonly used components of Apache Airflow that make up the architecture of the Apache Airflow pipeline.
Web Server: It is the graphical user interface built as a Flask app. It provides details regarding the status of your tasks and gives the ability to read logs from a remote file store.
Scheduler: This component is primarily responsible for scheduling tasks, i.e., the execution of DAGs. It retrieves and updates the status of the task in the database.
Executer: It is the mechanism that initiates the execution of different tasks one by one.
Metadata Database: It is the centralized database where Airflow stores the status of all the tasks. All read/write operations of a workflow are done from this database.
Now that you understand the basic architecture of the Airflow, let us begin by installing Python and Apache Airflow into our system.
Create your first Airflow pipeline
The Apache Airflow pipeline is basically an easy and scalable tool for data engineers to create, monitor and schedule one or multiple workflows simultaneously. The pipeline requires a database backend for running the workflows, which is why we will start by initializing the database using the command:
airflow initdb
Upon initializing the database, you can now start the server using the command
airflow webserver -p 8080
This will start an Airflow webserver at port 8080 on your localhost. Note: You can specify any port as per your preference.
Now, open up another terminal and start the airflow scheduler using the command:
airflow scheduler
Now, if you have successfully started the airflow scheduler, you access the tool from your browser to monitor as well as manage the status of completed and ongoing tasks in your workflow at localhost:8080/admin.
As a part of Data Engineering Series, we have already covered part-1(Data Engineering-Introduction) ,part-2(Basic Python) and part-3 (Advance Python). As a continuity of previous post on Advance python, we are going to see how to writing efficient code in python in this post
In python, Enumerate is used to write efficient python code. Many a times we need to keep a count of iterations. Python’s enumerate takes a collection i.e iterable, adds counter to it and returns it as an enumerate object
Syntax :
enumerate(iterable, start=0)
Implementation —
""" Enumerate : Use enumerate() function : Python’s enumerate takes a collection i.e iterable, adds counter to it and returns it as an enumerate object. """countries = ['USA','Canada','Singapore','Taiwan'] enum_countries = enumerate(countries)enumerate_countries = enumerate(countries,5) print(list(enumerate_countries)) print(type(enumerate_countries))
countries = ['USA','Canada','Singapore','Taiwan'] for i,item in enumerate(countries): print(i,item)
Output —
0 USA 1 Canada 2 Singapore 3 Taiwan
In python, Zip takes one or more iterables(list,tuples etc) and aggregates them into tuple and returns the iterator object
Syntax :
zip(*iterators)
Implementation —
# Use Zip : Zip takes one or more iterables and aggregates them into # tuple and returns the iterator objectname = ["Steve","Paul","Brad"] roll_no = [4,1,3] marks = [20,40,50]mapped = zip(name,roll_no,marks) mapped = set(mapped) print(mapped)
To make code work faster use builtin functions and libraries like map() which applies a function to every member of iterable sequence and returns the result.
Implementation —
""" Map function : In Python, map() function applies the given function #to each item of a given iterable construct (i.e lists, tuples etc) and returns a map object. """numbers =(100,200,300) result = map(lambda x:x+x,numbers) total = list(result) print(total)
Output —
[200, 400, 600]
NumPy arrays are homogeneous and provide a fast and memory efficient alternative to Python lists.NumPy arrays vectorization technique, vectorize operations so they are performed on all elements of an object at once which allows the programmer to efficiently perform calculations over entire arrays.
Implementation —
import numpy as np def reciprocals(values): output = np.empty(len(values)) for i in range(len(values)): output[i] = 1.0/values[i] return outputvalues = np.random.randint(1,15,size=6) reciprocals(values)
# Use multiple assignmentf_name,l_name,city = "Steve","Paul","NewYork"print(f_name,l_name,city)#To swap variablea = 5 b = 10a,b = b,a print(a,b)
Output —
Steve Paul NewYork 10 5
Use Comprehensions
Implementation —
#List Comprehensionlist_two = [5,10,15,20,20,40,50,60] new_list = [x**3 for x in list_two] print(new_list)#Dictionary Comprehensiondict_one = [1,2,3,4] new_dict = {x:x**2 for x in dict_one if x%2 ==0} print(new_dict)
Membership : To check if membership of a list, it’s generally faster to use the “in” keyword
Implementation —
days = ["sunday","monday","tuesday"] for d in days: print('Today is {}'.format(d)) print('tuesday' in days) print('friday' in days)
Output —
Today is sunday Today is monday Today is tuesday True False
Counter : Counter is one of the high performance container data types
Implementation —
from collections import Counter sample_dict = {'a':4,'b':8,'c':2} print(Counter(sample_dict))
Output —
Counter({'b': 8, 'a': 4, 'c': 2})
Python Itertools are fast, memory efficient functions — a collection of constructs for handling iterators.
Implementation —
import itertools for i in itertools.count(30,4): print(i) if i>30: break
Output —
30 34
Implementation 2 —
import itertools countries =[("West","USA"), ("East","Singapore"),("West","Canada"),("East","Taiwan")]iter_one = itertools.groupby(countries,lambda x:x[0]) for key,group in iter_one: result = {key:list(group)} print(result)
Let’s discuss different kinds of data file formats used in Big Data, These are widely used but unspoken of but are building blocks of all data engineering tasks they could be data processing, data visualization, or could be ML. You will have to retrieve data from someplace.
If this storage/file format is wrong then shall result in many common issues like slow query runtime, slow dashboard refreshes even certain joins on spark taking more time.
Anyone aspiring to be a Data Engineer/Data Analyst/ML engineer should be aware of this magic of file format in big data.
Why do we even need a data file format?
Imagine you visit a grocery store and nothing is in order, You will find all items on different shelves would this make your shopping experience better? I think the answer is No, In fact, you might never visit this grocery store again.
If you understood the example above you now can imagine the impact of unorganized data in a company can be.
Every company gets 10s & 1000s of GB data every day. If these are not stored in a proper format, then understanding this data will be difficult or impossible sometimes.
More time you spend sorting through the data, The company is missing out on the opportunity to retain customers or generate more orders/revenue.
This is why data file formats are used in Data lake and play a key role.
Different kinds of file formats
We have CSV,JSON,ORC,Parquet,Avro. These are the leading data file formats used today.
Which one to choose from and why to choose them completely depends on the use case and information you’re looking for.
Consider a use-case of sales in a company. Which file format would be the best way to store this information?
Use-case 1:
If you are looking into total sales data from a table, Then this requires 1 column in your table sale_amount to be scanned/queried mostly.
Hence storing this table in columnar format is the best way to go.
File formats to choose: ORC, Parquet
Use-case 2 :
If you are trying to identify the consumer behavior :
What kind of items are customers placing the order for?
Which category of item has the customer placed the most orders from?
To gather this information you have to scan row-level information spanning multiple columns like user_id, item_name, sale_amount.
Here, Storing the data in row-level format is the right choice to make.
File formats to choose: CSV, JSON, Avro
These 2 use-cases should be enough to showcase the power of data file formats.
Conclusion
It’s important to know that we don’t have one fit-all file format yet (Anything can happen in the future). The question you must ask “How will this data be used?” Based on this file format should be defined.
Now that you are aware of the impact this can do I believe you are sure what data file format to choose when you write the data into the data lake.
