This is a 16-hour hands-on workshop on Airflow with ultimate goal to make participants feel comfortable with the fundamentals of Airflow, discuss practices and patterns and gain insights by implementing a small use case end to end.
The use case will read data from one or more sources, transform the data in different formats and schemas in order to make it easier for data scientists to run queries and store them in a filesystem and/or a Data Lake query engine. Participants will also explore and work with the most commonly used operators to define ETL flows and apply practices to implement a layered data lake.
Who should attend
IT Professionals with some experience in programming, data analysis and statistics interested in expanding and building skills on a state of the art platform used to programmatically author, schedule and monitor workflows over big amounts of data
Prerequisites
There are no prerequisites but familiarity with IT terminology and IT related work experience would be beneficial.
What will you learn
During this hands-on course, the following topics will be covered through practical examples and exercises:
- Introduction to Workflow management systems
- Airflow Concepts
- DAGs
- Tasks
- Developing DAGs
- Basic DAG structure
- Parametricity in DAGs
- Using Task Operators
- Using Task Sensors
- Taskflow API
- Variables / Connections
- DAG development patterns
- Success/Failure/Retry Conditions and Patterns
- Taking alternative routes in DAGs
- DAG Factories
- Testing DAGs
- Deploying DAGs
- How to organize source control repository
- CI/CD scenarios for deployments
- Monitoring DAGs
- Web console
- Web API
- Logs
- Managing Airflow Plugins and Extensions
- Installing Extensions
- Implementing Extensions
