AIRFLOW
This course provides a practical introduction to Apache Airflow, an open-source platform used to programmatically author, schedule, and monitor data workflows. Students will learn how to build, manage, and optimize data pipelines using Directed Acyclic Graphs (DAGs), integrate Airflow with cloud services and databases, and apply best practices for automation and monitoring in modern data engineering.
Why should I join?
Airflow helps you manage:
Cloud workflows
ETL/ELT processes
Data movement between systems
Machine learning pipelines
Please contact us if you have any questions relating to any of the vika technologies features.
Course Content
Oracle Introduction
You don't currently have access to this content
3 Topics
What is Airflow?
You don't currently have access to this content
Airflow vs other schedulers (Cron, Luigi)
You don't currently have access to this content
Airflow architecture (Web Server, Scheduler, Worker, Metadata DB)
You don't currently have access to this content
Unix Commands
You don't currently have access to this content
3 Topics
Installing Airflow with Pip, Docker, or in the Cloud
You don't currently have access to this content
Understanding the Airflow UI
You don't currently have access to this content
Airflow CLI basics
You don't currently have access to this content
OS information
You don't currently have access to this content
4 Topics
What is a DAG?
You don't currently have access to this content
Creating your first DAG in Python
You don't currently have access to this content
DAG arguments: start_date, schedule_interval, default_args
You don't currently have access to this content
Backfilling and catchup
You don't currently have access to this content
Copy and move commands using cp, mv command
You don't currently have access to this content
4 Topics
• PythonOperator, BashOperator, DummyOperator
You don't currently have access to this content
Sensors (e.g., FileSensor, ExternalTaskSensor)
You don't currently have access to this content
Branching and conditional tasks
You don't currently have access to this content
Task retries, SLA, and timeouts
You don't currently have access to this content
Compression and un-compression
You don't currently have access to this content
3 Topics
Inter-task communication using XComs
You don't currently have access to this content
Task dependencies with set_upstream() / >> / <<
You don't currently have access to this content
Dynamic task mapping (Airflow 2.3+)
You don't currently have access to this content
Changing file permission
You don't currently have access to this content
4 Topics
Scheduling vs Triggering
You don't currently have access to this content
DAG runs and task instances
You don't currently have access to this content
Monitoring and alerting with email or Slack
You don't currently have access to this content
Logs and error handling
You don't currently have access to this content
Scheduling the job
You don't currently have access to this content
4 Topics
Running Airflow with Docker or Kubernetes
You don't currently have access to this content
Airflow on AWS (MWAA), GCP (Cloud Composer), Azure
You don't currently have access to this content
Managing connections and secrets
You don't currently have access to this content
Scaling with CeleryExecutor or KubernetesExecutor
You don't currently have access to this content
Editor command
You don't currently have access to this content
4 Topics
S3, GCS, Azure Blob storage
You don't currently have access to this content
Snowflake, BigQuery, Redshift
You don't currently have access to this content
Databricks, Spark, Hadoop
You don't currently have access to this content
SQL databases (PostgreSQL, MySQL)
You don't currently have access to this content
Best Practices and Advanced Concepts
You don't currently have access to this content
4 Topics
• DAG versioning and modularization
You don't currently have access to this content
CI/CD with Airflow
You don't currently have access to this content
Airflow plugins and custom operators
You don't currently have access to this content
Monitoring DAG health
You don't currently have access to this content
Final Project Ideas
You don't currently have access to this content
4 Topics
Daily ETL pipeline (CSV → Clean → Load to Database
You don't currently have access to this content
ML model retraining pipeline
You don't currently have access to this content
Data warehouse pipeline with Snowflake or BigQuery
You don't currently have access to this content
Real-time file ingestion using sensors
You don't currently have access to this content
Includes
10 Lessons
37 Topics
Admin
