Jenkins vs Apache Airflow for ETL Jobs

Created by Andrii Malaman

Andrii Malaman

• Software/Data Engineer
• Python, Scala, Golang, JS, Java
• https://github.com/malaman

What is ETL Pipeline?

Questions to answer ?

• Automatic start (schedule)
• Dependencies between jobs
• Monitoring
• Restart after failure

Historical solution: CRON!

CRON

• Automatic start (schedule) ✅
• Dependencies between jobs 🚫
• Monitoring 🚫
• Restart after failure ⚠️

Popularity (stackshare.io)

Comparison params

• Easy to setup
• Easy to use
• Dependencies between jobs
• Monitoring
• Restart after failure

Easy to setup: Jenkins

Easy to setup: Airflow

Airflow: Provider packages

Dependencies between jobs: Jenkins

Dependencies between jobs: Airflow