#Airflow kubernetes worker install#

Thanks to K8S you can leverage Keda to autoscale Celery workers. It’s worth mention that Airflow with Celery executor can be deployed in Kubernetes. It would be nice if this two kinds of a job get different amount of resources.Īll of mentioned problems can be solved using Airflow with Kubernetes executor. On the other hand you can run machine learning model training as Airflow task. Some tasks like SparkSubmitOperator require small amount of resources, as they only watch process running on external system. resource allocation - you can’t configure available resources per each Airflow task.

#Airflow kubernetes worker install#

Nobody will allow you to install your favorite library without corporate process. In big company most of the time you’re only user of Airflow. library management - you need to install all your dependencies (like Java) beforehand on all of your workers.Maybe it would be better to give this resources to other parts of the system? waste of resources - Celery workers are created on start-up.autoscaling - there is no mechanism to scale Celery workers based on task queue usage out of the box,.Nothing is perfect, let’s have a look on limitation of Celery executors: prioritization - you can give priority to your critical tasks.horizontal scalability - new Airflow workers can be added anytime,.speed - workers are always ready to use immediately,.Running Airflow with Celery executor has several major advantages: Starting from Airflow 2.0 webserver works only in stateless mode.

This means DAGs are now fetched from the database, instead of being parsed by Webserver as before. Starting from Airflow version 1.10.7, webserver can be stateless.