Skip to content

Commit

Permalink
Add a guide for configuring the kubernetes executor
Browse files Browse the repository at this point in the history
  • Loading branch information
chrishronek committed Oct 11, 2023
1 parent 4bb87b2 commit ae397e9
Showing 1 changed file with 47 additions and 0 deletions.
47 changes: 47 additions & 0 deletions airflow/plural/docs/kubernetes-executor.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
## Configuring KubernetesExecutor

The [KubernetesExecutor](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/kubernetes.html)
in Apache Airflow is a valuable choice for specific use cases and scenarios. It offers several advantages that make it a
suitable execution engine for running your Airflow tasks.

### edit values.yaml

You'll then want to edit `airflow/helm/airflow/values.yaml` in your installation repo with something like:

```yaml
airflow:
airflow:
airflow:
config:
AIRFLOW__KUBERNETES__WORKER_PODS_CREATION_BATCH_SIZE: 5
executor: KubernetesExecutor
kubernetesPodTemplate:
resources:
limits:
cpu: 1
memory: 1Gi
requests:
cpu: 0.5
memory: 512Mi
flower:
enabled: false
redis:
enabled: false
workers:
enabled: false
```
The resources that you configure in the `kubernetesPodTemplate` section will determine the amount of resources that will
be available to your Airflow tasks, so if you get SIGTERM errors from your Airflow tasks, you may need to increase the
default cpu/memory and limits accordingly.

Additionally, we recommend to increase the [worker_pods_creation_batch_size](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/configurations-ref.html#worker-pods-creation-batch-size)
for your production workloads. This guide has it set to `5`, but you will need to experiment what works best for your
Airflow workloads to decrease any scheduling latency.

Alternatively, you should be able to do this in the configuration section for airflow in your plural console as well.

### redeploy

From there, you should be able to run `plural build --only airflow && plural deploy --commit "use kubernetesexecutor"`
to use the Airflow kubernetesexecutor.

0 comments on commit ae397e9

Please sign in to comment.