Best Practices

Apache Airflow, when scaled using KEDA (Kubernetes Event-Driven Autoscaler) has a few considerations and limitations users should be aware of. These mostly arise due to Airflow’s architectural assumptions and the nature of event-driven autoscaling.

Best Practices

• Use cooldownPeriod and pollingInterval settings in ScaledObject to avoid flapping.
• Configure resource limits to avoid starvation of scheduler or webserver.
• Always test autoscaling behavior under load with representative DAGs.

Considerations

1. Worker Lifecycle Awareness

Airflow workers spun up via KEDA may not always gracefully shut down, especially if they’re scaled down during a running task. This can result in task retries or missed heartbeats if workers are terminated before finishing their assigned DAG tasks.

Mitigation
Ensure proper terminationGracePeriodSeconds and use KEDA’s cooldownPeriod to prevent premature scale-down.

2. Task Queue Visibility

Airflow uses Celery, KubernetesExecutor, or CeleryKubernetesExecutor. KEDA typically monitors external queues (like Redis or RabbitMQ) for scaling signals. As a result, KEDA may not see task queue depth in real time if not properly configured with a Scaler compatible with Airflow’s executor backend.

Mitigation
Use a KEDA scaler (e.g. Redis or Prometheus) tailored to the executor’s backend.

3. No Native Awareness of DAG State

KEDA scales based on external metrics (e.g., queue length, CPU usage). It does not understand Airflow-specific concepts like DAG concurrency limits, task dependencies, or execution windows.

Impact
Over-aggressive scaling can overload the system or exhaust resources, especially if DAGs are not parallelizable.

4. Cold Start Overhead

Scaling from zero or very few replicas introduces latency (cold start) before tasks start executing. This can affect SLA guarantees, especially for short-lived or high-frequency DAGs.

Mitigation
Set a minReplicaCount > 0 to keep a baseline capacity alive at all times.

5. Metrics Availability

KEDA relies on metrics for autoscaling decisions. If metrics (like queue depth) are delayed or noisy, scaling can be slow or inaccurate.

Mitigation
Use stable, low-latency metrics sources and fine-tune polling intervals.