124 lines
5.4 KiB
ReStructuredText
124 lines
5.4 KiB
ReStructuredText
.. Licensed to the Apache Software Foundation (ASF) under one
|
|
or more contributor license agreements. See the NOTICE file
|
|
distributed with this work for additional information
|
|
regarding copyright ownership. The ASF licenses this file
|
|
to you under the Apache License, Version 2.0 (the
|
|
"License"); you may not use this file except in compliance
|
|
with the License. You may obtain a copy of the License at
|
|
|
|
.. http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
.. Unless required by applicable law or agreed to in writing,
|
|
software distributed under the License is distributed on an
|
|
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
KIND, either express or implied. See the License for the
|
|
specific language governing permissions and limitations
|
|
under the License.
|
|
|
|
Autoscaling with KEDA
|
|
=====================
|
|
|
|
KEDA stands for Kubernetes Event Driven Autoscaling.
|
|
`KEDA <https://github.com/kedacore/keda>`__ is a custom controller that
|
|
allows users to create custom bindings to the Kubernetes `Horizontal Pod
|
|
Autoscaler <https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/>`__.
|
|
The autoscaler will adjust the number of active Celery workers based on the number
|
|
of tasks in ``queued`` or ``running`` state.
|
|
|
|
One advantage of KEDA is that it allows you to scale your application to/from 0 workers, meaning no workers are idle when there are no tasks.
|
|
|
|
KEDA Installation and usage
|
|
---------------------------
|
|
|
|
To install KEDA in your Kubernetes cluster, run the following commands:
|
|
|
|
.. code-block:: bash
|
|
|
|
helm repo add kedacore https://kedacore.github.io/charts
|
|
helm repo update
|
|
kubectl create namespace keda
|
|
helm install keda kedacore/keda \
|
|
--namespace keda \
|
|
--version "v2.0.0"
|
|
|
|
To enable KEDA for the Airflow instance, it has to be enabled by setting ``workers.celery.keda.enabled=true``
|
|
in your Helm command or in the ``values.yaml`` like:
|
|
|
|
.. code-block:: bash
|
|
|
|
kubectl create namespace airflow
|
|
helm repo add apache-airflow https://airflow.apache.org
|
|
helm install airflow apache-airflow/airflow \
|
|
--namespace airflow \
|
|
--set executor=CeleryExecutor \
|
|
--set workers.celery.keda.enabled=true
|
|
|
|
.. note::
|
|
|
|
Make sure ``values.yaml`` shows that either KEDA or HPA is enabled, but not both. It is recommended not
|
|
to use both KEDA and HPA to scale the same workload. They will compete with each other resulting in odd scaling behavior.
|
|
|
|
After installation, the KEDA ``ScaledObject`` and an ``HPA`` will be created in the Airflow namespace.
|
|
|
|
In the default configuration, KEDA will derive the desired number of Celery workers by querying Airflow metadata database with following SQL statement:
|
|
|
|
.. code-block:: none
|
|
|
|
SELECT
|
|
ceil(COUNT(*)::decimal / {{ .Values.config.celery.worker_concurrency }})
|
|
FROM
|
|
task_instance
|
|
WHERE
|
|
(state='running' OR state='queued')
|
|
AND queue IN <queue names>
|
|
|
|
where ``<queue names>`` is a list of queue names used by
|
|
`Celery worker queues <https://airflow.apache.org/docs/apache-airflow-providers-celery/stable/celery_executor.html#queues>`_
|
|
mechanism (with default configuration it has one element ``default``).
|
|
|
|
.. note::
|
|
|
|
Set Celery worker concurrency through the Helm Chart value
|
|
``config.celery.worker_concurrency`` (e.g. instead of airflow.cfg or
|
|
environment variables), so that the KEDA trigger will be consistent with
|
|
the worker concurrency setting.
|
|
|
|
Triggers (aka Scalers)
|
|
----------------------
|
|
|
|
Triggers refer to the metrics (or formulae) that KEDA should refer to when scaling workers.
|
|
|
|
It is recommended to use multiple triggers within a ScaledObject, rather than creating different objects for different triggers.
|
|
This keeps all your rules and formulae in one place, and it avoids multiple ScaledObjects being created by the same workload.
|
|
|
|
ScaledObject
|
|
------------
|
|
|
|
To configure KEDA's triggers and scaling behaviors, you need to create a ScaledObject. Below ScaledObject parameters:
|
|
|
|
* ``cooldownPeriod`` specifies the number of seconds to wait before downscaling to 0 workers, does not apply to downscaling to n workers while n >= 1.
|
|
* ``idleReplicaCount`` can be set to any number less than ``minReplicaCount``, but it must be set to 0, otherwise KEDA will not work. Change ``minReplicaCount`` to n > 0 if you need idle workers.
|
|
|
|
Triggerers value ``targetQueryValue`` is used as ``TargetValue`` of workers, which must be between ScaledObject ``minReplicaCount`` and ``maxReplicaCount`` values.
|
|
|
|
.. note::
|
|
|
|
To avoid strange behavior, best practice is to set ``cooldownPeriod`` to an integer slightly larger than ``terminationGracePeriodSeconds`` so that your cluster does not downscale to 0 workers before cleanup is finished.
|
|
|
|
Metrics
|
|
-------
|
|
|
|
The HPA controller, refreshes metrics defined in triggers every ``--horizontal-pod-autoscaler-sync-period`` and the values are routed to
|
|
KEDA Metrics Server directly. To reduce the load on the KEDA Scaler, you can set ``useCachedMetrics`` to true, to enabling reading metrics
|
|
from cache first. Cache is updated periodically every ``pollingInterval``.
|
|
|
|
.. note::
|
|
|
|
When number of workers = 0, KEDA will still poll for metrics using ``pollingInterval``.
|
|
When number of workers >= 1, both KEDA and the HPA will poll your defined triggers.
|
|
|
|
KEDA offers two ``metricTypes`` that provide more granular scaling control than the standard HPA ``Target`` metric:
|
|
|
|
* AverageValue (default) controls a per-worker average.
|
|
* Value controls total system load.
|