101 lines
3.6 KiB
ReStructuredText
101 lines
3.6 KiB
ReStructuredText
.. Licensed to the Apache Software Foundation (ASF) under one
|
|
or more contributor license agreements. See the NOTICE file
|
|
distributed with this work for additional information
|
|
regarding copyright ownership. The ASF licenses this file
|
|
to you under the Apache License, Version 2.0 (the
|
|
"License"); you may not use this file except in compliance
|
|
with the License. You may obtain a copy of the License at
|
|
|
|
.. http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
.. Unless required by applicable law or agreed to in writing,
|
|
software distributed under the License is distributed on an
|
|
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
KIND, either express or implied. See the License for the
|
|
specific language governing permissions and limitations
|
|
under the License.
|
|
|
|
Manage logs
|
|
===========
|
|
|
|
You have a number of options when it comes to managing your Airflow logs.
|
|
|
|
No persistence
|
|
--------------
|
|
|
|
With this option, Airflow will log locally to each pod. As such, the logs will only be available during the lifetime of the pod.
|
|
|
|
.. code-block:: bash
|
|
|
|
helm upgrade --install airflow apache-airflow/airflow \
|
|
--set logs.persistence.enabled=false
|
|
|
|
.. note::
|
|
|
|
Setting the ``workers.celery.persistence.enabled=false`` is required when ``CeleryExecutor`` is used.
|
|
|
|
Celery worker log persistence
|
|
-----------------------------
|
|
|
|
If you are using ``CeleryExecutor``, workers persist logs by default to a volume claim created with a ``volumeClaimTemplate``.
|
|
|
|
You can modify the template:
|
|
|
|
.. code-block:: bash
|
|
|
|
helm upgrade --install airflow apache-airflow/airflow \
|
|
--set executor=CeleryExecutor \
|
|
--set workers.celery.persistence.size=10Gi
|
|
|
|
.. note::
|
|
|
|
With this option only task logs are persisted, unlike when log persistence is enabled which will also persist scheduler logs.
|
|
|
|
Log persistence enabled
|
|
-----------------------
|
|
|
|
This option will provision a ``PersistentVolumeClaim`` with an access mode of ``ReadWriteMany``. Each component of Airflow will
|
|
then log onto the same volume.
|
|
|
|
Not all volume plugins have support for ``ReadWriteMany`` access mode.
|
|
Refer `Persistent Volume Access Modes <https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes>`__
|
|
for details.
|
|
|
|
.. code-block:: bash
|
|
|
|
helm upgrade --install airflow apache-airflow/airflow \
|
|
--set logs.persistence.enabled=true
|
|
# You can also override the other persistence
|
|
# by setting the logs.persistence.* values
|
|
# Please refer to values.yaml for details
|
|
|
|
Externally provisioned PVC
|
|
--------------------------
|
|
|
|
In this approach, Airflow will log to an existing ``ReadWriteMany`` PVC. You pass in the name of the volume claim to the chart
|
|
by using the ``logs.persistence.existingClaim`` parameter:
|
|
|
|
.. code-block:: bash
|
|
|
|
helm upgrade --install airflow apache-airflow/airflow \
|
|
--set logs.persistence.enabled=true \
|
|
--set logs.persistence.existingClaim=my-volume-claim
|
|
|
|
.. note::
|
|
|
|
The volume has to be writable by the Airflow user. The easiest way to do this is to ensure GID ``0`` has a write permission.
|
|
More information can be found in the :ref:`Docker image entrypoint documentation <docker-stack:arbitrary-docker-user>`.
|
|
|
|
Elasticsearch
|
|
-------------
|
|
|
|
If your cluster forwards logs to Elasticsearch, you can configure Airflow to retrieve task logs from it.
|
|
See the :doc:`Elasticsearch providers guide <apache-airflow-providers-elasticsearch:logging/index>` for more details.
|
|
|
|
.. code-block:: bash
|
|
|
|
helm upgrade --install airflow apache-airflow/airflow \
|
|
--set elasticsearch.enabled=true \
|
|
--set elasticsearch.secretName=my-es-secret
|
|
# Other choices exist. Please refer to values.yaml for details.
|