Unable to read remote logs with celery executor #32063
Replies: 3 comments 1 reply
-
|
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval. |
Beta Was this translation helpful? Give feedback.
-
|
This might not be related to remote logging since the task fails immediately after being sent to the worker. I believe there could be an issue with the parsing of DAG files or the availability of these files in the workers' pods/containers/processes. Have you checked the workers' logs? I convert the issue to discussion, and we can reopen it if needed. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I have verified the worker logs sometimes Its working and sometimes its not working. Not able to find the root cause. |
Beta Was this translation helpful? Give feedback.
-
|
I have attached the worker logs over here. started celery worker in Airflow VM as Systemd. [Unit] [Service] [Install] ● celery-worker.service - Airflow Celery Flower UI Jun 22 05:19:12 airflow-dds-dev2 airflow[3040]: [2023-06-22 05:19:12,911: INFO/ForkPoolWorker-16] Task airflow.executors.celery_executor.execute_command[8fc7be32-3a66-4e23-bc87-ca9945c4d541] succeeded in 132.22622969000076s: None |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Apache Airflow version
Other Airflow 2 version (please specify below)
What happened
We migrated from Local Executor to Celery Executor where using RabbitMQ as a broker.
With Local executor, remote logging is working as expected. Able to read logs
Airflow Version: 2.6.1
Celery Version: 5.3.0
Airflow Config for Remote Logging:
base_log_folder = /opt/airflow/logs remote_logging = True remote_log_conn_id = google_cloud_default delete_local_logs = True google_key_path = remote_base_log_folder = gs://bucket-name/logs remote_task_handler_kwargs = encrypt_s3_logs = False logging_level = INFO celery_logging_level = fab_logging_level = WARNING logging_config_class = colored_console_log = False colored_log_format = [%(blue)s%(asctime)s%(reset)s] {%(blue)s%(filename)s:%(reset)s%(lineno)d} %(log_color)s%(levelname)s%(reset)s - %(log_color)s%(message)s%(reset)s colored_formatter_class = airflow.utils.log.colored_log.CustomTTYColoredFormatter log_format = [%(asctime)s] {%(filename)s:%(lineno)d} %(levelname)s - %(message)s simple_log_format = %(asctime)s %(levelname)s - %(message)s dag_processor_log_target = file dag_processor_log_format = [%(asctime)s] [SOURCE:DAG_PROCESSOR] {%(filename)s:%(lineno)d} %(levelname)s - %(message)s log_formatter_class = airflow.utils.log.timezone_aware.TimezoneAware secret_mask_adapter = task_log_prefix_template = log_filename_template = dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index }}/{% endif %}attempt={{ try_number }}.log log_processor_filename_template = {{ filename }}.log dag_processor_manager_log_location = /opt/airflow/logs/dag_processor_manager/dag_processor_manager.log task_log_reader = task extra_logger_names = worker_log_server_port = 8793 trigger_log_server_port = 8794 file_task_handler_new_folder_permissions = 0o775 file_task_handler_new_file_permissions = 0o664when we trigger dags, dags are going into queue state and then failed state by stating unable to read the remote logs like below error.
Celery Config:
Accessing Airflow webserver and Flower UI using NGINX reverse proxy
We have verified the object path manually, Its existing. But from airflow we are getting error like above.
We have verified all the permissions and config.
Can you please help us to resolve.
What you think should happen instead
I expected I could read the task log on UI
How to reproduce
While I run any DAGs, the log always shows up this.
The log files have already exists on worker and store on GCS.
Operating System
RHEL 7
Versions of Apache Airflow Providers
No response
Deployment
Virtualenv installation
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
Code of Conduct
Beta Was this translation helpful? Give feedback.
All reactions