Unable to read remote logs with celery executor where airflow version is 2.6.1

[lohithasree]

We migrated from Local Executor to Celery Executor where using RabbitMQ as a broker.

With Local executor, remote logging is working as expected. Able to read logs

Airflow Version: 2.6.1
Celery Version: 5.3.0

apache-airflow-providers-common-sql==1.1.0
apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-google==8.3.0
apache-airflow-providers-http==4.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-postgres==5.2.0
apache-airflow-providers-celery==3.2.0

Airflow Config for Remote Logging:

base_log_folder = /opt/airflow/logs
remote_logging = True
remote_log_conn_id = google_cloud_default
delete_local_logs = True
google_key_path =
remote_base_log_folder = gs://bucket-name/logs
remote_task_handler_kwargs =
encrypt_s3_logs = False
logging_level = INFO
celery_logging_level =
fab_logging_level = WARNING
logging_config_class =
colored_console_log = False
colored_log_format = [%(blue)s%(asctime)s%(reset)s] {%(blue)s%(filename)s:%(reset)s%(lineno)d} %(log_color)s%(levelname)s%(reset)s - %(log_color)s%(message)s%(reset)s
colored_formatter_class = airflow.utils.log.colored_log.CustomTTYColoredFormatter
log_format = [%(asctime)s] {%(filename)s:%(lineno)d} %(levelname)s - %(message)s
simple_log_format = %(asctime)s %(levelname)s - %(message)s
dag_processor_log_target = file
dag_processor_log_format = [%(asctime)s] [SOURCE:DAG_PROCESSOR] {%(filename)s:%(lineno)d} %(levelname)s - %(message)s
log_formatter_class = airflow.utils.log.timezone_aware.TimezoneAware
secret_mask_adapter =
task_log_prefix_template =
log_filename_template = dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index }}/{% endif %}attempt={{ try_number }}.log
log_processor_filename_template = {{ filename }}.log
dag_processor_manager_log_location = /opt/airflow/logs/dag_processor_manager/dag_processor_manager.log
task_log_reader = task
extra_logger_names =
worker_log_server_port = 8793
trigger_log_server_port = 8794
file_task_handler_new_folder_permissions = 0o775
file_task_handler_new_file_permissions = 0o664

when we trigger dags, dags are going into queue state and then failed state by stating unable to read the remote logs like below error.

Celery Config:

celery_app_name = airflow.executors.celery_executor
worker_concurrency = 16
worker_prefetch_multiplier = 1
worker_enable_remote_control = True
broker_url = amqps://username:[email protected]:5671/host_name;amqps://username:[email protected]:5671/host_name;amqps://username:[email protected]:5671/host_name
flower_host = 0.0.0.0
flower_url_prefix = flower
flower_port = 5555
flower_basic_auth = QW3Q3uWPJqloOhEWv:pvmGvSATFxBMuD4El
sync_parallelism = 0
celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
ssl_active = True
ssl_key = /opt/airflow/cert/ddsdev-airflow.uk.hsbc.key
ssl_cert = /opt/airflow/cert/ddsdev-airflow.uk.hsbc.crt
ssl_cacert =
pool = prefork
operation_timeout = 1.0
task_track_started = True
task_publish_max_retries = 3
worker_precheck = False
result_backend = db+postgresql://username:password@localhost:5432/database

Accessing Airflow webserver and Flower UI using NGINX reverse proxy

***** Unable to read remote log from gs://**********/logs/dag_id=DDS_catalog_update/run_id=manual__2023-06-20T15:22:58.216561+00:00/task_id=insert_dds_ct_start_ts/attempt=1.log *** 404 GET https://storage.googleapis.com/download/storage/v1/b/*********/o/logs%2Fdag_id%3DDDS_catalog_update%2Frun_id%3Dmanual__2023-06-20T15%3A22%3A58.216561%2B00%3A00%2Ftask_id%3Dinsert_dds_ct_start_ts%2Fattempt%3D1.log?alt=media: No such object: *******88/logs/dag_id=DDS_catalog_update/run_id=manual__2023-06-20T15:22:58.216561+00:00/task_id=insert_dds_ct_start_ts/attempt=1.log: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>) *** Could not read served logs: Request URL is missing an 'http://' or 'https://' protocol.

We have verified the object path manually, Its existing. But from airflow we are getting error like above.

We have verified all the permissions and config.

Can you please help us to resolve.