Metric dag_processing.total_parse_time is exposed incorrectly

[KulykDmytro]

Apache Airflow version

2.5.1

What happened

Following metric dag_processing.total_parse_time exposing smth not relevant to overall processing time

DAG File Processing Stats                                                                                                     
                                                                                                                              
File Path                                            PID  Runtime      # DAGs    # Errors  Last Runtime    Last Run           
-------------------------------------------------  -----  ---------  --------  ----------  --------------  -------------------
/opt/airflow/dags/player_time_features.py                                   1           0  0.89s           2023-02-24T09:18:59
/opt/airflow/dags/maxmind_uploader.py                                       1           0  0.81s           2023-02-24T09:19:29
/opt/airflow/dags/player_team_similarities.py                               1           0  1.34s           2023-02-24T09:18:31
/opt/airflow/dags/aws_sanitize.py                                           1           0  3.58s           2023-02-24T09:18:58
/opt/airflow/dags/ad_backoffice.py                                          1           0  1.05s           2023-02-24T09:19:28
/opt/airflow/dags/gdpr.py                                                   1           0  0.46s           2023-02-24T09:18:39
/opt/airflow/dags/aws_regional_split_backfill.py                            1           0  2.55s           2023-02-24T09:19:12
/opt/airflow/dags/reconciliation_daily.py                                   1           0  1.04s           2023-02-24T09:19:16
/opt/airflow/dags/maintenance.py                                            1           0  0.24s           2023-02-24T09:17:51
/opt/airflow/dags/athena_repair_partitions.py                               1           0  0.81s           2023-02-24T09:19:28
/opt/airflow/dags/trending_events.py                                        1           0  1.74s           2023-02-24T09:19:07
/opt/airflow/dags/casino_games_similarities.py                              1           0  1.17s           2023-02-24T09:18:18
/opt/airflow/dags/player_segmentation.py                                    1           0  1.44s           2023-02-24T09:17:41
/opt/airflow/dags/test_copy.py                                              1           0  0.71s           2023-02-24T09:19:01
/opt/airflow/dags/datahub_ingest.py                                         1           0  0.90s           2023-02-24T09:19:05
/opt/airflow/dags/gs_exporter.py                                            1           0  2.65s           2023-02-24T09:17:37
/opt/airflow/dags/canary.py                                                 1           0  1.75s           2023-02-24T09:19:04
/opt/airflow/dags/sfsc_load.py                                              1           0  1.06s           2023-02-24T09:19:15
/opt/airflow/dags/led_recognition_processed.py                              1           0  2.28s           2023-02-24T09:19:09
/opt/airflow/dags/casino_games_recommendations.py                           1           0  1.77s           2023-02-24T09:18:59
/opt/airflow/dags/vip_export.py                                             1           0  1.29s           2023-02-24T09:18:29
/opt/airflow/dags/autotest.py                                               2           0  0.63s           2023-02-24T09:19:30
/opt/airflow/dags/anomaly_detection.py                                      2           0  3.67s           2023-02-24T09:17:37
/opt/airflow/dags/native_rls.py                                             3           0  1.14s           2023-02-24T09:19:26
/opt/airflow/dags/dbt.py                                                    4           0  1.23s           2023-02-24T09:18:14
/opt/airflow/dags/front_data.py                                             4           0  8.55s           2023-02-24T09:18:10
/opt/airflow/dags/gs_loader.py                                              4           0  1.37s           2023-02-24T09:19:25
/opt/airflow/dags/backfill_lambda.py                                        6           0  1.91s           2023-02-24T09:19:14
/opt/airflow/dags/tableau_refresher.py                                      7           0  1.85s           2023-02-24T09:17:39
/opt/airflow/dags/extract_load.py                                           8           0  1.30s           2023-02-24T09:18:49
/opt/airflow/dags/validator.py                                             30           0  1.91s           2023-02-24T09:19:24
/opt/airflow/dags/aws_transforms.py                 1768  34.15s           56           0  151.91s         2023-02-24T09:17:01

history of file parse - it is volatile

What you think should happen instead

No response

How to reproduce

process dags by dag-processor
observe exposed metric: dag_processing.total_parse_time

Operating System

n/a

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

AWS EKS

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[vandonr-amz]

I made a change recently (#30290) to edit the doc about this metric. The number of dag (files) parsed can vary from one run to the next, explaining (some of) the volatility.
You could try dividing one metric by the other, but since they are not emitted at the same time, it can still yield to bad results.
Also, if you have one dag that is slow to parse and others that are fast, the metric will still be janky because there will be spikes when the long dag is picked up.

[potiuk]

I think this is something to be addressed with the Open-Telemetry implementation @feruzzi @howardyoo - possibly dropping or replacing this metric with some aggregated one.

I think we should not keep it as an issue, now when it is documented, I doubt we will do anything with it but it should be looked at with the OTEL work.

converting to a discussion.