Replies: 14 comments 9 replies
-
|
I also just upgraded to the latest version of the "microsft-azure" provider (12.6.1) and still got the same error. provider versions: apache-airflow @ file:///home/build/dist/apache_airflow-3.0.4-py3-none-any.whl |
Beta Was this translation helpful? Give feedback.
-
|
Have been trying to debug it, and for some reason within the " get_connection_from_secrets" function when it is in the "for secrets_backend in ensure_secrets_loaded():" loop, when it gets to the "MetastoreBackend" it is causing an exception... and when I run this same code SSHing into the DAG pod and opening up Python, it works just fine... Will try to look into this further but I am very inexperienced with looking into Airflow code, so if anybody else could help that would be greatly appreciated. |
Beta Was this translation helpful? Give feedback.
-
|
It shouldn't get to the Metastore backend on the worker. The workers don't have access to the metastore directly anymore. What config do you have around secrets backends etc? |
Beta Was this translation helpful? Give feedback.
-
|
Looks like for some reason within the loop below: When in the try block for the metastorebackend it is getting this exception: "Session must be set before!" Not sure why this is not getting set? |
Beta Was this translation helpful? Give feedback.
-
|
It looks like this line "conn = secrets_backend.get_connection(conn_id=conn_id)" is the line that is causing the exception |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I’d like to work on this issue. I plan to investigate the root cause and work on a solution. |
Beta Was this translation helpful? Give feedback.
-
|
I would appreciate that! |
Beta Was this translation helpful? Give feedback.
-
|
UPDATE for anybody having this problem, I switched my connection over to environment variables and it worked:
But, I feel as if this issue should stay open until a fix for the problem at hand has been found |
Beta Was this translation helpful? Give feedback.
-
|
This makes sense, and is not a bug. If you are configuring logging on remote Workers, these Workers do NOT have access to connections defined in the Airflow UI. There are two ways to handle this:
|
Beta Was this translation helpful? Give feedback.
-
|
@jroachgolf84 that does not make any sense, because in Airflow 2.11 it works with the same configuration. So what changed in Airflow 3 that makes this not possible? Why would this functionality be taken away in Airflow 3? |
Beta Was this translation helpful? Give feedback.
-
|
Good question! Airflow 3.0 brings an architectural change that prevents Airflow Workers from communicating directly with the metadata DB. Yes, previously this was functionality that was supported in Airlfow 2.X. However, since Connections are stored in the metadata DB, this functionality is not supported OOB with 3.0. |
Beta Was this translation helpful? Give feedback.
-
|
Ahhh yes that’s right, I knew that too I just didn’t connect it to the logging piece. Thank you! Marking this as the answer. An update to the remote logging docs might be in order with this information for the next person down the line. |
Beta Was this translation helpful? Give feedback.
-
|
Actually I would like to update that a bit. You should be able to retrieve connection directly in the worker, without having to define them via variables. I believe it'a already been fixed before - it did not work in first 3.* versions. But if you use the "task.sdk" to retrieve the connection rather thant trying to use secrets_backends to retrieve it - it should work. Generally workers should never access "airflow" code directly they should only access task_sdk. https://airflow.apache.org/docs/task-sdk/stable/api.html -> you will find that Connection (from task.sdk) hass class method "get" that returns you the connection.
The quesiton is in this case where the "get_connection_from_secrets" is directly called - but it's hard to say, but if the authors bothers a little and tries to format stack trace that is currently convoluted and hard to read - maybe we will be able to find out whether it's the code of author of the issue, or whether this is some provider not yet converted to use task.sdk properly (there is a work in progress to track and fix those, so @ahipp13 if you bother to help a little those who look and maybe get a better stack traces of the erorrs, that would be helpfu. |
Beta Was this translation helpful? Give feedback.
-
|
@potiuk cleaned up stack trace is below. I threw it on DEBUG too just to make sure. If they need to be cleaner, let me know. The rest of my configuration values are above. If anybody needs more info please let me know and I can try to provide it. Thank you. Right now I unmarked as answer as it sounds like some work can be done on this. If not let me know and I can remark as answer. |
Beta Was this translation helpful? Give feedback.
-
|
This should have been fixed by #53719, and it's backport 5523514dbd9326847a6a4c0dca930b8ab52e8bda-- can you please confirm that you are on Airflow 3.0.4 everywhere in your stack, particularly in the workers? |
Beta Was this translation helpful? Give feedback.
-
|
I run the same image, so yes the workers have 3.0.4. This is a pull directly from the worker pod: apache-airflow @ file:///home/build/dist/apache_airflow-3.0.4-py3-none-any.whl |
Beta Was this translation helpful? Give feedback.
-
|
@ahipp13 Can you let us know what if any settings you have relating to secrets backends in your airflow.cfg file or environment vars? |
Beta Was this translation helpful? Give feedback.
-
|
All environment variables I have set are below, as well as the airflow.cfg that appears on the pods. I have never set anything in my helm chart for a secrets backend. NOTE some values are changed/omitted airflow.cfg: |
Beta Was this translation helpful? Give feedback.
-
|
@ahipp13 Okay, yeah my fix was incomplete. See #54679 for the actual fix which will be in 3.0.6. If you are okay applying a patch (and assuming you are using Docker) then this sort of thing should get you unblocked.: USER root
RUN apt update && apt install -y patch patchutils
RUN set -ex; \
cd /usr/local/lib/python3.12/site-packages/airflow; \
curl -L https://patch-diff.githubusercontent.com/raw/apache/airflow/pull/54679.patch \
| filterdiff -p1 -i 'task-sdk/src/airflow/*' | patch -p4 -u --verbose
USER airflow(You'll need to change a few details in that to make it work like paths or users, but hopefully that gives you the details) |
Beta Was this translation helpful? Give feedback.
-
|
I belive this is fixed now in 3.0.5. Could you retest please @ahipp13 ? |
Beta Was this translation helpful? Give feedback.
-
|
I will paste my comment from #37461, maybe it will help some of you with setting this up: If anyone's stumbles upon this issue when trying to make it work, here are my notes.
If you want to configure it via Environment Variable, I recommend this format: bare minimum version: I recommend setting it up as Finally, the configuration itself: A few notes: format, but for version 3.0.6 do Regarding Thanks to everyone who participated in this discussion and other discussion related to this topic: |
Beta Was this translation helpful? Give feedback.
-
|
@ashb do you mind to take a look at #58140 ? Looks like it's a similar issue but with different remote storage. |
Beta Was this translation helpful? Give feedback.
-
Check if credentials work correctly. S3 should be hassle free compared to Azure Blob Storage: |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Apache Airflow Provider(s)
microsoft-azure
Versions of Apache Airflow Providers
apache-airflow @ file:///home/build/dist/apache_airflow-3.0.4-py3-none-any.whl
apache-airflow-core==3.0.4
apache-airflow-providers-amazon @ file:///home/build/providers/amazon/dist/apache_airflow_providers_amazon-9.7.0-py3-none-any.whl
apache-airflow-providers-apache-kafka==1.10.2
apache-airflow-providers-celery @ file:///home/build/providers/celery/dist/apache_airflow_providers_celery-3.10.6-py3-none-any.whl
apache-airflow-providers-cncf-kubernetes @ file:///home/build/providers/cncf/kubernetes/dist/apache_airflow_providers_cncf_kubernetes-10.4.2-py3-none-any.whl
apache-airflow-providers-common-compat==1.7.3
apache-airflow-providers-common-io==1.6.2
apache-airflow-providers-common-sql==1.27.5
apache-airflow-providers-docker @ file:///home/build/providers/docker/dist/apache_airflow_providers_docker-4.3.1-py3-none-any.whl
apache-airflow-providers-elasticsearch @ file:///home/build/providers/elasticsearch/dist/apache_airflow_providers_elasticsearch-6.2.2-py3-none-any.whl
apache-airflow-providers-fab @ file:///home/build/providers/fab/dist/apache_airflow_providers_fab-2.2.0-py3-none-any.whl
apache-airflow-providers-ftp @ file:///home/build/providers/ftp/dist/apache_airflow_providers_ftp-3.12.3-py3-none-any.whl
apache-airflow-providers-google @ file:///home/build/providers/google/dist/apache_airflow_providers_google-15.1.0-py3-none-any.whl
apache-airflow-providers-grpc @ file:///home/build/providers/grpc/dist/apache_airflow_providers_grpc-3.7.3-py3-none-any.whl
apache-airflow-providers-hashicorp @ file:///home/build/providers/hashicorp/dist/apache_airflow_providers_hashicorp-4.1.1-py3-none-any.whl
apache-airflow-providers-http @ file:///home/build/providers/http/dist/apache_airflow_providers_http-5.2.2-py3-none-any.whl
apache-airflow-providers-imap @ file:///home/build/providers/imap/dist/apache_airflow_providers_imap-3.8.3-py3-none-any.whl
apache-airflow-providers-microsoft-azure @ file:///home/build/providers/microsoft/azure/dist/apache_airflow_providers_microsoft_azure-12.3.1-py3-none-any.whl
apache-airflow-providers-mysql @ file:///home/build/providers/mysql/dist/apache_airflow_providers_mysql-6.2.1-py3-none-any.whl
apache-airflow-providers-odbc @ file:///home/build/providers/odbc/dist/apache_airflow_providers_odbc-4.9.2-py3-none-any.whl
apache-airflow-providers-openlineage @ file:///home/build/providers/openlineage/dist/apache_airflow_providers_openlineage-2.1.3-py3-none-any.whl
apache-airflow-providers-postgres @ file:///home/build/providers/postgres/dist/apache_airflow_providers_postgres-6.1.3-py3-none-any.whl
apache-airflow-providers-redis @ file:///home/build/providers/redis/dist/apache_airflow_providers_redis-4.0.2-py3-none-any.whl
apache-airflow-providers-sendgrid @ file:///home/build/providers/sendgrid/dist/apache_airflow_providers_sendgrid-4.0.1-py3-none-any.whl
apache-airflow-providers-sftp @ file:///home/build/providers/sftp/dist/apache_airflow_providers_sftp-5.2.0-py3-none-any.whl
apache-airflow-providers-slack @ file:///home/build/providers/slack/dist/apache_airflow_providers_slack-9.0.4-py3-none-any.whl
apache-airflow-providers-smtp==2.2.0
apache-airflow-providers-snowflake==6.5.2
apache-airflow-providers-sqlite @ file:///home/build/providers/sqlite/dist/apache_airflow_providers_sqlite-4.0.2-py3-none-any.whl
apache-airflow-providers-ssh @ file:///home/build/providers/ssh/dist/apache_airflow_providers_ssh-4.0.1-py3-none-any.whl
apache-airflow-providers-standard==1.5.0
apache-airflow-task-sdk==1.0.4
google-cloud-orchestration-airflow==1.17.5
Apache Airflow version
3.0.4
Operating System
bash-5.3$ cat /etc/os-release ID=chainguard NAME="Chainguard" PRETTY_NAME="Chainguard" VERSION_ID="20230214" HOME_URL="https://chainguard.dev/"
Deployment
Official Apache Airflow Helm Chart
Deployment details
Deploying helm chart on to Kubernetes version v1.32.4-gke.200. For remote logging I have these environment variables set:
And I have this connection created:
What happened
Whenever I run one of my DAGs, it will complete successfully, but then when it tries to write the logs it will give an error that says "The conn_id
edw_az_connisn't defined" when it clearly is as you can see from the screenshots above. Full log belowWhat you think should happen instead
It should log to the blob storage like normal. I have this same configuration running in Airflow 2.11
How to reproduce
Would need to set up Azure remote logging with a storage account and use the airflow provider.
Anything else
I brought up this issue here: #54192
I was told upgrading to 3.0.4 would fix it, but it is still occurring to me. Another note, I know it is not the connection that is broken, because I can read logs in the blob storage before I upgraded the environment...
Are you willing to submit PR?
Code of Conduct
Beta Was this translation helpful? Give feedback.
All reactions