Allow Airflow to run in FIPs enabled environment

[longslvr]

Currently, we are running a forked Airflow 1.10.15 with additional changes to make Airflow runnable in FIPs enabled production environment.

With fips enabled you can't use md5 hash. Related to the issues mentioned here. #14966

This is proven to be very problematic whenever we want to upgrade the airflow version and we would have to reapply the changes with a fresh fork of Airflow.

To avoid the hassle of forking every time we want to upgrade airflow, we can submit a PR to apply the changes we make so that Airflow is runnable in FIPs-enabled environments. Let me know if anyone has any thoughts.

[potiuk]

I believe (please correct me if I am wrong, pointing to specific FIPS requirements), FIPS mandates not using some algorithms if they are used for cryptography, but is does not tell you that it is forbidden to use certain algorithms if they are not used for security.

Even python 3.9 introduced a flag "usedforsecurity" when hashlib is used:

https://docs.python.org/3/library/hashlib.html

Changed in version 3.9: All hashlib constructors take a keyword-only argument usedforsecurity with default value True. A false value 
allows the use of insecure and blocked hashing algorithms in restricted environments. False indicates that the hashing algorithm is not 
used in a security context, e.g. as a non-cryptographic one-way compression function.

I think md5 in general is so universally used for many non-security context, that getting rid of it completely is a bad idea.

So there is nothing wrong to use md5 hash if it is not used in security context. I believe in any kind of certification you can mark certain usages of certain algorithms for security.

We are still Airlfow 3.7 and there is no "useforsecurity" and also we are using md5 probably in some javascript. I am not aware of any "security" context where we would use it, however if you are using it for security, then yeah, it should be changed. But I see no reason why we should change the algorithm if we are not using it in security context.

In this context, I would be very open (but of course it should be revidwed by others) if you provide a PR (or several of them), but I tihnk it should be not straight replacing md5 use but make it more "pluggable" and prepared for "useforsecurity" parameter once we go Python 3.9+.

I think the sequence/scope of implementation shoudl be:

• inventory of all usages and extracting them to a common "util" library
• adding "useforsecurity" parameter in our util, anticipating that we will replace the util in Python 3.9+ directly (and we should actually pass this parameter to hashlib when we are on 3.9+. Likely similar approach for javascript.
• if you have some specific requirements of not using md5 at all, you could add an option to use other algorithms instead in this "util" of ours. I understand there might be some hardware/environment limits (which useforsecurity and 3.9+ should generally address) but maybe it's not perfect
• the behaviour should be controlled by a configuration parameter on a global level if added (security/fips_compliance).

I think if you do it in fully backwards-compatible way so that nothing changes for the users, this should be possible.

[longslvr]

@potiuk you are correct about the FIPS requirement. Our change is mainly to use "useforsecurity=False" to still use MD5 and introduce a CACHE_OPTIONS in flask_caching to use SHA256 instead of MD5.
Thank you for the suggestion on the changes and I think that is a great suggestion! We used a force repository so we can enforce airflow is running on Python 3.9. I will derive a solution that is backward compatible with all Airflow V2 versions and python from 3.7 and up!

[potiuk]

Fantastic! Looking forward to the PR!

[ngearhart]

Any progress on this? I am interested in running in a FIPS-enabled environment.

[ngearhart]

Actually, I was able to get my production environment running with the following modifications:

• Flask caching - add option to use hashlib.sha256 instead of the default md5
• Airflow scheduler - find and replace all hashlib.md5 calls with hashlib.sha256

Obviously, this may not work in every environment, and I have not tested it thoroughly outside of my own environment. Use at your own risk.

You can quickly accomplish this by running:
sed -i "s/cache_config = {'CACHE_TYPE': 'flask_caching.backends.filesystem', 'CACHE_DIR': gettempdir()}/import hashlib\n cache_config = {'CACHE_TYPE': 'flask_caching.backends.filesystem', 'CACHE_DIR': gettempdir(), 'CACHE_OPTIONS': {'hash_method': hashlib.sha256}}/g" ./venv/lib/python3.8/site-packages/airflow/www/app.py \ && find ./venv/lib/python3.8/site-packages/airflow -type f -exec sed -i "s/hashlib.md5/hashlib.sha256/g" {} \;
(if you have Airflow installed at ./venv).

[r0ck37]

latest versions use flask-2.2.5 which still use sha1 for digest.
Flask-3.0.3 has a fix.
pallets/flask#5448

[potiuk]

Airflow 3 is going to get rid of Flask Application Builder as dependency. Currently Even 5.0* alpha/rc releases have flask<3. If you would like to help with Airflow 3 @r0ck37 development , that might be a good idea.

[andrii-korotkov-verkada]

Is getting rid of this dependency still a plan? Do you need help?

[potiuk]

Sure. We need to implement Auth Manager providing the same capabilities - KeyCloak likely is the choice. We might not be able to have a fully Fledged implementation to replace what FAB providers there (i.e. integration with Oauth and others), which means that Airflow 3 without Flask might not be able to provide sophisticated mechanisms of authentication - LDAP/OAUTH and the like.

So yes. Developing a KeyCloak (or similar) Auth manager that will fully replace FAB is a help that is most welcome - more details in
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311627974#AIP79:RemoveFlaskAppBuilderasCoredependency-Why? (see the Why? section) and @vincbeck @o-nikolas have been running an evaluation earlier to see if it is feasible. It is, and possible, but we need someone who is enterprise-level focused and already needs those enterprise-level authentication to implement the AuthManager.

I am sure if you volunteer for that @o-nikolas and @vincbeck might provide some leads.

[o-nikolas]

We still think Keycloak is a great option. If someone has domain specific knowledge with it or authn/authz in general, we'd be more than happy to collaborate on it!

[andrii-korotkov-verkada]

Would Airflow 3 be completely FIPS-compliant? What needs to be done for this to happen?

[potiuk]

Would Airflow 3 be completely FIPS-compliant? What needs to be done for this to happen?

You tell us what you miss

[andrii-korotkov-verkada]

Okay, will work on it

[longslvr]

Airflow should already be FIPs compliance if you change the encryption suite to one of the allowed suites (SHA256 instead of MD5). If my completely compliance meaning it SHA256 should be the default instead of MD5?

…

On Wed, Nov 27, 2024, 9:11 PM Andrii Korotkov ***@***.***> wrote: Okay, will work on it — Reply to this email directly, view it on GitHub <#25625 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF3GPEWS4SY25NGD7UHZVNT2CZ3URAVCNFSM6AAAAABMU3BV5GVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNBQGEZDSNA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[andrii-korotkov-verkada]

Yeah, I guess we may need SHA256 as default, but need to confirm if that's enough.

[potiuk]

ALso some dependencies of Airflow might still be problematic (see all the discussion above). It also depends on how many providers and which you install).

[longslvr]

We have been running airflow in FIPs environment before I made the change in Airflow repo (we used to have a private fork to apply our changes to be able to run in FIPs enabled AWS cloud) Since 2.7.x, Airflow is able to run in FIPs environment with the suite change.

…

On Wed, Nov 27, 2024, 9:21 PM Andrii Korotkov ***@***.***> wrote: Yeah, I guess we may need SHA256 as default, but need to confirm if that's enough. — Reply to this email directly, view it on GitHub <#25625 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF3GPET7BDACGWQMMFU3XAD2CZ42ZAVCNFSM6AAAAABMU3BV5GVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNBQGEZTIOA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ketozhang]

The official Airflow container image isn't as FIPs-friendly since md5 is used in apt update step. Currently this affects Debian bookworm and its bug 1014517. The Airflow base image is using Debian bookworm.

Other than waiting for Debian to fix this (currently an if not when), Airflow can provide another variant of the image based off Debian bullseye

[potiuk]

You can also build your own images. Airflow images are "reference" images - and if you have more security expectations, you can rebuild your image from scratch - there are some patches in the thread you noted. Also likely upgrading airflow to newer version is a good idea, we take the latest base images when we build ours and likely they contain more fixes

More information on why we are fixing the image in release time is here: https://airflow.apache.org/docs/docker-stack/index.html and it contains links and explanation on what to do - including how to build the image when you have "more" expectations than what the image fixed at the time of release of Airflow providers (so for example you can absolutely take the latest bookwork images from debian, build them as base images if that helps with FIPS compliance.

You have to remember that Airflow image is a "convenience" thing only from the open-source team - volunteers - and people who have FIPS expectation, likely is part of a commercial company that FIPS expectations are intrinsicly connected with - so they have paid staff who can do things a as part of their employment, and when you get a "free software" that comes "without guarantees" it means that sometimes you need to do some work to fullfill certain requirements, rather than expect non-paid volunteers will do it.

[ketozhang]

Yeah I think that's very fair and reasonable expectation. I meant to beg the question if contributors can add an extra build variant; I can imagine it's not something a random contributor can do with a PR.

Unfortunately, I'm not in one of those fortunate commercial companies with the resources to do this. So, we ask the community before presenting "if we wanna use this, we're going to have to put more work in it" to our team.

[jonathan-morales-sp]

How can you override the use of MD5 hashing in airflow/models/dagbag.py (version 2.11.0)?

[potiuk]

I think you can't. But contributions are welcome to change it to use different algorithm if you would like to make it happen.