Best Way to Track Step Execution Like Airflow Tasks in a Python Data Workflow

[guocity]

In Airflow 3, I want to process a large DataFrame entirely in memory but still track each step like separate Airflow tasks. When I split it into multiple tasks, each runs in a separate process, so the DataFrame can’t stay in memory. How can I track each step as tasks while keeping everything in the same process?

[potiuk]

You can't now. This has been discussed that in the future we might want to serve this better (and we already started to discuss add Arrow ADBC provider. So far we hsmeave not seen anyone wanting this feature - usually Airflow is orchestrating external systems and data processing itself by Airflow task has not been the main focus. Some of the workflows requireing multiple tasks sharing memory were done with - for example - Ray (Astronomer has Ray provider https://github.com/astronomer/astro-provider-ray).

But yes, seeing a "need" is a signal that yeah, maybe we should implement something like that.

[guocity]

Thank you, good to know. I am just starting to use Airflow. It’s a great tool.