RunContext.versions is meant to hash each declared input once per run (assets.Versions docstring; materialize.py comment at the RunContext construction). But the context goes to scheduler.submit by value, so every task deserializes its own empty memo. Every task in the closure, current ones included, calls context.versions.of(path) to classify (worker._materialize). An input declared by N outputs is therefore read N times, on both the local (processes=False) and allocation branches.
Repro: submit 4 Dask tasks that each call .of(p) on one shared assets.Versions(). Each task sees a distinct object with an empty memo.
Observed: a 20 GB source: declared by 14 outputs was read about 14 times, so a one-minute figure took 17 min. lc status on the same tree hashes once.
Expected: one hash per declared input per run.
Suggested fix: resolve declared-input digests in the driver before submission and ship them with each task.
lc 0.5.0rc4 (main 3aa823b), distributed 2026.8.0, Linux/Slurm.
RunContext.versionsis meant to hash each declared input once per run (assets.Versionsdocstring;materialize.pycomment at theRunContextconstruction). But the context goes toscheduler.submitby value, so every task deserializes its own empty memo. Every task in the closure, current ones included, callscontext.versions.of(path)to classify (worker._materialize). An input declared by N outputs is therefore read N times, on both the local (processes=False) and allocation branches.Repro: submit 4 Dask tasks that each call
.of(p)on one sharedassets.Versions(). Each task sees a distinct object with an empty memo.Observed: a 20 GB
source:declared by 14 outputs was read about 14 times, so a one-minute figure took 17 min.lc statuson the same tree hashes once.Expected: one hash per declared input per run.
Suggested fix: resolve declared-input digests in the driver before submission and ship them with each task.
lc 0.5.0rc4 (main 3aa823b), distributed 2026.8.0, Linux/Slurm.