possible race condition

[aftersnow]

Now inflightKey is volumeName/mountID, suppose:

(1) Pod A request Model X, inflightKey = "pull-volumeA/mountA"
(2) Pod B request Model X, inflightKey = "pull-volumeB/mountB"

which means same model X can be dowloaded concurrently, causes possible race condition.

[imeoer]

Thanks for identifying this issue, it can indeed occur when concurrently pulling the same model on the same node.

Our initial design uses volumeName/mountID as the inflightKey is to prevent frequent pull requests from same PVC (in controller mode) or pod (in inline/dynamic mode).

If we improve it by using reference as the inflightKey, two cases need consideration:

1. With dragonfly hardlink: upgrade singleinflight to a key-based mutex so puller.Pull runs sequentially, allowing each volumeName/mountID to create its own model directory.

2. Without dragonfly hardlink: hard-link files from the previous model directory to the new one.

This adds a bit of complexity to the logic, but it's still achievable to handle scenario where the same node might concurrently pull the same model.

[rishi-jat]

Hey @aftersnow , I would like to work on this issue can you please assign this issue to me ? Thanks!

[imeoer]

@rishi-jat Cool! assigned. :)

[aftersnow]

Hi @rishi-jat, are you still working on this issue?

There is now an active PR for the same duplicate-pull problem: #34. It implements digest-based cross-volume pull deduplication. Please let us know if you still plan to continue with this issue, or whether we should use #34 as the current implementation path.

[rishi-jat]

@imeoer @aftersnow , I have open the PR to fix the issue PTAL when you get a chance. thanks