Repository navigation
[df] Rework RNTuple MT scheduling with a split work-stealing approach - #23327
Open
vepadulano wants to merge 1 commit into
Open
vepadulano wants to merge 1 commit into
vepadulano wants to merge 1 commit into
Conversation
Test Results 23 files 23 suites 3d 19h 0m 0s ⏱️ For more details on these failures, see this check. Results for commit 167b27a. ♻️ This comment has been updated with latest results. |
hageboeck
reviewed
Sep 11, 2026
hageboeck
left a comment
hageboeck
left a comment
Member
There was a problem hiding this comment.
🚀
Here's a first pass with a few suggestions and a few questions.
jblomer
reviewed
Oct 4, 2026
jblomer
left a comment
jblomer
left a comment
Contributor
There was a problem hiding this comment.
Thank you for this work!
I think it may be easier to follow the code if the lambdas became class methods. I would also make documentation easier. Generally, I think a high-level documentation of the scheduling approach would be helpful.
Regarding the use of TBB: would it be possible/easier to simply use a TTaskGroup?
vepadulano
force-pushed
the
df-rntuple-imt-work-stealing
branch
from
October 19, 2026 08:54
ece30c1 to
06ccada
Compare
vepadulano
marked this pull request as ready for review
October 19, 2026 08:58
Member
Author
|
HI @hageboeck @jblomer , I have addressed your PR reviews and updated the changes. Documentation has been added throughout, and I've also memorialised the strategy in a separate markdown document as requested by @jblomer . Let me know your thoughts, thanks! |
vepadulano
force-pushed
the
df-rntuple-imt-work-stealing
branch
from
October 19, 2026 09:27
06ccada to
9acbfde
Compare
silverweed
reviewed
Oct 19, 2026
vepadulano
force-pushed
the
df-rntuple-imt-work-stealing
branch
2 times, most recently
from
October 19, 2026 18:06
da362e3 to
5c79602
Compare
pcanal
reviewed
Oct 19, 2026
pcanal
reviewed
Oct 19, 2026
pcanal
reviewed
Oct 19, 2026
pcanal
reviewed
Oct 19, 2026
pcanal
reviewed
Oct 19, 2026
The MT scheduling of RNTupleDS is changed. The data source adopts now a similar pattern as TTreeDS, implementing a ProcessMT(ROOT::Detail::RDF::RLoopManager &lm) method. This allows the data source to manage the creation and scheduling of multithreaded tasks, inside of which the loop manager is invoked to apply the computation graph to the range of entries in the task. A brief summary of the scheduling approach follows: 1. One task is launched in parallel per input file 2. At the beginning of the file-based task, a page source is created and the list of clusters is computed a. If the user has not specified a global entry range, it is the actual list of clusters of the page source. This happens in ScheduleTasksForFile. b. If the user has specified a global entry range, before launching the file-based tasks the data source had computed a full list of global entry offsets across all input files. The list of clusters of the page source is then adjusted according to the position of the global entry range w.r.t. the global offsets of this file. This happens in ScheduleTasksForFileWithGlobalOffset. 3. The task goes on and calls an internal function to invoke TBB's parallel_for with two arguments, one respecting the Range and the other respecting the Body requirements. a. The Range argument is of type RPageSourceSplittableRange, a struct representing a task with a single page source and an assigned list of clusters to process. In the beginning, the full list of clusters as computed in step 2 is assigned to the Range. b. While processing the task, TBB may call the Range object's SplitWork method, which will split the cluster range in roughly two equal parts, then create a new Range with a clone of the page source and the split cluster range. c. The Body argument calls into ParallelForBody with the page source and the assigned list of clusters of the Range argument, finally applying the computation graph to it by calling RLoopManager::RNTupleThreadTask. This approach delegates to the scheduling engine (TBB in this case) the responsibility to optimise the task processing and split tasks that may be running for too long (e.g. some files may contain many more entries than others). The strategy is also documented in a markdown file for posterity.
vepadulano
force-pushed
the
df-rntuple-imt-work-stealing
branch
from
October 19, 2026 22:57
5c79602 to
167b27a
Compare
jblomer
reviewed
Oct 20, 2026
jblomer
left a comment
jblomer
left a comment
Contributor
There was a problem hiding this comment.
Thanks!
Some first comments on the documentation. I'll review the code later today.
|
|
||
| RNTuple data reading is handled in `RNTupleDS`, an `RDataSource`-derived class. The reading is triggered by the | ||
| computation graph being run, and nodes requesting data from the data source. The computation graph is started and | ||
| managed by an `RLoopManager`, requesting data from the `RNTupleDS` until exhaustion. This is a description of the |
Contributor
There was a problem hiding this comment.
Suggested change
| managed by an `RLoopManager`, requesting data from the `RNTupleDS` until exhaustion. This is a description of the | |
| managed by an `RLoopManager`, requesting data from the `RNTupleDS` until done. This is a description of the |
| b. `RNTupleDS::ScheduleTasksForFileWithGlobalOffset` will schedule a task with an adjusted list of clusters. The | ||
| user-provided global entry range is matched against the global offsets computed by 2c, and a new list of | ||
| clusters is generated to restrict processing of this page source according to the match. Consequently, some | ||
| tasks may be stopped early if the page source boundaries do not overlap with the global entry range. |
Contributor
There was a problem hiding this comment.
Do you mean: you open the page source and close it right away because there are effectively no entries to process? Then I'd write:
Suggested change
| tasks may be stopped early if the page source boundaries do not overlap with the global entry range. | |
| tasks may be stopped immediately if the page source boundaries do not overlap with the global entry range. |
| b. During the computation graph run, TBB may call `RPageSourceSplittableRange::SplitWork`, which will split the | ||
| cluster range in roughly two equal parts, re-assigning the first part to the current task and creating a new task | ||
| with the second part and a clone of the page source. This is the core of the strategy: we cannot know a priori | ||
| if one of the input files will be much larger than the others, or if one thread may finish earlier than others. |
Contributor
There was a problem hiding this comment.
Suggested change
| if one of the input files will be much larger than the others, or if one thread may finish earlier than others. | |
| if one of the input files will be much larger than the others, or if one thread finishes earlier than others. |
| For this reason, we allow tasks to split themselves, so that idle threads can steal work from existing, | ||
| longer-running tasks. The criteria for the splitting is decided by `RPageSourceSplittableRange::IsDivisible`, | ||
| which currently dictates that a task can split itself only if it has three or more clusters to process. The | ||
| reason is that the page source will anyway prefetch the next cluster while processing the current one. While TBB |
Contributor
There was a problem hiding this comment.
Suggested change
| reason is that the page source will anyway prefetch the next cluster while processing the current one. While TBB | |
| reason for requiring at least three clusters is that the page source will prefetch the next cluster while processing the current one. While TBB |
| clusters is generated to restrict processing of this page source according to the match. Consequently, some | ||
| tasks may be stopped early if the page source boundaries do not overlap with the global entry range. | ||
| 4. Both file-based task implementations described above eventually call `RNTupleDS::ScheduleTaskForPageSource`. This | ||
| method is going to call the TBB parallel_for overload that takes two arguments, one respecting the Range and the other respecting the Body requirements. Note that the parallel_for call is hidden by |
Contributor
There was a problem hiding this comment.
What are the requirements of range and body? I understand it takes a range and the task function, I'm just confused by the word "requirements".
|
|
||
| ## Multi-threading | ||
|
|
||
| ### RNTuple |
Contributor
There was a problem hiding this comment.
Can you add here a description of the mechanism that keeps the number of concurrently opened files limited? Likewise, we should have a test that opens several thousands files in IMT mode.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Alternative to #22631 , implementing native IMT work-stealing for the RNTuple data source