Repository navigation
fix(dataset): materialize routing tag column via batched map instead of add_column - #10607
Open
zoechen281 wants to merge 1 commit into
Open
zoechen281 wants to merge 1 commit into
zoechen281 wants to merge 1 commit into
Conversation
…of add_column _inject_dataset_routing_tag used Dataset.add_column(), which creates an in-memory column over a memory-mapped table. Pickling a slice of that mixed layout re-carries the shared parent buffers, so each shard sent to a multiprocess map worker is inflated by orders of magnitude (e.g. a 32-row shard: 1.2 KB -> 105 KB; larger runs show 3.7 KB -> 3.3 MB). Write the constant routing column through a batched map instead so it is materialized to disk together with the other columns. The write is wrapped in safe_ddp_context so ranks serialize the fingerprint-derived cache file the same way dataset loading already does, preserving the multi-rank race fix from modelscope#10121. Fixes modelscope#10543 Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Generated with [Devin](https://devin.ai)
zoechen281
force-pushed
the
codex/routing-tag-disk
branch
from
November 20, 2026 14:43
1b4a34f to
205fc8d
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR type
PR information
Fixes #10543
_inject_dataset_routing_taginjects the constantdatasetrouting column viaDataset.add_column(), which creates an in-memory column over a memory-mapped table. When a multiprocessmappickles a shard of that mixed layout, the slice's shared parent buffers are re-carried — a 32-row shard grows from ~1.2 KB to ~105 KB (the issue reports 3.7 KB → 3.3 MB on larger inputs, and 368 MB for a 100-row shard).This PR materializes the column through a batched
mapinstead, so the routing column is written to disk together with the other columns and shard pickling stays near baseline. The write runs insidesafe_ddp_context— the same rank-serialization patternload_datasetalready uses — so the fingerprint-derived cache file keeps the multi-rank behavior fixed in #10121 (flatten_indicesis still never called; the existing routing-tag tests pass unchanged). Datasets with zero rows or no columns keep theadd_columnfallback, sincemapskips its callback entirely there and the column would never be created.AI assistance was used in preparing this PR.
Experiment results
Repro (datasets 5.0.1, pyarrow 25.0.1, 16 disk chunks × 32 rows, shard = first 32 rows):
add_column()routing tagTests run:
Checklist