Skip to content

fix(dataset): materialize routing tag column via batched map instead of add_column - #10607

Open
zoechen281 wants to merge 1 commit into
modelscope:mainfrom
zoechen281:codex/routing-tag-disk
Open

zoechen281 wants to merge 1 commit into
modelscope:mainfrom
zoechen281:codex/routing-tag-disk

Conversation

@zoechen281

@zoechen281 zoechen281 commented Nov 20, 2026 •

Copy link
Copy Markdown

PR type

  • Bug Fix

PR information

Fixes #10543

_inject_dataset_routing_tag injects the constant dataset routing column via Dataset.add_column(), which creates an in-memory column over a memory-mapped table. When a multiprocess map pickles a shard of that mixed layout, the slice's shared parent buffers are re-carried — a 32-row shard grows from ~1.2 KB to ~105 KB (the issue reports 3.7 KB → 3.3 MB on larger inputs, and 368 MB for a 100-row shard).

This PR materializes the column through a batched map instead, so the routing column is written to disk together with the other columns and shard pickling stays near baseline. The write runs inside safe_ddp_context — the same rank-serialization pattern load_dataset already uses — so the fingerprint-derived cache file keeps the multi-rank behavior fixed in #10121 (flatten_indices is still never called; the existing routing-tag tests pass unchanged). Datasets with zero rows or no columns keep the add_column fallback, since map skips its callback entirely there and the column would never be created.

AI assistance was used in preparing this PR.

Experiment results

Repro (datasets 5.0.1, pyarrow 25.0.1, 16 disk chunks × 32 rows, shard = first 32 rows):

layout pickled shard size
disk columns only 1,187 B
add_column() routing tag 104,550 B
batched-map routing tag (this PR) 1,296 B

Tests run:

pytest tests/general/test_dataset_routing_tag.py -v
# 4 passed, 4 subtests passed

Checklist

  • I have linked the related issue above, if there is one.
  • I have reviewed and understand every changed line, and I can explain how the change works and why it is correct.
  • I have run the relevant tests or reproductions locally, and included the commands and results above (if applicable).
  • I am responsible for the content of this PR, regardless of how it was produced.

…of add_column

_inject_dataset_routing_tag used Dataset.add_column(), which creates an
in-memory column over a memory-mapped table. Pickling a slice of that
mixed layout re-carries the shared parent buffers, so each shard sent
to a multiprocess map worker is inflated by orders of magnitude
(e.g. a 32-row shard: 1.2 KB -> 105 KB; larger runs show 3.7 KB ->
3.3 MB).

Write the constant routing column through a batched map instead so it
is materialized to disk together with the other columns. The write is
wrapped in safe_ddp_context so ranks serialize the fingerprint-derived
cache file the same way dataset loading already does, preserving the
multi-rank race fix from modelscope#10121.

Fixes modelscope#10543

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Generated with [Devin](https://devin.ai)
@zoechen281
zoechen281 force-pushed the codex/routing-tag-disk branch from 1b4a34f to 205fc8d Compare November 20, 2026 14:43

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dataset 路由列注入导致混合 mmap/内存 Arrow 表在多进程 map 中出现序列化与内存放大

1 participant