[mpmd] Add custom pipeline API for user-defined scheduling and merging #33333
+6
−20
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[mpmd] Add custom pipeline API for user-defined scheduling and merging
Introduces a new API for defining custom MPMD pipeline schedules and fragment
merging logic using Python predicates.
How to Use:
The API centers around the PipelineSchedule object. Users can either define
their own custom schedules or use pre-built ones:
Custom Schedules:
There are two approaches for defining custom scheduling and merging logic:
Predicate-based (recommended for simple patterns): Define scheduling and
merging logic using binary predicate functions. Helper functions like
build_schedule_rules_from_predicateandbuild_merge_rules_from_predicatecreate
RuleBuildersfrom your predicates. See the docstring inpipeline.pyfor a simple example.Direct construction (for complex custom schedules): Directly build execution
order and merge rules for full control over fragment scheduling. This
approach is useful for advanced patterns that may not neatly translate to a
predicate. See the docstring in
pipeline.pyfor a comparative example.Pre-built Schedules:
Pre-defined common pipeline schedules (e.g., GPIPE, ONE_FWD_ONE_BWD) are
available through the
pipeline_registrymodule. Theget_pipeline_schedulefunction returns a
PipelineScheduleobject for a given name. See the docstringin pipeline_registry.py for usage.
In all cases, the resulting
PipelineScheduleobject is passed tompmd.jitviaMpmdConfig- this integration is done in a subsequent CL.