Skip to content

multi-step job submission proposal#76

Open
gabor-lbl wants to merge 1 commit intomainfrom
job_dag
Open

multi-step job submission proposal#76
gabor-lbl wants to merge 1 commit intomainfrom
job_dag

Conversation

@gabor-lbl
Copy link
Copy Markdown
Contributor

A proposal for running more complex job scripts, using a DAG of steps:

  • each step is a slurm "srun" (or similar) with its own optional environment (that overrides the global defintion)
  • can optionally declare a parent step (by name)
  • Steps with no dependencies run first, then the steps that depend on these.
  • Steps run in parallel when they have no dependencies between them.
  • Steps run serially when one step depends on another.
  • a child step can run optionally based on a condition where it can use the parent step's EXIT_CODE for the test

Note, this proposal does not pretend to be an entire workflow engine but rather a simple (?) way to write cross-platform job scripts with multiple steps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant