Add Jobs 2.0 decision guidance for populate modes

[dimitri-yatsenko]

Summary

Adds comprehensive decision guidance for choosing between different populate() modes, addressing a critical gap for users deploying distributed pipelines.

Problem

From cohesion review (COHESION-REVIEW.md Issue #9, Medium Priority):

• run-computations.md doesn't explain when reserve_jobs=True is necessary
• distributed-computing.md jumps to advanced patterns without decision criteria
• Users asking: "Should I use distributed mode?" or "When do I need reserve_jobs?"
• No guidance on performance trade-offs

Solution

New "When to Use Distributed Mode" section in how-to/run-computations.md

Content Structure

Three Populate Modes with Clear Criteria:

1. populate() (Default - Simple Mode)

Use when:
✅ Single worker
✅ Fast computations (< 1 minute each)
✅ Small job count (< 100 entries)
✅ Development/testing

Advantages:

• Simplest approach
• No job management overhead
• Immediate execution
• Easy debugging

2. populate(reserve_jobs=True) (Distributed Mode)

Use when:
✅ Multiple workers (different machines/processes)
✅ Long computations (> 1 minute each)
✅ Production pipelines
✅ Worker crashes expected

Advantages:

• Prevents duplicate work
• Fault tolerance
• Job status tracking
• Error isolation

Performance note:

• Overhead: ~100ms per job
• Worth it when: computations > 10 seconds

3. populate(reserve_jobs=True, processes=N) (Parallel Mode)

Use when:
✅ Multi-core machine
✅ CPU-bound tasks
✅ Independent computations

Advantages:

• Parallel execution on single machine
• No network coordination needed
• Combines safety with parallelism

Caution: Don't exceed CPU core count

Decision Tree

How many workers?
├─ One → populate()
└─ Multiple → Continue...

How long per computation?
├─ < 1 minute → populate() (overhead not worth it)
└─ > 1 minute → Continue...

Need fault tolerance?
├─ Yes → populate(reserve_jobs=True)
└─ No → populate() (simpler)

Multiple cores on one machine?
└─ Yes → populate(reserve_jobs=True, processes=N)

User Impact

Before (Confusion)

• "Do I need reserve_jobs=True?"
• "Why is my single-worker pipeline using job reservation?"
• "Should I use distributed mode for 100 quick jobs?"
• Reading distributed-computing.md to understand basics

After (Clarity)

• Clear decision criteria for each mode
• Understand performance trade-offs (100ms overhead vs fault tolerance)
• Choose optimal mode for workload
• Know when distributed mode helps vs hurts

Performance Guidance

Key insights added:

• Job reservation overhead: ~100ms per job
• Worth it threshold: computations > 10 seconds
• Multi-core: Don't exceed CPU count
• Fast jobs (< 1 min): Default mode often better

Placement

Inserted before "Distributed Computing" section in run-computations.md:

1. Basic Usage (existing)
2. Restrict What to Compute (existing)
3. Error Handling (existing)
4. When to Use Distributed Mode ← NEW (decision guide)
5. Distributed Computing (existing, now with context)
6. make() Method (existing)

Rationale: Users need decision guidance before learning distributed computing patterns.

Related

• Addresses COHESION-REVIEW.md Issue Fix typos #9 (Medium Priority)
• Complements distributed-computing.md (which covers advanced patterns)
• Works with learning paths (PR Add learning paths to tutorials for better navigation #113) - users understand when to use distributed path
• Part of comprehensive decision guide series (storage codecs, specs, etc.)

Completes Medium-Priority Cohesion Review

This PR completes all medium-priority issues from the cohesion review:

• ✅ Issue Update github repo references after name change #5: Learning paths (PR Add learning paths to tutorials for better navigation #113)
• ✅ Issue Add cards to core #6: Storage codec decision guide (PR Add comprehensive storage codec decision guide #114)
• ✅ Issue Transfer material from datajoint/datajoint-elements repository #7: Reference specs index (PR Enhance specs index with reading order and cross-references #115)
• ✅ Issue Broken glossary link #8: Object storage index (PR Add comprehensive object storage documentation index #116)
• ✅ Issue Fix typos #9: Jobs 2.0 decision guidance (this PR)

[dimitri-yatsenko]

Consolidated into #119 - Documentation Cohesion Review: Comprehensive Improvements for DataJoint 2.0