SLO-aware scheduler, spillover, and admission control

[kms12425-ctrl]

Background

The scheduler is the core logic for trading off SLO targets, memory pressure, and multi-model resource sharing.

Scope

• Implement decision flow: admission -> instance selection -> multi-model resource reallocation -> spillover.
• Prefer local execution while accounting for risk and remote transfer cost.
• Optimize for high VRAM utilization and lower serving cost under dynamic workloads.

Deliverables

• Scheduler implementation.
• Policy configurations.
• Decision audit logs.

Acceptance Criteria

• Score factors include: SLO risk, memory risk, model residency/switch cost, queue risk, spillover cost.
• Log decision reasons and selected action for each request.
• Support multi-model dynamic resource allocation on shared GPU/NPU devices.

Dependencies

• Epic: Epic: SLO-aware LLM inference service system on SAGE #1506

Notes

Heart of the project's technical innovation.

[Whatsonyourmind]

A few suggestions on the score-factor design that have worked well in similar SLO-aware schedulers:

1. Make each score factor unitless and bounded [0, 1] before composition. Mixing raw ms, raw MiB, raw count, etc. into one composite score forces ad-hoc weights and makes the formula hard to tune. Normalize each factor to a "risk probability" or a cumulative-distribution-percentile (P(this factor causes SLO violation)), then combine.

2. For multi-criteria composition, prefer Pareto-frontier elimination over weighted-sum as the first pass: eliminate any instance dominated on all dimensions, then apply weighted-sum only on the frontier. This catches the "marginally better on one axis but catastrophically worse on another" case that weighted-sum hides.

3. Score factors with strong correlation should be merged or one should be dropped. queue_risk and pending_request_count will correlate near-perfectly; memory_risk and kv_cache_residency likewise. Two correlated factors at equal weight is the same as one factor at 2× weight — usually not what you want.

4. Model residency / switch cost is the trickiest factor. Switch cost depends on what's evicted (some weights cold-start cheaply, others have large embedding tables and warm 30s+). A simple bool model_resident is too coarse; consider a swap_cost_estimate_ms that interpolates over recent observed swap-in times for that specific model.

5. Decision audit logs: emit (action_taken, top_3_alternatives, top_factor_per_alternative) rather than just action_taken + reason. When the scheduler does something surprising in production, knowing what it nearly did is half the debugging.

For the scoring framework itself, I've packaged 21 decision algorithms (composite scoring, Pareto pickers, contextual bandits, weighted-sum with auto-tuned weights) as an MCP server that might be useful as a reference for the policy module: https://github.com/Whatsonyourmind/oraclaw — but the points above stand independently.