model-router

A small reference implementation for routing LLM requests to the cheapest model that clears the bar.

It does three jobs:

Layer	Job
Tier decision	Label each request trivial, standard, or hard by task tag, rules, or a small classifier model
Router	Pick the first allowed model for that tier from policy.yaml
Verifier	Check the output and escalate to a stronger tier when the cheap attempt fails

request -> tier decision -> policy picks tier/model -> execute -> verify -> optional escalation

The app code talks OpenAI protocol over stdlib HTTP. A local LiteLLM gateway can map the model aliases in policy.yaml to OpenAI, Anthropic, Bedrock, GLM, or any other provider LiteLLM supports.

Run The Sample Workload

Mock mode needs no keys and no network. It replays the synthetic fixtures in examples/mock_fixtures.json so the pipeline stays deterministic.

python3 -m model_router.cli run
python3 -m model_router.cli run --json

Compare tiering strategies:

python3 -m model_router.cli run --route tag
python3 -m model_router.cli run --route rules
python3 -m model_router.cli run --route classifier

The important lesson is that the classifier is not free. If your calling workload already knows the task type, route by tag. If the work is predictable, route by rules. Use a classifier when difficulty is genuinely unpredictable and the extra model call earns its keep.

Run Live

Live mode sends OpenAI-compatible /chat/completions calls to a LiteLLM proxy. The proxy fans out to real providers. Token usage comes back from the provider response, so live costs are measured instead of estimated.

# 1. create a venv and install the optional gateway
uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python "litellm[proxy]"

# 2. start LiteLLM with the sample gateway config
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
export LITELLM_API_KEY=sk-local
DATABASE_URL='' .venv/bin/litellm --config litellm.config.yaml --port 4000

# 3. point the router at the gateway
export LITELLM_BASE_URL=http://localhost:4000
python3 -m model_router.cli run --live

Edit litellm.config.yaml to map the aliases in policy.yaml to the providers and models you actually want to test.

Edit The Policy

policy.yaml is the control surface:

• routing.mode: tag, rules, or classifier
• routing.task_tiers: maps caller-provided task tags to tiers
• tiers: allowed models by tier
• pricing: model prices used for the blended cost table
• verifier.checks: task-aware gates that decide whether to accept or escalate
• routing.min_confidence: bumps shaky classifications up one tier

Unknown model prices fail loudly. A typo should not price at $0 and fake savings.

What It Prints

The CLI prints a routing table and monthly cost summary:

• selected tier and model
• verifier result
• escalation path when one happened
• routed cost
• all-premium baseline
• classifier overhead
• savings

Companion Skill: model-router-audit

This repo also ships an audit skill for Claude Code and Codex-style agent folders. Point it at a codebase and it finds LLM call sites, infers each call's job, and recommends the cheapest model on your allow-list that still clears the bar.

Install for Claude Code:

ln -s "$(pwd)/.claude/skills/model-router-audit" ~/.claude/skills/model-router-audit

Install for agent skill folders:

ln -s "$(pwd)/.agents/skills/model-router-audit" ~/.agents/skills/model-router-audit

Then ask your agent to audit the repo's model usage. See .claude/skills/model-router-audit/SKILL.md or .agents/skills/model-router-audit/SKILL.md.

LiteLLM Built-In Router

This repo keeps the classifier/routing/verifier logic explicit so you can see and modify each decision. LiteLLM also ships a built-in complexity router that routes by prompt signals with no extra model call. See litellm.complexity-router.config.yaml for a gateway-only version of the same idea.