Cost calculation crashes with AttributeError for unmapped models (e.g. local Ollama models)

[na55imK]

Summary

fastllm's cost tracking crashes for model not present in the bundled
model_prices.json . Only a small hand-picked subset of Ollama tags
are keys in that file — e.g. ollama/llama2 is present, but ministral-3:3b
(and many other locally-pulled models) isn't.

Repro

ssage --vendor_name openai_chat --base_url http://localhost:11434/v1
--api_key ollama --model ministral-3:3b "hi"

The completion succeeds, but crashes right after while tracking usage:

File ".../fastllm/openai_chat.py", line 204, in cost
cost = in_txt * m.input_cost_per_token + out_txt * m.output_cost_per_token
AttributeError: input_cost_per_token

Cause

cost() looks up the model name in model_prices.json and assumes the
result always has input_cost_per_token/output_cost_per_token. There's as far as I can see no
fallback for models that aren't a key in that file.

Suggested fix

Default to cost = 0 when a model isn't found, rather than raising. Probably
also add a wildcard fallback for ollama models generally.

[KeremTurgutlu]

Can you pelase check whether the instructions from Adding a new model from types notebook work for you first?

[na55imK]

Yes — register_model_info does work as documented, confirmed with:

from fastllm.types import register_model_info, get_model_info, get_model_pricing, modern_llm
register_model_info('ministral-3:3b', 'openai_chat', **modern_llm,
    max_tokens=262_144, max_input_tokens=262_144, max_output_tokens=262_144,
    input_cost_per_token=0.0, output_cost_per_token=0.0)
print(get_model_info('ministral-3:3b', 'openai_chat'))
print(get_model_pricing('ministral-3:3b', 'openai_chat'))

But each ssage invocation is a fresh CLI process, and registrations don't persist between them — so this has to run again on every call. I worked around this with a sitecustomize.py that re-registers on interpreter startup:

cat > ~/.local/share/uv/tools/shell-sage/lib/python3.12/site-packages/sitecustomize.py << 'EOF'
from fastllm.types import register_model_info, modern_llm

try:
    import urllib.request, json
    with urllib.request.urlopen('http://localhost:11434/api/tags', timeout=1) as r:
        models = json.load(r)['models']
    for m in models:
        ctx = m.get('details', {}).get('context_length') or 32_768
        register_model_info(m['name'], 'openai_chat', **modern_llm,
            max_tokens=ctx, max_input_tokens=ctx, max_output_tokens=ctx,
            input_cost_per_token=0.0, output_cost_per_token=0.0)
except Exception:
    pass  # Ollama not running / unreachable — skip silently, don't break other Python startups
EOF

I'm not sure if this is really the best fix, and a more permanent solution either in fastllm or in shell_sage would be handy.

[KeremTurgutlu]

Thanks for reporting, perhaps having an option for users to be able to default to cost=0 for unregistered models might be an ok solution.