Skip to content

LLP 0290/0291: local inference is an out-of-tree lane; core owes a projector precedence contract #956

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bgmcmullen wants to merge 1 commit into
base: master
Choose a base branch
from
Open

LLP 0290/0291: local inference is an out-of-tree lane; core owes a projector precedence contract #956

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
204 changes: 204 additions & 0 deletions llp/0290-local-inference-capture-lane.decision.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,204 @@
# LLP 0290: Local inference is an out-of-tree lane; core owes it a precedence contract

**Type:** Decision
**Status:** Draft
**Systems:** Gateway, Plugins, Config
**Author:** Brendan / Claude
**Date:** 2026-08-18
**Related:** LLP 0016 (the gateway owns transport, adapters own wire shapes), LLP 0030 (`session_id` is the partition key), LLP 0157 (R5), LLP 0167 / 0193 / 0194 (the OpenClaw route that already lands `ollama` rows), LLP 0234 (recording follows the path anchor)

> Capturing a local inference server (Ollama, LM Studio, llama.cpp, vLLM) is
> **not bundled work**. The `hypaware.ai-gateway@2.0.0` capability already
> names "future custom integrations" as a consumer and the install path is
> complete, so the whole lane is buildable out of tree. What core owes is the
> one thing an out-of-tree author cannot secure for themselves: a **documented
> projector dispatch precedence**, because today such a plugin's correctness
> rests on outranking an undocumented priority number inside
> `@hypaware/codex`. This document settles ownership and records the design
> constraints any implementation must satisfy.

## Context {#context}

Ollama writes no conversation record. `~/.ollama` holds `models/`,
`config.json`, a keypair, `cache/`, and server logs whose lines carry method,
path, status, and latency but no prompt or completion text. There is no REPL
history file, so a terminal `ollama run` exists only in the process.

The desktop app is different: it keeps `chats`, `messages` (with `thinking`),
`tool_calls`, and `attachments` in
`~/Library/Application Support/Ollama/db.sqlite`. That is a local-store sweep
target on the claude-desktop pattern, unrelated to the proxy lane below.

HypAware already lands `ollama` rows by exactly one route: an OpenClaw turn
against an Ollama backend, projected by the transcript sweep because LLP 0193
made the exclusion a denylist of CLI backends rather than a vendor allowlist,
and stamped with its own turn's provider by LLP 0194. That works because
OpenClaw keeps a transcript. Nothing else that talks to `localhost:11434`
does.

## Decision {#decision}

### D1. The lane is not bundled {#ownership}

Every bundled adapter captures a **named client it can find and attach
to**: Claude Code, Codex, Claude Desktop, OpenClaw. Each discovers its
client's config, writes its own overrides, and reports what it changed.

A local-inference lane can do none of that. Ollama is a server, not a client;
it has no session concept and an open-ended population of callers; and the
lane's value depends entirely on the user editing each app's `base_url` by
hand, which HypAware cannot detect, perform, or verify. A bundled picker entry
would therefore advertise a capability that does nothing until per-app manual
work happens elsewhere.

Bundling would also commit this repo to ongoing compatibility with LM Studio,
llama.cpp, vLLM, and any OpenAI-compatible proxy, for a surface where the
per-install variation (which server, which port, which apps, which session
convention) is exactly the kind of thing config and a local plugin exist to
absorb.

So the lane ships **out of tree**: a user-authored or separately published
plugin on `hypaware.ai-gateway@2.0.0`. This is the case that capability was
designed for, and the install path (`src/core/plugin_install/`, with
`buildPluginCatalog` merging installed manifests over bundled ones) already
supports it.

### D2. Core owes a projector precedence contract {#core-owes}

`dispatchProjector` filters projectors by `match()`, sorts by priority then
registration sequence, and returns the first valid projection; a decline
(`undefined` or an empty `messages` array) continues the walk.

`@hypaware/codex`'s projector matches `isOpenAiChatPath(path)`, which is true
for `/v1/chat/completions` and a bare `/chat/completions`, at priority `100`.
Ollama's OpenAI-compatible surface is exactly those paths. So any out-of-tree
local-inference projector must outrank `100` or its rows are stamped by
Codex's provider resolution, reading `provider = 'openai'` with
`model = 'gemma4:12b'`: the precise defect LLP 0194 exists to fix, arriving
through a new door.

That number is presently an implementation detail of a sibling plugin. An
out-of-tree author has no way to depend on it and no way to notice when it
changes, and the failure is silent: rows keep arriving, mislabeled.

Core therefore owes:

- a **documented priority band** reserving a range above the bundled adapters
for third-party projectors, published with the capability rather than
discoverable only by reading Codex;
- a **test in this repo** that pins bundled projector priorities against that
band, so a future bump to a bundled adapter fails here instead of silently
degrading an installed plugin.

This is the whole of core's obligation for the repoint lane.

### D3. Constraints any implementation must satisfy {#constraints}

Recorded here because they were established while investigating, and because
an out-of-tree author would otherwise rediscover them the expensive way.

**Match on `upstream`, never on path.** `AiGatewayExchangeInput` carries
`upstream: string` beside `provider`, `path`, and the bodies. Matching on
path collides with Codex per {#core-owes}. The projector must also **never
decline** for its own upstream, since a decline hands the exchange straight
back to Codex.

**Synthesize `session_id`.** It is `nullable: false`, the partition key
(LLP 0030), and a stateless server supplies nothing to fill it. Two rungs: an
explicit request header first, exact for apps the user controls; otherwise a
hash of the conversation prefix (upstream, canonicalized system prompt,
canonicalized first user message), which is stable as a thread grows because
every request resends the whole history. The mechanism is the gateway's own
fallback message identity lifted from message to thread. Known failure: two
threads with identical prefixes merge, which is what makes the header rung
matter rather than being an optimization.

**Project assistant turns from the response only, never from request
history.** Every request resends the whole thread, so prompts and tool results
must be re-emitted each turn and collapse on the gateway's content-hash
identity. Assistant turns must not: an echoed assistant message is a lossy
copy of one the response already produced (the client drops `usage`, may drop
reasoning, and re-serializes `tool_calls`), so its hash differs and it does
NOT dedupe. Verified by running: a two-turn tool-calling thread produced the
same `tool_call` row twice until echoes were excluded. The bounded cost is
that a thread first seen at turn N recovers no assistant turn before N.

**Translate the tool wire forms.** The gateway's part schema is
Anthropic-shaped: `tool_use` blocks (`id`, `name`, `input`) become `tool_call`
rows with `tool_name` / `tool_call_id` / `tool_args`, and `tool_result` blocks
(`tool_use_id`) become `tool_result` rows the gateway back-fills the tool name
onto from its own lookup. An OpenAI assistant turn carrying `tool_calls` has
`content: null`, so a projector that reads only text drops the entire turn.
Streaming splits `function.arguments` across arbitrarily many fragments keyed
by `index`, which must be accumulated and assembled at end of stream.

**Target `/v1` first.** Ollama's OpenAI-compatible surface streams real SSE,
which the gateway's WHATWG eventsource parser already turns into
`stream_events`. Native `/api/chat` and `/api/generate` stream NDJSON, which
that parser yields nothing for, and a second parser selected by response
content type is core work, not plugin work. An implementation that wants the
native surface must come back here first.

### D4. Shadow mode is out of scope, and is the only part needing core work {#shadow}

Binding the server's own port (`OLLAMA_HOST=127.0.0.1:11435` on the server,
gateway on `11434`) is the only design that captures terminal `ollama run` and
apps with a hardcoded endpoint. It is deferred, and if revived it needs a new
request, because unlike the repoint lane it cannot be done out of tree and it
carries three findings that a future author should not have to rediscover:

- **`OLLAMA_HOST` is both the server bind and the CLI client target.**
Verified 2026-08-18: with a live server on `11434`,
`OLLAMA_HOST=127.0.0.1:19999 ollama list` reaches `19999` and fails. A
user-session-wide set moves the CLI off the shadowed port along with the
server, defeating the lane silently. It must be scoped to the server
process, which the macOS desktop app (which spawns its own server and
injects `OLLAMA_MODELS`, `OLLAMA_NO_CLOUD`) offers no seam for.
- **An unanswered port gets reclaimed.** The CLI tries to start a server at
the address it is pointed at when nothing answers. With the daemon down, a
client can bind a real Ollama to the shadowed port, after which capture is
gone with no error anywhere.
- **Recording is unconditional on the reverse-proxy branch.**
`shouldRecordProxyExchange` honors an upstream's `record_prefix`, but the
caller short-circuits `recording` to true for reverse-proxy traffic and
never reaches the anchor. Since the recorder buffers whole bodies with no
size cap, a shadowed `11434` would buffer `ollama pull` blobs into the
daemon heap. Extending that gate is core work.

It also puts the daemon in the hard path of all local inference on the
machine, which no current adapter does, against LLP 0157 R5's spirit.

## Consequences {#consequences}

- Nothing ships in this repo for the repoint lane except {#core-owes}: a
documented band and a test. The plugin is somebody's own.
- The plugin is trivially replaceable and per-install tunable, which suits a
surface whose every parameter varies by machine.
- `ollama run`, the desktop app, and hardcoded-endpoint apps stay uncaptured.
That is the accepted cost of not doing D4.
- Bundled projector priorities become a published contract, mildly
constraining future adapter work in exchange for making out-of-tree
adapters viable at all.
- A user who declares only an upstream, with no plugin, still gets a working
proxy and raw rows in `ai_gateway_exchanges`; only the projection into
`ai_gateway_messages` needs the plugin.

## Open {#open}

- **What `ollama run` posts.** Assumed `/api/chat` with accumulated messages,
which is what would make the {#constraints} prefix hash work for it. Older
Ollama used `/api/generate` with a token `context` array carrying no message
history. Only matters if D4 is revived.
- **Where the band boundary sits.** {#core-owes} asserts a reserved range
above bundled adapters without fixing the numbers; the implementing plan
picks them.
- **Whether the desktop `db.sqlite` sweep is worth a separate adapter.** Same
vendor, entirely different mechanism, and it would capture the GUI app the
proxy lane cannot.

## References

- LLP 0016: AI Gateway as a Plugin
- LLP 0030: Split `session_id` from `conversation_id` in ai_gateway_messages
- LLP 0193 / LLP 0194: the OpenClaw route that lands `ollama` rows today
- LLP 0234: recording follows the path anchor
127 changes: 127 additions & 0 deletions llp/0291-local-inference-capture-implementation.plan.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# LLP 0291: Projector precedence contract implementation plan

**Type:** Plan
**Status:** Draft
**Systems:** Gateway, Plugins
**Author:** Brendan / Claude
**Date:** 2026-08-18
**Related:** LLP 0290 (the decision this executes), LLP 0016 (the capability the band is published with), LLP 0194 (the mislabeling the band prevents)

> Turns the one first-party obligation of
> [LLP 0290](./0290-local-inference-capture-lane.decision.md) into three
> tasks. The local-inference plugin itself is explicitly **not** in this plan:
> [#ownership](./0290-local-inference-capture-lane.decision.md#ownership) puts
> it out of tree.

## Sequencing principle {#sequencing}

**Small, and shippable independently of anything using it.** The band is
useful the moment it is published and pinned, whether or not a local-inference
plugin ever exists, because it also protects any other third-party adapter on
`hypaware.ai-gateway@2.0.0`. Nothing here blocks on, or waits for, work
outside this repo.

## What was verified against the tree {#verified}

Checked 2026-08-18.

- **Dispatch is priority-ordered, and declining continues the walk.**
`dispatchProjector` filters by `match`, sorts `byPriorityThenSeq`, and per
projector: a usage-policy drop returns terminally, an invalid or `undefined`
result `continue`s, an empty `messages` array `continue`s, the first valid
projection wins (`ai-gateway/src/message_projector.js:791-830`).
- **The collision is real and silent.** Codex's projector matches
`isOpenAiChatPath(path)` (`codex/src/exchange-projector.js:63`), which
accepts `/v1/chat/completions`, `/chat/completions`, any path ending in
`/chat/completions`, and both as prefixes (`:351-357`), at `priority: 100`
(`:61`). Ollama's OpenAI-compatible surface is exactly that set.
- **Third-party adapters are an intended consumer.** The gateway registers
`hypaware.ai-gateway@2.0.0` so adapter plugins, "`@hypaware/claude`,
`@hypaware/codex`, future custom integrations", can contribute "upstream
presets, client wiring, and exchange projectors"
(`ai-gateway/src/index.js:20-25,46`).
- **The install path is complete.** `src/core/plugin_install/` carries
resolver, fetch, git fetch, lock, install, and update check, and
`buildPluginCatalog(bundledManifests, installedManifests)` merges installed
manifests with bundled names taking precedence
(`src/core/plugin_catalog.js:32,46`).
- **`upstream` is on the projector input**, so an out-of-tree projector has a
collision-free match key available
(`hypaware-plugin-kernel-types.d.ts:1959-1971`).
- **Only bundled projectors exist today**, so introducing a band reorders
nothing at runtime on a current install.
- **The install path is a real product surface, and the scaffolder has a
hole.** `resolveSource` fixes five precedence rungs and requires third-party
names to be `hypaware-plugin-<name>` or `@scope/hypaware-plugin-<name>`
(`src/core/plugin_install/resolver.js:11-16`), and `hyp plugin`
carries install / list / info / outdated / update / remove / doctor / new
(`src/core/cli/core_commands.js:169-219`). But `scaffoldPlugin` accepts only
`source | sink | dataset` (`src/core/plugin_doctor/scaffold.js:31,91-96`):
there is no gateway-adapter kind, so an out-of-tree adapter author starts
from a source skeleton and reverse-engineers the capability wiring from
bundled plugin source. T4 closes that.

Not verified: how many distinct priorities the bundled adapters actually use,
which T1 must enumerate before choosing boundaries.

## The task graph {#tasks}

### Wave 1 (deps `[]`)

- **T1, define and publish the band.** Enumerate every bundled projector's
`priority`, then reserve a numeric range **above** them for third-party
projectors and document it where the capability is documented, not in a
sibling adapter: the `hypaware.ai-gateway@2.0.0` surface in
`ai-gateway/src/api.js` and the `AiGatewayExchangeProjector` docs in
`hypaware-plugin-kernel-types.d.ts`. State the two rules an out-of-tree
projector depends on: it outranks every bundled projector, and a decline
hands the exchange to the next match, so it must not decline for traffic it
owns.

### Wave 2 (deps `[T1]`), three-wide

- **T2, pin it.** A traditional test asserting every bundled projector's
priority sits inside the bundled range, so raising one into the third-party
band fails here rather than silently restamping an installed plugin's rows.
Assert the ordering property directly too (a higher-priority projector wins
a path both match), so the guarantee is tested and not merely asserted by
the constants.
- **T3, the author's guide.** A short `docs/` page for writing a gateway
adapter out of tree: register on the capability, contribute an upstream,
match on `upstream` rather than path and why
([LLP 0290 #core-owes](./0290-local-inference-capture-lane.decision.md#core-owes)),
the band, the no-decline rule, and the `session_id` synthesis constraint
from [#constraints](./0290-local-inference-capture-lane.decision.md#constraints).
Use the local-inference lane as the worked example, including that an
upstream declared with no plugin still yields rows in
`ai_gateway_exchanges`.

- **T4, an `adapter` scaffold kind.** Add `adapter` to `SCAFFOLD_KINDS` and
give it a template that registers on `hypaware.ai-gateway@2.0.0`,
contributes one upstream, and registers a projector stub that matches on
`upstream`, sits in the third-party band, and carries the no-decline rule as
a comment. The scaffold is where an author meets these rules whether or not
they find T3's guide, which makes it the durable half of the pair. Deps
`[T1]` (it hardcodes a band value), parallel to T2 and T3.

## Annotations owed {#annotations}

- The band constants in T1: `@ref LLP 0290#core-owes [implements]`.
- The pin test in T2: `@ref LLP 0290#core-owes [tests]`, naming the LLP 0194
mislabeling as the failure it guards.
- Codex's `priority: 100` literal gains
`@ref LLP 0290#core-owes [constrained-by]`, since it is now a published
boundary rather than a local choice.

## Out of scope {#out-of-scope}

- **The local-inference plugin itself** (upstream, projector, session
identity). Out of tree per
[LLP 0290 #ownership](./0290-local-inference-capture-lane.decision.md#ownership).
- **Shadow mode** and the reverse-proxy record anchor it would require.
Deferred by [LLP 0290 #shadow](./0290-local-inference-capture-lane.decision.md#shadow);
reviving it needs a new request.
- **NDJSON stream parsing** for Ollama's native `/api/*` surface. Core work,
but only worth doing behind a decision that wants the native surface.
- **The Ollama desktop `db.sqlite` sweep.** Different mechanism entirely;
flagged in [LLP 0290 #open](./0290-local-inference-capture-lane.decision.md#open).
Loading