diff --git a/llp/0290-local-inference-capture-lane.decision.md b/llp/0290-local-inference-capture-lane.decision.md new file mode 100644 index 00000000..1da60f60 --- /dev/null +++ b/llp/0290-local-inference-capture-lane.decision.md @@ -0,0 +1,204 @@ +# LLP 0290: Local inference is an out-of-tree lane; core owes it a precedence contract + +**Type:** Decision +**Status:** Draft +**Systems:** Gateway, Plugins, Config +**Author:** Brendan / Claude +**Date:** 2026-08-18 +**Related:** LLP 0016 (the gateway owns transport, adapters own wire shapes), LLP 0030 (`session_id` is the partition key), LLP 0157 (R5), LLP 0167 / 0193 / 0194 (the OpenClaw route that already lands `ollama` rows), LLP 0234 (recording follows the path anchor) + +> Capturing a local inference server (Ollama, LM Studio, llama.cpp, vLLM) is +> **not bundled work**. The `hypaware.ai-gateway@2.0.0` capability already +> names "future custom integrations" as a consumer and the install path is +> complete, so the whole lane is buildable out of tree. What core owes is the +> one thing an out-of-tree author cannot secure for themselves: a **documented +> projector dispatch precedence**, because today such a plugin's correctness +> rests on outranking an undocumented priority number inside +> `@hypaware/codex`. This document settles ownership and records the design +> constraints any implementation must satisfy. + +## Context {#context} + +Ollama writes no conversation record. `~/.ollama` holds `models/`, +`config.json`, a keypair, `cache/`, and server logs whose lines carry method, +path, status, and latency but no prompt or completion text. There is no REPL +history file, so a terminal `ollama run` exists only in the process. + +The desktop app is different: it keeps `chats`, `messages` (with `thinking`), +`tool_calls`, and `attachments` in +`~/Library/Application Support/Ollama/db.sqlite`. That is a local-store sweep +target on the claude-desktop pattern, unrelated to the proxy lane below. + +HypAware already lands `ollama` rows by exactly one route: an OpenClaw turn +against an Ollama backend, projected by the transcript sweep because LLP 0193 +made the exclusion a denylist of CLI backends rather than a vendor allowlist, +and stamped with its own turn's provider by LLP 0194. That works because +OpenClaw keeps a transcript. Nothing else that talks to `localhost:11434` +does. + +## Decision {#decision} + +### D1. The lane is not bundled {#ownership} + +Every bundled adapter captures a **named client it can find and attach +to**: Claude Code, Codex, Claude Desktop, OpenClaw. Each discovers its +client's config, writes its own overrides, and reports what it changed. + +A local-inference lane can do none of that. Ollama is a server, not a client; +it has no session concept and an open-ended population of callers; and the +lane's value depends entirely on the user editing each app's `base_url` by +hand, which HypAware cannot detect, perform, or verify. A bundled picker entry +would therefore advertise a capability that does nothing until per-app manual +work happens elsewhere. + +Bundling would also commit this repo to ongoing compatibility with LM Studio, +llama.cpp, vLLM, and any OpenAI-compatible proxy, for a surface where the +per-install variation (which server, which port, which apps, which session +convention) is exactly the kind of thing config and a local plugin exist to +absorb. + +So the lane ships **out of tree**: a user-authored or separately published +plugin on `hypaware.ai-gateway@2.0.0`. This is the case that capability was +designed for, and the install path (`src/core/plugin_install/`, with +`buildPluginCatalog` merging installed manifests over bundled ones) already +supports it. + +### D2. Core owes a projector precedence contract {#core-owes} + +`dispatchProjector` filters projectors by `match()`, sorts by priority then +registration sequence, and returns the first valid projection; a decline +(`undefined` or an empty `messages` array) continues the walk. + +`@hypaware/codex`'s projector matches `isOpenAiChatPath(path)`, which is true +for `/v1/chat/completions` and a bare `/chat/completions`, at priority `100`. +Ollama's OpenAI-compatible surface is exactly those paths. So any out-of-tree +local-inference projector must outrank `100` or its rows are stamped by +Codex's provider resolution, reading `provider = 'openai'` with +`model = 'gemma4:12b'`: the precise defect LLP 0194 exists to fix, arriving +through a new door. + +That number is presently an implementation detail of a sibling plugin. An +out-of-tree author has no way to depend on it and no way to notice when it +changes, and the failure is silent: rows keep arriving, mislabeled. + +Core therefore owes: + +- a **documented priority band** reserving a range above the bundled adapters + for third-party projectors, published with the capability rather than + discoverable only by reading Codex; +- a **test in this repo** that pins bundled projector priorities against that + band, so a future bump to a bundled adapter fails here instead of silently + degrading an installed plugin. + +This is the whole of core's obligation for the repoint lane. + +### D3. Constraints any implementation must satisfy {#constraints} + +Recorded here because they were established while investigating, and because +an out-of-tree author would otherwise rediscover them the expensive way. + +**Match on `upstream`, never on path.** `AiGatewayExchangeInput` carries +`upstream: string` beside `provider`, `path`, and the bodies. Matching on +path collides with Codex per {#core-owes}. The projector must also **never +decline** for its own upstream, since a decline hands the exchange straight +back to Codex. + +**Synthesize `session_id`.** It is `nullable: false`, the partition key +(LLP 0030), and a stateless server supplies nothing to fill it. Two rungs: an +explicit request header first, exact for apps the user controls; otherwise a +hash of the conversation prefix (upstream, canonicalized system prompt, +canonicalized first user message), which is stable as a thread grows because +every request resends the whole history. The mechanism is the gateway's own +fallback message identity lifted from message to thread. Known failure: two +threads with identical prefixes merge, which is what makes the header rung +matter rather than being an optimization. + +**Project assistant turns from the response only, never from request +history.** Every request resends the whole thread, so prompts and tool results +must be re-emitted each turn and collapse on the gateway's content-hash +identity. Assistant turns must not: an echoed assistant message is a lossy +copy of one the response already produced (the client drops `usage`, may drop +reasoning, and re-serializes `tool_calls`), so its hash differs and it does +NOT dedupe. Verified by running: a two-turn tool-calling thread produced the +same `tool_call` row twice until echoes were excluded. The bounded cost is +that a thread first seen at turn N recovers no assistant turn before N. + +**Translate the tool wire forms.** The gateway's part schema is +Anthropic-shaped: `tool_use` blocks (`id`, `name`, `input`) become `tool_call` +rows with `tool_name` / `tool_call_id` / `tool_args`, and `tool_result` blocks +(`tool_use_id`) become `tool_result` rows the gateway back-fills the tool name +onto from its own lookup. An OpenAI assistant turn carrying `tool_calls` has +`content: null`, so a projector that reads only text drops the entire turn. +Streaming splits `function.arguments` across arbitrarily many fragments keyed +by `index`, which must be accumulated and assembled at end of stream. + +**Target `/v1` first.** Ollama's OpenAI-compatible surface streams real SSE, +which the gateway's WHATWG eventsource parser already turns into +`stream_events`. Native `/api/chat` and `/api/generate` stream NDJSON, which +that parser yields nothing for, and a second parser selected by response +content type is core work, not plugin work. An implementation that wants the +native surface must come back here first. + +### D4. Shadow mode is out of scope, and is the only part needing core work {#shadow} + +Binding the server's own port (`OLLAMA_HOST=127.0.0.1:11435` on the server, +gateway on `11434`) is the only design that captures terminal `ollama run` and +apps with a hardcoded endpoint. It is deferred, and if revived it needs a new +request, because unlike the repoint lane it cannot be done out of tree and it +carries three findings that a future author should not have to rediscover: + +- **`OLLAMA_HOST` is both the server bind and the CLI client target.** + Verified 2026-08-18: with a live server on `11434`, + `OLLAMA_HOST=127.0.0.1:19999 ollama list` reaches `19999` and fails. A + user-session-wide set moves the CLI off the shadowed port along with the + server, defeating the lane silently. It must be scoped to the server + process, which the macOS desktop app (which spawns its own server and + injects `OLLAMA_MODELS`, `OLLAMA_NO_CLOUD`) offers no seam for. +- **An unanswered port gets reclaimed.** The CLI tries to start a server at + the address it is pointed at when nothing answers. With the daemon down, a + client can bind a real Ollama to the shadowed port, after which capture is + gone with no error anywhere. +- **Recording is unconditional on the reverse-proxy branch.** + `shouldRecordProxyExchange` honors an upstream's `record_prefix`, but the + caller short-circuits `recording` to true for reverse-proxy traffic and + never reaches the anchor. Since the recorder buffers whole bodies with no + size cap, a shadowed `11434` would buffer `ollama pull` blobs into the + daemon heap. Extending that gate is core work. + +It also puts the daemon in the hard path of all local inference on the +machine, which no current adapter does, against LLP 0157 R5's spirit. + +## Consequences {#consequences} + +- Nothing ships in this repo for the repoint lane except {#core-owes}: a + documented band and a test. The plugin is somebody's own. +- The plugin is trivially replaceable and per-install tunable, which suits a + surface whose every parameter varies by machine. +- `ollama run`, the desktop app, and hardcoded-endpoint apps stay uncaptured. + That is the accepted cost of not doing D4. +- Bundled projector priorities become a published contract, mildly + constraining future adapter work in exchange for making out-of-tree + adapters viable at all. +- A user who declares only an upstream, with no plugin, still gets a working + proxy and raw rows in `ai_gateway_exchanges`; only the projection into + `ai_gateway_messages` needs the plugin. + +## Open {#open} + +- **What `ollama run` posts.** Assumed `/api/chat` with accumulated messages, + which is what would make the {#constraints} prefix hash work for it. Older + Ollama used `/api/generate` with a token `context` array carrying no message + history. Only matters if D4 is revived. +- **Where the band boundary sits.** {#core-owes} asserts a reserved range + above bundled adapters without fixing the numbers; the implementing plan + picks them. +- **Whether the desktop `db.sqlite` sweep is worth a separate adapter.** Same + vendor, entirely different mechanism, and it would capture the GUI app the + proxy lane cannot. + +## References + +- LLP 0016: AI Gateway as a Plugin +- LLP 0030: Split `session_id` from `conversation_id` in ai_gateway_messages +- LLP 0193 / LLP 0194: the OpenClaw route that lands `ollama` rows today +- LLP 0234: recording follows the path anchor diff --git a/llp/0291-local-inference-capture-implementation.plan.md b/llp/0291-local-inference-capture-implementation.plan.md new file mode 100644 index 00000000..00c7d80b --- /dev/null +++ b/llp/0291-local-inference-capture-implementation.plan.md @@ -0,0 +1,127 @@ +# LLP 0291: Projector precedence contract implementation plan + +**Type:** Plan +**Status:** Draft +**Systems:** Gateway, Plugins +**Author:** Brendan / Claude +**Date:** 2026-08-18 +**Related:** LLP 0290 (the decision this executes), LLP 0016 (the capability the band is published with), LLP 0194 (the mislabeling the band prevents) + +> Turns the one first-party obligation of +> [LLP 0290](./0290-local-inference-capture-lane.decision.md) into three +> tasks. The local-inference plugin itself is explicitly **not** in this plan: +> [#ownership](./0290-local-inference-capture-lane.decision.md#ownership) puts +> it out of tree. + +## Sequencing principle {#sequencing} + +**Small, and shippable independently of anything using it.** The band is +useful the moment it is published and pinned, whether or not a local-inference +plugin ever exists, because it also protects any other third-party adapter on +`hypaware.ai-gateway@2.0.0`. Nothing here blocks on, or waits for, work +outside this repo. + +## What was verified against the tree {#verified} + +Checked 2026-08-18. + +- **Dispatch is priority-ordered, and declining continues the walk.** + `dispatchProjector` filters by `match`, sorts `byPriorityThenSeq`, and per + projector: a usage-policy drop returns terminally, an invalid or `undefined` + result `continue`s, an empty `messages` array `continue`s, the first valid + projection wins (`ai-gateway/src/message_projector.js:791-830`). +- **The collision is real and silent.** Codex's projector matches + `isOpenAiChatPath(path)` (`codex/src/exchange-projector.js:63`), which + accepts `/v1/chat/completions`, `/chat/completions`, any path ending in + `/chat/completions`, and both as prefixes (`:351-357`), at `priority: 100` + (`:61`). Ollama's OpenAI-compatible surface is exactly that set. +- **Third-party adapters are an intended consumer.** The gateway registers + `hypaware.ai-gateway@2.0.0` so adapter plugins, "`@hypaware/claude`, + `@hypaware/codex`, future custom integrations", can contribute "upstream + presets, client wiring, and exchange projectors" + (`ai-gateway/src/index.js:20-25,46`). +- **The install path is complete.** `src/core/plugin_install/` carries + resolver, fetch, git fetch, lock, install, and update check, and + `buildPluginCatalog(bundledManifests, installedManifests)` merges installed + manifests with bundled names taking precedence + (`src/core/plugin_catalog.js:32,46`). +- **`upstream` is on the projector input**, so an out-of-tree projector has a + collision-free match key available + (`hypaware-plugin-kernel-types.d.ts:1959-1971`). +- **Only bundled projectors exist today**, so introducing a band reorders + nothing at runtime on a current install. +- **The install path is a real product surface, and the scaffolder has a + hole.** `resolveSource` fixes five precedence rungs and requires third-party + names to be `hypaware-plugin-` or `@scope/hypaware-plugin-` + (`src/core/plugin_install/resolver.js:11-16`), and `hyp plugin` + carries install / list / info / outdated / update / remove / doctor / new + (`src/core/cli/core_commands.js:169-219`). But `scaffoldPlugin` accepts only + `source | sink | dataset` (`src/core/plugin_doctor/scaffold.js:31,91-96`): + there is no gateway-adapter kind, so an out-of-tree adapter author starts + from a source skeleton and reverse-engineers the capability wiring from + bundled plugin source. T4 closes that. + +Not verified: how many distinct priorities the bundled adapters actually use, +which T1 must enumerate before choosing boundaries. + +## The task graph {#tasks} + +### Wave 1 (deps `[]`) + +- **T1, define and publish the band.** Enumerate every bundled projector's + `priority`, then reserve a numeric range **above** them for third-party + projectors and document it where the capability is documented, not in a + sibling adapter: the `hypaware.ai-gateway@2.0.0` surface in + `ai-gateway/src/api.js` and the `AiGatewayExchangeProjector` docs in + `hypaware-plugin-kernel-types.d.ts`. State the two rules an out-of-tree + projector depends on: it outranks every bundled projector, and a decline + hands the exchange to the next match, so it must not decline for traffic it + owns. + +### Wave 2 (deps `[T1]`), three-wide + +- **T2, pin it.** A traditional test asserting every bundled projector's + priority sits inside the bundled range, so raising one into the third-party + band fails here rather than silently restamping an installed plugin's rows. + Assert the ordering property directly too (a higher-priority projector wins + a path both match), so the guarantee is tested and not merely asserted by + the constants. +- **T3, the author's guide.** A short `docs/` page for writing a gateway + adapter out of tree: register on the capability, contribute an upstream, + match on `upstream` rather than path and why + ([LLP 0290 #core-owes](./0290-local-inference-capture-lane.decision.md#core-owes)), + the band, the no-decline rule, and the `session_id` synthesis constraint + from [#constraints](./0290-local-inference-capture-lane.decision.md#constraints). + Use the local-inference lane as the worked example, including that an + upstream declared with no plugin still yields rows in + `ai_gateway_exchanges`. + +- **T4, an `adapter` scaffold kind.** Add `adapter` to `SCAFFOLD_KINDS` and + give it a template that registers on `hypaware.ai-gateway@2.0.0`, + contributes one upstream, and registers a projector stub that matches on + `upstream`, sits in the third-party band, and carries the no-decline rule as + a comment. The scaffold is where an author meets these rules whether or not + they find T3's guide, which makes it the durable half of the pair. Deps + `[T1]` (it hardcodes a band value), parallel to T2 and T3. + +## Annotations owed {#annotations} + +- The band constants in T1: `@ref LLP 0290#core-owes [implements]`. +- The pin test in T2: `@ref LLP 0290#core-owes [tests]`, naming the LLP 0194 + mislabeling as the failure it guards. +- Codex's `priority: 100` literal gains + `@ref LLP 0290#core-owes [constrained-by]`, since it is now a published + boundary rather than a local choice. + +## Out of scope {#out-of-scope} + +- **The local-inference plugin itself** (upstream, projector, session + identity). Out of tree per + [LLP 0290 #ownership](./0290-local-inference-capture-lane.decision.md#ownership). +- **Shadow mode** and the reverse-proxy record anchor it would require. + Deferred by [LLP 0290 #shadow](./0290-local-inference-capture-lane.decision.md#shadow); + reviving it needs a new request. +- **NDJSON stream parsing** for Ollama's native `/api/*` surface. Core work, + but only worth doing behind a decision that wants the native surface. +- **The Ollama desktop `db.sqlite` sweep.** Different mechanism entirely; + flagged in [LLP 0290 #open](./0290-local-inference-capture-lane.decision.md#open).