feat(memory): structured memory storage + per-agent opt-in gate + control API#1225
Open
pbranchu wants to merge 3 commits into
Open
feat(memory): structured memory storage + per-agent opt-in gate + control API#1225pbranchu wants to merge 3 commits into
pbranchu wants to merge 3 commits into
Conversation
Lays the groundwork for per-user memory by giving every install a stable default-user UUID and tagging every session with an owning user. Sessions are now consistently user-scoped: - `Session::user_id: UserId` (required, not Option) — defaults to the kernel's persistent default user - `Session::parent_session_id: Option<SessionId>` — foundation for future tree-scoped cascade deletion of forked sessions (no producer yet) - `MessageSource` enum + optional `Message::source` — additive type that later PRs (structured extraction filtering) will read; no consumer here - `UserConfig::is_default: bool` — `[[users]]` blocks can attach display name and channel bindings to the persistent default identity Kernel boots the default user once and caches it process-wide: - `bootstrap_default_user` — load-or-generate the UUID from `kv_store[shared, "default_user_uuid"]`, install via `set_default_user_id`, then run a one-shot rewrite of legacy nil-UUID sessions, gated by the `default_user_bootstrap_done` sentinel - `resolve_user_id` (strict, HTTP boundary) — folds the deprecated "test" alias and the nil UUID to the default user with `warn!` logs so reserved-bucket abuse is auditable - `resolve_user_id_internal` (raw mapper) — preserves the pre-fix behaviour for in-process test callers - `AuthManager::new_with_default` — binds the `is_default = true` user (or the first user) to the persistent UUID Storage and migration: - Schema v9 adds `user_id` (NOT NULL, default nil UUID) and `parent_session_id` (nullable) to `sessions`, plus `(agent_id, user_id)` and `parent_session_id` indexes - `MemorySubstrate::rewrite_nil_user_sessions` — atomic transaction wrapping the legacy-bucket UPDATE; the kernel only sets the bootstrap-done sentinel after a clean rewrite, so a failure leaves the retry path intact Single-user installs see no behaviour change: everyone is the default user, one session per agent, same as today. Tests: - `MessageSource` deserialises cleanly from pre-field payloads (JSON + msgpack) and survives full round-trips - `UserConfig::is_default` defaults to `false` for existing configs - `AuthManager::new_with_default` honours `is_default`, falls back to first user, and is a no-op for empty configs - Migration v9 adds the columns and indexes, and a v8-built DB upgrades cleanly to v9 with pre-existing rows preserved - `default_user_id`/`test_user_id` are distinct - `create_session(agent, user)` round-trips through SQLite — both the default and an explicit user - `rewrite_nil_user_sessions` is idempotent, targeted, and atomic - Kernel: default-user UUID persists across kernel restarts; the strict filter folds `"test"` and nil to default while passing other UUIDs through; the internal mapper preserves the raw `"test"` alias Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Cargo update bumps lettre from 0.11.21 to 0.11.22 to clear RUSTSEC-2026-0141. Pulls in transitive dependency updates as a side effect (mostly windows-sys/socket2 version consolidation) — no API surface change in our own code. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…trol API
Adds the storage layer, opt-in gate, and management/control HTTP API for
structured memory. No producer (extraction + dreamer) and no UI changes —
those land in follow-up PRs. Default agents see zero behavior change:
storage tables are created on migration but only populated by agents that
opt in via `[memory] system = "structured"` in their manifest.
Per-agent opt-in:
- `MemorySystem` enum (`Summarization` default / `Structured`) on
`AgentManifest`, with `MemoryConfig` wrapper carrying
`skip_serializing_if = is_default` so manifests that don't opt in
round-trip clean TOML.
- Same `[memory]` field on `HandAgentConfig`; `activate_hand` copies
through to the spawned manifest.
- `MemoryConfig::is_structured()` is the single gate consulted by every
call site that touches structured memory.
Schema migration v10 (PR 1 added v9) consolidates the three structured-
memory storage tables with denormalized columns from the start:
- `session_extractions(user_id, agent_id, ...)` — audit attribution
survives session deletes; idx on `(user_id, created_at)` for the
audit endpoint.
- `user_memory_topics` with `expires_at` + `embedding` columns.
- `user_agent_memory_topics` keyed by `(user_id, agent_id, topic)`.
Storage modules:
- `user_memory.rs` / `user_agent_memory.rs` — CRUD, embedding store, prune.
- `SessionExtraction` + `SessionExtractionStore` in `session.rs`.
- `MemorySubstrate::wipe_user(user_id) -> WipeUserCounts` wraps the three
bucket DELETEs in a single SQLite transaction so a partial failure rolls
back rather than leaving a user half-wiped.
- `MemorySubstrate::list_user_extraction_audit` joins with `sessions`
purely to surface the `session_deleted` flag — attribution comes from
the denormalized `user_id` column.
Kernel prompt-build gate:
- `build_user_memory_context(memory, user_id, agent_id, memory_cfg)`
returns `None` immediately when `memory_cfg.is_structured() == false`
(no SQLite roundtrip for default agents). Called from both prompt-build
sites in `kernel.rs`. `PromptContext::user_memory_context` carries the
value through to a new "What I Remember About You" section that is
skipped entirely for subagents and for empty indexes.
Control API (`/api/users/*`):
- `GET /api/users` — list users + default
- `GET /api/users/{user_id}/memory` — list topics
- `GET /api/users/{user_id}/memory/{topic}` — topic content
- `DELETE /api/users/{user_id}/memory/{topic}` — delete one (404 if absent)
- `DELETE /api/users/{user_id}/memory` — atomic wipe; returns per-bucket counts
- `GET /api/users/{user_id}/agents/{agent_id}/memory` — per-agent topics
- `DELETE /api/users/{user_id}/agents/{agent_id}/memory` — delete per-agent
- `GET /api/users/{user_id}/memory/audit` — extraction events with `session_deleted`
- `GET /api/users/{user_id}/memory/export` — JSON dump
- `parse_user_id()` accepts `"default"` or any non-nil UUID; rejects the
nil UUID (legacy anonymous-bucket sentinel) and the deprecated `"test"`
alias with 400.
- Module doc + per-handler `AUTHORIZATION:` comments call out the
single-tenant RBAC limitation (API-key holder == full memory admin).
PATCH /api/agents/:id/config gains `memory_system: Option<String>`,
validated against `MemorySystem`'s serde and persisted to disk via
`Registry::update_memory_config` + `persist_manifest_to_disk`. GET
/api/agents/:id surfaces both a flat `memory_system` string and the
nested `manifest.memory.system` shape so dashboards can read either form.
Tests (98 new):
- v10 migration creates all three tables + indexes and upgrades cleanly
from a v9 baseline without touching pre-existing rows.
- `wipe_user`: per-bucket counts, scope (user A's wipe leaves user B
untouched), idempotent zero-count run.
- `MemorySystem` defaults to `Summarization`; `[memory] system = "..."`
parses both variants; TOML round-trip skips `[memory]` for the default
case and re-emits it when opted in.
- `build_user_memory_context` returns `None` for default agents (even
with seeded topics), returns the formatted block when opted in with
topics, returns `None` when opted in with an empty index.
- `parse_user_id`: accepts `default` and any UUID, rejects nil UUID,
`"test"`, garbage, and empty string.
- Audit endpoint preserves attribution after session delete and flips
the `session_deleted` flag to true.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Storage and management surface for an opt-in structured memory feature. Default agents see zero behavior change. Storage tables are created on migration but only populated by agents that opt in via `[memory] system = "structured"`. PR 3 will add the producer that fills them; PR 4 surfaces them in the dashboard.
Depends on #1224 (memory foundation — sessions become user-scoped).
This is PR 2 of a 4-PR memory series.
What this PR adds
Per-agent opt-in
Storage layer
Opt-in gate at the consumer
`build_user_memory_context()` returns `None` immediately when `!manifest.memory.is_structured()`. Default `Summarization` agents skip the SQLite roundtrip entirely on prompt build.
Control API (9 endpoints)
`parse_user_id` rejects nil UUID and "test" alias at the HTTP boundary.
PATCH endpoint update
`PATCH /api/agents/{id}/config` accepts `memory_system: Option`. Validates as `MemorySystem` (400 on invalid). Persists to disk. `GET /api/agents/{id}` exposes both flat `memory_system` and nested `manifest.memory.system`.
Authorization model
Documented limitation: All `/api/users/*` endpoints accept any valid API key — they do NOT enforce that the caller matches `{user_id}` in the URL. This matches the current single-tenant deployment model. Multi-tenant deployments should add per-user authorization at the middleware layer.
Module doc + per-handler `AUTHORIZATION:` comments call this out explicitly.
Compatibility
Test plan
Reviewed independently
This PR was carved and reviewed by an independent agent. The agent verified no scope leakage (no extract/dreamer/UI/agent_loop changes), opt-in gate works at all call sites, migration v10 is forward-compatible, atomic wipe is real, and route registration order is correct.