diff --git a/rfcs/0010-acl-based-memory-partitioning.md b/rfcs/0010-acl-based-memory-partitioning.md new file mode 100644 index 00000000..851bf65c --- /dev/null +++ b/rfcs/0010-acl-based-memory-partitioning.md @@ -0,0 +1,964 @@ +--- +title: ACL-Based Memory Partitioning and Cross-User Access +authors: + - Galin Iliev +created: 2026-07-06 +last_updated: 2026-07-06 +status: draft +issue: +rfc_pr: +--- + +# Proposal: ACL-Based Memory Partitioning and Cross-User Access + +## Summary + +Introduce ACL-enforced memory partitioning in OpenClaw where memory items are +scoped to the **session** that produced them — user-scoped for verified DM +sessions, channel-scoped for group sessions. When the upstream identity +provider is an enterprise directory (Entra ID, Okta, Google Workspace, Teams), +the tenant ID and user ID from that directory become the canonical ACL +principals. This enables per-user memory isolation and channel-scoped access, +with deliberate sharing via tenant-shared placement and per-item projections. +A direct cross-user grant mechanism is sketched but deferred (Appendix A); +cross-instance federation is out of scope (see Non-Goals). + +**Isolation strength, stated up front:** Phase 0 provides retrieval- and +tool-layer isolation. Deployments with untrusted channel members and `exec` +enabled require the out-of-process broker tier before claiming adversarial +isolation — see §2.1 and risk 1 in the Security Posture section. + +## Motivation + +Today, OpenClaw's memory system indexes content into per-agent SQLite databases +without fine-grained access control tied to the *source identity* of +contributions. In multi-user and multi-channel deployments — particularly +enterprise environments with Teams, Slack, or Discord channels — several +problems arise: + +1. **No user-level isolation.** Memory learned from user A's messages is + retrievable by user B if they share the same agent, even when the content is + personal or sensitive. + +2. **No channel-scoped boundaries.** Facts observed in a private channel can + surface in a different channel's context without the contributor's awareness. + +3. **Enterprise identity is ignored.** Organizations already have authoritative + identity (Entra ID tenant + object ID, Okta org + user ID) but the memory + layer does not use these as access principals. + +4. **Cross-user sharing is absent.** Two users of the same OpenClaw instance + (e.g., colleagues in the same tenant) have no controlled way to share a + slice of memory — it is either fully shared (same agent, no isolation) or + requires copying content out of band. There is no explicit, auditable, + revocable grant mechanism between users. + +These gaps prevent enterprise adoption and violate the principle that data +should flow only along paths the contributing user's identity authorizes. + +## Goals + +1. **Session-scoped partitioning.** Every memory item records the scope of the + session that produced it: the canonical user for verified DM sessions, the + channel for group sessions. That scope is the primary ACL principal + controlling read access. + +2. **Channel-as-scope.** Memory observed in a channel is readable only within + that channel's context by live members, unless explicitly declassified. + +3. **Enterprise IdP integration.** When an identity provider (Entra ID, Okta, + Google Workspace) is connected, use its `tenant_id` / `user_id` pair as the + canonical principal. Channel identities (Teams user ID, Slack member ID) bind + to these canonical principals through a verified pairing flow. + +4. **Deny-by-default cross-user boundaries, with deliberate sharing paths.** + No user's sessions can read another user's memory. Sharing across users + happens by deliberate placement: tenant-`shared/` or a per-item + projection (§1.2). A direct user-to-user grant mechanism is deferred + (§4, Appendix A). + +5. **Deny-by-default evaluation.** A memory item is invisible unless an explicit + allow rule matches the requesting principal set and no deny rule matches. + +## Non-Goals + +1. **Preventing semantic laundering through the context window.** Once a memory + item is loaded into a model's context for an authorized request, the model + may reason over it, rephrase it, and produce outputs influenced by it. This + is inherent to how transformer inference works — there is no token-level + provenance inside the context window. A model can rephrase "Alice prefers + morning meetings" as original text, and to the broker a subsequent write of + that rephrased content looks indistinguishable from something the current + user said directly. + + **What the broker CAN enforce (structural boundaries):** + - Directory-level write isolation: a session running under user B's scope + never mounts user A's subtree; the only path into it is the write-only + postbox (§1.2). Subject to the file-view enforcement limits in §2.1. + - Explicit derivation gating: the `derive` API accepts parent item IDs and + checks `derive` permission on each parent. Calling `derive(parents=[A's item])` + under B's principal is denied. + - Cross-principal write blocking: the broker rejects `remember` calls that + specify an `owner_user_id` different from the resolved requesting principal. + + **What the broker CANNOT enforce (semantic boundary):** + - Content rephrased by the model and written as a "new" observation under the + current user's principal. The broker sees a fresh write with no declared + lineage — it cannot prove the content originated from another user's memory. + - Indirect influence: the model's behavior is shaped by everything in context. + A response to user B may reflect patterns learned from A's memory in the + same context window. + + **Mitigations (detect, not prevent):** + - Exposure-correlated write anomaly detection: flag elevated write volume + immediately following cross-user memory exposure. + - Semantic similarity checks: compare new writes against recently-exposed + cross-user items; flag high-similarity writes for review. + - Rate limiting on writes after cross-principal reads. + - Audit trail: the `exposure_audit` table records what was exposed, enabling + post-hoc forensic queries ("did any writes by user B suspiciously correlate + with items exposed from user A?"). + + This is the "you can't un-read information" problem — the same reason DLP + systems cannot prevent a human from reading a document and typing a summary + into a different system. The enforceable boundary is structural (file + isolation, derivation API gating). Beyond that, operational controls provide + detection and deterrence, not prevention. + +2. **Content-level DLP or classification.** This RFC does not introduce + sensitivity classification of content beyond what the source identity and + channel scope imply. Content-aware DLP (PII detection, regulatory labeling) + is a future concern layered on top — and is the natural place to add + semantic similarity checks against the exposure log. + +3. **Sync protocol.** Cloud reconciliation and multi-device sync are covered by + the broader Memory Broker design and are not in scope here. + +4. **Model-side enforcement.** Preventing the model from surfacing memorized + content from its weights (training data) is out of scope. This RFC concerns + only the retrieval-augmented memory store. + +5. **Plugin state.** Plugin KV storage (`plugin_state_entries`) is a separate + system with its own isolation model and is not affected by this proposal. + +6. **Cross-instance federation.** Everything in this RFC operates within a + **single OpenClaw instance** — one broker, one host, one set of identity + bindings; the deferred grant sketch (Appendix A) is same-instance too. + Sharing memory between *different instances* (different hosts or + different tenants) requires transport authentication, offline-verifiable + capabilities, revocation propagation, and audit exchange — a federation + protocol that deserves its own RFC. The evaluation semantics here are + deliberately shaped so a capability-token federation layer can be added + later without changing them. + +## Proposal + +### What Accepting This RFC Means (Normative Core vs. Follow-Ups) + +This proposal covers a lot of surface. To keep the acceptance decision +tractable, it is split explicitly: accepting this RFC accepts the **normative +core** below — the invariants every deployment gets — and nothing more. The +follow-up column is design direction, individually revisitable without +reopening the core. + +| Normative core (this RFC) | Follow-ups / non-normative | +|---|---| +| Session-scoped attribution and the mount model (§1.1–§1.2) | Cross-user grants (§4, Appendix A — sketch only) | +| Postbox mechanism + trust tiering, deployment-controlled enablement (§1.3) | Cross-instance federation (Non-Goals §6 — future RFC) | +| Scoped subtree partitioning; SQLite as index + policy catalog (§2) | Out-of-process broker / sandboxed agents (broker design §12 Option B — required for adversarial isolation; designed separately) | +| Session file-view enforcement and its stated limits (§2.1) | Enterprise plugin internals (P1–P6 — evolve with the plugin) | +| Permission lattice + evaluation order (§3) | Anomaly detection tuning (advisory-only signal) | +| Identity binding framework + adapter registry hardening (§5, §5.1) | Sync/reconciliation semantics (broker design §10) | +| Retrieval flow (§6) and transcript/compaction scoping (§7) | Teams auto-binding policy (Unresolved Q5) | + +Note that §7 (transcripts/compaction) is deliberately in the core: it closes +a laundering channel, and deferring it would leave the core invariants +bypassable. The postbox (§1.3) is core as a *mechanism* — whether a tenant +enables it is a deployment decision (`postbox.mode`, including `off`). + +### 0. Architecture: Core vs. Plugin Split + +The ACL memory system is split into a **core enforcement layer** and optional +**enterprise plugins**: + +**Core** (`src/memory-acl/`) — always present, cannot be disabled: +- ACL evaluator (deny-by-default evaluation logic) +- Memory broker contract (the only surface agents use for memory access) +- Principal resolver framework (resolves identity → principal set) +- Directory-level memory isolation (scoped subtree layout + session mounts) +- Prefilter SQL builder (CTE generation for candidate queries) +- Lineage walker (derived-item ancestry check) +- Exposure auditor (records what entered prompts) +- Channel membership state (scope-narrowing enforcement) +- Local identity adapter (`local:{instance}:{user}` for single-user installs) + +**Enterprise plugin** (`extensions/memory-acl-enterprise/`) — optional: +- IdP adapters (Entra ID, Okta, Google Workspace OAuth flows) +- Anomaly detection (write-after-exposure semantic similarity) +- IdP group → principal mapping and refresh +- *Deferred with the grant design (Appendix A):* cross-user grant management + tools and grant admin CLI commands — no grant surface ships with this RFC + +**Rationale:** Security boundaries must not be optional. A plugin can be +disabled, uninstalled, or fail to load — if the ACL evaluator lived in a +plugin, raw memory access would be the fallback state. Core enforcement means +every memory read/write passes through the broker regardless of which plugins +are installed. The enterprise plugin adds *richer identity sources* and +*operational tooling*, not the enforcement itself. + +**The dividing test:** core owns anything whose absence would change an +allow/deny decision — the broker entry point, session-scope resolution, +the evaluator, mount/file-view rules, audit hooks, and deny-by-default +behavior. Plugins own anything whose absence only removes an identity source +or an operational convenience — IdP adapters, group sync, admin UX, anomaly +detection, and (when they land) grant tools. If disabling a component could +widen access, it was never plugin material. + +For single-user local installs, the core resolves a single +`local:{instance}:{user}` principal that owns everything. No configuration +required, no performance overhead beyond the (trivially fast) evaluator +confirming owner === requester. + +This mirrors existing OpenClaw architecture: channels are core +(`src/channels/`) while channel implementations are plugins +(`extensions/telegram/`, `extensions/discord/`). The plugin SDK is core +(`src/plugin-sdk/`) while plugins are extensions. + +### 1. Identity Model + +Sender identity is derived from the channel transport's verified assertion, +resolved through the identity binding table to a canonical user: + +``` +Channel transport assertion (e.g., Teams webhook signature) + → channel_user_id (transport-specific, e.g., Teams AAD object ID) + → channel_identity binding (verified via OAuth/pairing) + → canonical user_id (enterprise IdP: {provider}:{tenant_id}:{user_id}) +``` + +Supported identity providers and their canonical principal format: + +| Provider | Principal format | Source of truth | +|----------|-----------------|-----------------| +| Entra ID / Teams | `entra:{tenant_id}:{object_id}` | Azure AD token claims | +| Okta | `okta:{org_id}:{user_id}` | Okta session / token | +| Google Workspace | `google:{domain}:{user_id}` | Google OAuth subject | +| Local / self-hosted | `local:{instance_id}:{user_id}` | Local pairing code | + +The `channel_identity` table binds transport-asserted sender IDs to these +canonical principals. A Teams user ID, Slack member ID, or Discord snowflake is +never used directly as an ACL principal — it is always resolved to the canonical +form first. + +### 1.1 Session-Scoped Attribution + +ACL principals are resolved at **session granularity, not per tool call**. +OpenClaw already routes every inbound message to a session whose scope is +fixed at routing time by the gateway (group chats are isolated per channel; +DMs are isolated per peer with `session.dmScope: per-channel-peer`). The +gateway stamps the session with its scope at creation, and every memory read +and write inside that session inherits it: + +| Session kind | Scope | ACL principal for reads/writes | +|---|---|---| +| DM with a verified identity binding | `user` | `user:{canonical_user_id}` | +| Group channel | `channel` | `channel:{channel_id}` | +| Cron / webhook / autonomous run | `agent` | `agent:{agent_id}` | + +A mid-run steering message from another member does **not** change the +session principal — "who is the requesting user" is a property of the +session, not of the last message received. Per-message sender identity is +still resolved through the binding table and retained for auditing and for +postbox targeting (§1.2), but it is never the retrieval principal in a group +session. + +**Rationale.** In a shared session the context window is already shared: a +run triggered by Alice and steered by Bob has no single "requesting user." +Attributing individual tool calls to one of them is a fiction that produces +both leaks (Bob reading under Alice's principal because Alice triggered the +run) and breakage (mid-run denials when the sender changes). Attributing at +the session level aligns the ACL boundary with the real information +boundary: the session transcript. + +**Autonomous scopes.** Cron runs, webhooks, and sub-agents carry no user +principal at all. They resolve to `agent:{agent_id}` with access to +tenant-shared and agent-owned items only. Deny-by-default applies: +user-scoped items are invisible to autonomous sessions unless an explicit +grant exists. + +**DM session scope is a precondition, not an assumption.** OpenClaw's +default `session.dmScope: main` shares one DM session across all users — +such a session has no coherent single user and MUST NOT mount a personal +store. Multi-user deployments MUST set `session.dmScope` to +`per-channel-peer` (or stricter); the broker refuses to mount user scopes +for shared-DM sessions regardless of configuration, and `openclaw doctor` +flags multi-user installs that have not set an isolating `dmScope`. + +### 1.2 Multiplayer with Privacy: the Mount Model + +The privacy guarantee in group ("multiplayer") scenarios comes from +controlling **which stores a session mounts**, not from per-message ACL +juggling: + +| Store | DM session (user A) | Group session (channel C) | +|---|---|---| +| A's personal store | read/write | **not mounted** (postbox writes only) | +| Channel C store | not mounted | read/write, live members' sessions only | +| Tenant-shared store | read | read | +| A's shareable projection | read/write (managed here) | read, while A is a live member | + +Four mechanisms make "participate in a group without leaking personal +memory" concrete: + +1. **Channel memory is the group default.** Everything the agent learns in + channel C's session is written channel-scoped (`allow read channel:C`). + It was said in front of the channel, so the channel may remember it. + Members who leave lose access immediately (live-membership check at read + time). + +2. **Personal stores are never readable from group sessions.** "What do you + know about @alice?" asked in a group resolves against channel memory + only. A channel-scoped session never mounts another user's personal + subtree — the files are outside its view and their index rows outside + its scope; this is structural placement (§2), not a row filter. + +3. **Personal postbox — a one-way write valve.** During a group session the + agent may *file* an observation into a member's personal store + (`remember` with `postbox: user:A`), provided A's sender identity was + resolved in this session. The broker writes it as a markdown item file + under `memory/users/{A}/postbox/` with channel provenance in its + frontmatter, `allow read user:A` only, plus lineage back to its + channel-scoped source. The group session cannot read it back — the + postbox subtree is never in a group session's mounts. + This lets the agent get smarter about Alice from group interactions + (visible later in her DM sessions) without widening the group's access. + Nothing confidential is lost at write time: the content originated in + front of the channel. A hostile channel can at worst *pollute* a + member's store, never read it; postbox items carry channel provenance so + the owner can review and purge them. + +4. **Shareable projection — opt-in, audited declassification.** A user may + mark specific personal items as *shareable to channels I am in* (working + hours, pronouns, dietary preferences). These items receive an additional + allow entry scoped to the projection, and group sessions read them only + while the user is a live member of that channel. This is an explicit, + revocable, per-item grant recorded with `granted_by` — never a default, + never inferred by the model. + +The failure mode this prevents: user B in a shared channel pulling A's +inferred preferences merely by sharing a channel with her. B gets channel +memory plus whatever A deliberately projected — nothing else. + +### 1.3 Postbox Trust Tiering + +Hydration is where postbox risk concentrates: a hostile channel member can +steer the agent into filing crafted content into a member's store, and that +content later loads into the member's private DM context — where the agent +may hold broader tool permissions than it did in the channel. A postboxed +item is therefore a **distinct trust tier**, not ordinary personal memory: + +1. **Low-trust by default.** A postboxed item hydrates only as a retrieval + result, rendered with its provenance label (*"filed from #project-x, + 2026-07-06, unreviewed"*). It is never loaded into system-prompt-level / + always-loaded memory (the `MEMORY.md` tier), and it is treated as data, + never as an instruction source. + +2. **Promotion is explicit.** The owner may promote an item into their + curated personal memory — the file moves out of `postbox/` and only then + gains the normal trust tier. Purging is one command + (`openclaw memory postbox --purge --channel `). + +3. **Notification and rate limits.** The first postbox write from a channel + the user has not previously received postbox items from triggers a + notification. Per-channel rate limits (default: 20 items/day) bound + flooding; writes over the limit are dropped and audited. + +4. **Deployment-controlled enablement.** The postbox is a core *mechanism* + with an explicit posture knob — `memoryAcl.postbox.mode`: + + | Mode | Behavior | Default for | + |---|---|---| + | `labeled` | Items retrievable with provenance labels (the tier rules above) | Single-user / local installs | + | `review-required` | Items quarantined — excluded from retrieval — until the owner approves | Installs with an enterprise IdP configured | + | `off` | No postbox writes at all; group sessions cannot touch personal stores in any direction | Available everywhere; recommended until a tenant has validated the notification/review UX | + + The enterprise default is deliberately the conservative one, consistent + with the fail-closed posture elsewhere in this RFC; tenants opt *down* to + `labeled`, not up from it. `openclaw doctor` reports the active mode in + multi-user installs. + +These limits ship with the postbox itself, not as a follow-up (see Resolved +Questions). + +### 2. Memory Partitioning + +**Memory stays markdown files.** OpenClaw's memory source of truth today is +markdown in the agent workspace (`MEMORY.md`, `memory/*.md`), indexed into +SQLite for FTS and vector search. This RFC keeps that model — users can still +read, hand-edit, and git-sync their memory — and makes the **directory tree +the partition primitive**: + +``` +{workspace}/memory/ + users/{canonical_user_id}/ # personal store; mounted rw only in that + ... # user's DM sessions + postbox/ # machine-filed items from group sessions + channels/{channel_id}/ # channel store; mounted rw only in that + # channel's group sessions + shared/ # tenant-shared; read-only everywhere + projections/{user_id}/ # user-curated shareable items; read-only in + # channels the user is a live member of +``` + +Partitioning operates at three levels: + +**Directory-level isolation (structural boundary):** a session's memory view +is the union of the subtrees its scope mounts (§1.2). The group session for +channel C indexes and retrieves from `channels/{C}/`, `shared/`, and live +members' `projections/` — every other subtree is outside its view entirely: +never indexed into its candidate set, not resolvable via `memory_get`, not +part of any retrieval code path. Isolation comes from placement, not from a +per-item policy decision. Cross-user access has no exception path in this +RFC (§4; a grant mechanism is deferred to Appendix A). + +**Index-level scope (SQLite):** the existing memory index gains a `scope` +column derived from the file path at indexing time. The broker's prefilter +and postfilter evaluate scope and ACL against the session's principal set. +The SQLite layer is an **index plus policy catalog** — scope, ACL entries, +lineage, exposure audit, grants — never the content store. Content lives in +the files; the index holds pointers, hashes, and search structures (see the +broker design §3). + +**Row-level ACL (fine grain, only where placement is insufficient):** +per-item ACL entries apply to items in `shared/`, projection items, postboxed +items, and declassified derived items. The common cases — personal and +channel memory — are handled entirely by directory placement, so most items +never need an ACL row at all. + +### 2.1 Session File-View Enforcement (and its limits) + +Path partitioning is only as strong as the session's file and exec +confinement. This RFC therefore requires: + +- `memory_search` and `memory_get` resolve paths strictly inside the + session's mounted subtrees, using the same symlink-safe containment checks + the memory file reader performs today + (`packages/memory-host-sdk/src/host/read-file.ts`). +- File tools (`read`/`write`/`edit`/`apply_patch`) treat unmounted memory + subtrees as outside the workspace for path-validation purposes — a group + session's file tools cannot open `memory/users/*` any more than they can + open a path outside the workspace root. +- For multi-user deployments with `exec` enabled, the exec root (or sandbox + profile) must exclude unmounted memory subtrees. `openclaw doctor` warns + when a multi-user configuration leaves exec unconfined over the memory + tree. + +**Honest threat model.** With an in-process runtime and an unsandboxed exec +tool, this boundary holds against retrieval-path bugs, prompt-level probing, +and ordinary model behavior — not against a deliberately prompt-injected +`exec` reading arbitrary paths. That is consistent with OpenClaw's existing +security posture ("session or memory scoping reduces context bleed, but does +not create per-user host authorization boundaries" — SECURITY.md). Deployments +that need an adversarial boundary against exec-enabled agents must run the +out-of-process broker with sandboxed agent processes described in the broker +design (§8, §12) — that is the hardening tier, not the Phase 0 baseline. +This RFC makes the boundary's strength explicit instead of implying a +guarantee the deployment mode cannot deliver. + +### 3. ACL Evaluation + +#### 3.1 Permission lattice + +Permissions form an explicit implication chain for allows: + +``` +admin ⇒ derive ⇒ read ⇒ retrieve (sync is independent; admin ⇒ sync) +``` + +An allow entry for a stronger permission satisfies requests for the +permissions it implies. Each requested permission has a **required set** that +must be fully satisfied: + +| Requested perm | Required set | +|---|---| +| `retrieve` | `{retrieve}` | +| `read` | `{retrieve, read}` | +| `derive` | `{retrieve, read, derive}` | +| `sync` | `{sync}` | +| `admin` | `{admin}` | + +A request for permission P is **denied** if any deny entry matches any member +of required(P); it is **allowed** only if every member of required(P) is +satisfied — either by **placement** (the item's scope is the session's own +scope, §2: the common case, which carries no ACL rows) or by some allow +entry (directly or via implication). Deny entries are always honored even on +placement-allowed items. Consequences: +`deny retrieve` blocks all content flow for that principal; `deny read` +still permits candidate citation (retrieve) but never content exposure. The +prefilter evaluates required(`retrieve`); the postfilter evaluates +required(`read`) before content enters a prompt — the two stages check +different rungs of the same lattice, by design. + +This lattice is **normative** — the broker design doc, the prefilter SQL, and +the evaluator implementation must all derive from this table, and the +property-test suite asserts the recall invariant against it: **the prefilter +never excludes an item the evaluator would allow** (prefilter results ⊇ +evaluator-allowed). The prefilter is a deliberate over-approximation — it may +pass items the postfilter then denies (that mismatch is the audited drift +signal); the postfilter is the security boundary, so prefilter looseness is a +performance question, never a disclosure. + +#### 3.2 Evaluation order + +This order is normative and shared verbatim with the broker design and the +implementation plan (a single evaluator implements it; divergence between +documents is a spec bug): + +0. **Session scope & identity** — the session envelope is stamped at routing + time by the gateway. For user-scoped (DM) sessions, + `(channel_id, channel_user_id)` must resolve to a verified, unrevoked + binding; no binding = no user principal, denied before any subtree or + index is mounted (`deny-identity`). For group sessions the principal is + the channel; the sender binding is used for membership and postbox + targeting only. + +1. **Hard partition** — `item.tenant_id == ctx.tenant_id`; the item's file + lives in a subtree (and index partition) the session mounts; + `deleted_at IS NULL`; `expires_at` not passed (`deny-partition`). + +2. **Sensitivity ceiling** — `item.sensitivity > ctx.sensitivityCeiling` → + `deny-scope`. Sensitivity is assigned only by trusted producers: runtime + configuration defaults (per channel/agent), explicit values from admin + surfaces, and `max()` over parents on derivation — never from model text. + +3. **Explicit deny wins** — any `deny` entry matching any principal in the + resolved set, for any member of required(P), blocks access + (`deny-explicit`). No override possible. + +4. **Allow required** — every member of required(P) must be satisfied, + either by placement (item's scope == the session's own scope; no ACL + rows needed) or by a matching, unexpired `allow`. Otherwise default deny + (`deny-default`). + +5. **Channel scope narrowing** — channel-scoped items require + `ctx.channel_id == item.channel_id` AND live membership in that channel + (`deny-scope` / `deny-membership`). A user-level grant alone does not make + channel-scoped items visible in other channels. + +6. **Lineage re-check** — derived items walk their ancestor graph; any + tombstoned or ACL-failing ancestor denies (`deny-lineage`). Stale + embedding candidates (embedded rev ≠ current rev) are dropped from vector + results (`deny-stale`). + +#### 3.3 Structural write-back controls + +The broker enforces what is structurally enforceable: + - The `derive` API requires parent item IDs; `derive` permission is checked + on each parent against the requesting principal. Explicit re-attribution + through the derivation path is blocked. + - `remember` calls cannot specify an owner other than the session's + resolved scope (the broker ignores/rejects mismatched owner fields). + The only exception is the postbox path (§1.2), which *narrows* the + audience to a single member's personal scope — it can never widen it. + - Directory-level isolation prevents reads of another user's subtree + entirely; postbox writes go through the broker's write path, which + files the item into the target's postbox directory — a session never + holds a direct write handle into a foreign subtree. + + These controls prevent *structured* re-attribution but cannot prevent + *semantic* laundering (model rephrases content as a "new" observation). + Operational mitigations (write-after-exposure anomaly detection, semantic + similarity flagging) supplement the structural boundary. See Non-Goals §1 + for the full enforcement analysis. + +### 4. Cross-User Access (Deferred) + +Cross-user access is **deny-by-default, and this RFC ships no exception +path**: there is no mechanism by which one user's sessions read another +user's memory. The mount model never opens another user's subtree, and the +supported ways to share across users are deliberate placement — `shared/` +(tenant-wide) or a per-item projection (§1.2). + +A direct grant mechanism (Alice hands Bob time-bounded, revocable, audited +read access to a slice of her memory — e.g., a project handover) is a real +need, but it drags in its own UX and policy questions: how a user issues, +reviews, and reasons about grants; delegation; admin override; revocation +residuals. Rather than bolt a thin version onto this proposal, the grant +design is **deferred**. A worked sketch is preserved in Appendix A so the +schema and evaluation semantics stay forward-compatible with it; nothing in +Appendix A is normative until it graduates into a proposal section. + +Cross-instance federation remains out of scope entirely (Non-Goals §6). + +### 5. Enterprise IdP Integration + +When an enterprise identity provider is configured: + +```json +{ + "identity": { + "provider": "entra", + "tenantId": "contoso.onmicrosoft.com", + "clientId": "...", + "discovery": "https://login.microsoftonline.com/{tenantId}/v2.0/.well-known/openid-configuration" + } +} +``` + +The pairing flow uses the IdP's OAuth token to verify the user's identity and +create the `channel_identity` binding: + +1. User initiates pairing via channel command (`/pair`) or web flow. +2. OpenClaw redirects to IdP OAuth consent. +3. On callback, extracts `tid` (tenant) + `oid` (user object ID) from the + ID token. +4. Writes `channel_identity` row: `(channel_id, channel_user_id) → entra:{tid}:{oid}`. +5. Subsequent messages from that `channel_user_id` resolve to the canonical + principal automatically. + +For Teams specifically, the bot framework already provides the AAD object ID in +the activity envelope — the binding can be established on first interaction +without an explicit pairing step, since the transport itself is the IdP. + +### 5.1 Adapter Registry Hardening + +Identity resolution is the most security-critical input to the ACL system, +and the enterprise plugin supplies it. An unguarded registry would let a +malicious or compromised plugin register an adapter that resolves any sender +to any principal — which core would then faithfully enforce. Three rules +close this: + +1. **Core verifies; adapters only fetch.** Adapters never return principal + strings. The adapter contract returns raw verification material — the + token as received plus the provider's pinned discovery/JWKS metadata — + and **core** validates signature, issuer, audience, and expiry, then + constructs the canonical principal (`entra:{tid}:{oid}`, …) from the + verified claims itself. A lying adapter can deny service; it cannot forge + identity. Group refresh results are namespaced per provider + (`group:{provider}:{tenant}:{gid}`) and constructed by core, so an + adapter can only influence group principals under its own allowlisted + prefix. + +2. **Provider allowlist in operator config.** A plugin may serve a provider + prefix only if operator-owned configuration names it explicitly + (`memoryAcl.identity.providers: { "entra": "memory-acl-enterprise" }`). + Unlisted registrations are refused, duplicate registrations for a prefix + are refused, and every registration — accepted or refused — is + audit-logged. + +3. **Registration is startup-only.** Adapters register during plugin + initialization from manifest-declared capabilities, and the registry is + sealed before the first session is routed. There is no runtime swap of + identity resolution. + +### 6. Retrieval Flow Integration + +The retrieval pipeline from the Memory Broker design gains ACL enforcement: + +``` +Agent request (query + session envelope, stamped at routing time) + → Resolve PolicyContext (session scope → principal set; DM: verified binding) + → Resolve session mounts (subtrees + index scopes per §1.2/§2) + → FTS5 + Vector candidates (prefiltered by ACL CTE, required(retrieve)) + → Postfilter: per-item ACL re-eval (required(read)) + lineage check + + channel membership + → Write-back check: tag exposed items for derivation prohibition tracking + → Trim to token budget + → Audit exposure (item, rev, principal) + → Return context block +``` + +### 7. Session Transcripts and Compaction + +The memory store is not the only durable representation of a conversation. +Session transcripts are persisted and reloaded across turns, and compaction +(summarizing a long session to fit the context window) produces derived +artifacts. Without rules here, a compaction summary of a multi-user session +would be a derived memory with no lineage, no ACL, and no broker +involvement — channel memory that skipped every control in this RFC. + +1. **Transcripts carry the session's scope.** A session's transcript is + state of that session's scope: a group transcript is channel-scoped, a + DM transcript user-scoped. Transcript files on disk live under the same + scoped layout and fall under the same session file-view enforcement + (§2.1) as memory subtrees — a group session's transcript is not readable + from another user's DM session, and a user's DM transcript is never + readable from a group session. + +2. **Compaction artifacts are derived memories.** A compaction summary MUST + be written through the broker as a `derive` operation whose source is + the transcript segment it summarizes: it inherits the session's scope + (channel-scoped for group sessions), records lineage back to that + segment, and is subject to normal retrieval evaluation thereafter. + Compaction never persists an unscoped artifact, and no code path outside + the broker writes summaries to disk. + +3. **In-session sharing is by construction, not a leak.** Within a live + group session, all members see the shared context — that is what a group + conversation is, and no memory ACL changes it. The rules above govern + *persistence and reuse*: what the transcript becomes once it outlives + the session. + +4. **Revocation residuals are enumerable, and scrubbing is implementable.** + When an access path is withdrawn — a member leaves a channel, a + projection is revoked, or (once the deferred grant design of Appendix A + lands) a grant is revoked — content already exposed persists in the + affected sessions' transcripts. The exposure audit records exactly which + item revisions entered which sessions, so a revocation-triggered scrub — + redacting those items' content from retained transcripts of affected + sessions — is a well-defined operational job. This RFC requires the + audit linkage that makes the scrub possible and defines the scrub as an + optional operational tool, not a retroactive guarantee. + +## Rationale + +**Why core enforcement rather than a plugin?** +A plugin can be disabled, uninstalled, or fail to load. If the ACL layer were a +plugin, the failure mode would be "all memory accessible without authorization" +— the worst possible default. Core enforcement means the broker is the *only* +path to memory, always. Enterprise features (IdP adapters, grant UX, anomaly +detection) are deployment-specific and belong in a plugin that registers into +the core framework, not replaces it. + +**Why session-scoped partitioning over tag-based classification?** +Tag-based systems require every write path to correctly classify content. A +forgotten tag or misconfigured default silently over-shares. Session-scoped +partitioning is structural — the scope is inherent in where the conversation +happened (which session, fixed at routing time by the gateway), not in a +policy decision that must be made per-item, and not in a per-tool-call guess +about which participant is "the" requester. Tags can layer on top for finer +control, but the base isolation must be scope-driven. + +**Why directory-level isolation for users?** +A WHERE clause bug, SQL injection, or ORM misconfiguration in a +single-store-with-row-filters model exposes all users. Separate subtrees with +per-scope index partitions mean a bug in ACL evaluation cannot cross a user +boundary — the unmounted subtree's files are never read and its index rows +are never in the queried partition. This is the same rationale as process +isolation vs. thread isolation, applied to the store OpenClaw actually has: +markdown files plus a derived index. + +**Why enterprise IdP principals rather than channel-native IDs?** +Channel-native IDs (Discord snowflakes, Slack member IDs) are platform-specific +and can't federate. Enterprise IdPs provide stable, cross-platform identity that +HR already manages. When Alice leaves the company, disabling her Entra account +revokes all her memory grants without touching each channel individually. + +**Why prohibit write-back rather than attempting to track downstream usage?** +Once content enters a model's context, tracking what the model "does" with it +is intractable. The enforceable boundary is the broker's write path: the broker +knows who is writing and can check lineage. Attempting to restrict in-context +reasoning would require model-level enforcement that doesn't exist reliably. + +## Resolved Questions + +1. **Offline/degraded IdP behavior: fail closed, with a bounded cache.** + Deny-by-default must hold under degraded conditions, so fail-open is not an + acceptable default. When the IdP is unreachable, the broker operates from + the last-known group/binding snapshot **only within a bounded staleness + window** (default 24 hours). Within the window, cached group principals are + honored and every use is audited with a `stale` flag. Beyond the window, + group-derived principals are dropped from the resolved set — `user:` and + `tenant:` principals (anchored by the locally stored, verified binding) + are retained, so personal memory keeps working while group-gated access + tightens. Deployments may shorten the window or opt into strict + fail-closed (drop group principals immediately). Loosening beyond the + bounded default is **break-glass, not configuration**: the value is named + `fail-open-unsafe`, it is never a peer of the defaults, and `openclaw + doctor` reports it as a security finding (a standing deny-by-default + violation), not a notice. + +2. **Postbox abuse limits: ship with the postbox, as a trust tier.** + Resolved in §1.3 — postboxed items are low-trust by default + (provenance-labeled retrieval only, never system-prompt-level memory), + promotion is explicit, first-contact notification and per-channel rate + limits (default 20/day) are built in, and `postbox.mode` controls the + posture: `labeled` (single-user default), `review-required` (default when + an enterprise IdP is configured), or `off` (postbox disabled entirely). + +## Unresolved Questions + +1. **Group principal semantics.** Should Entra security groups / Okta groups + map directly to ACL principals, or should OpenClaw maintain its own group + abstraction? Direct mapping is simpler but couples tightly to IdP schema. + +2. **Delegation depth for cross-user grants** *(applies to the deferred + grant design, Appendix A)*. Can a grantee re-grant to a third user? + Sketch position: no — grants are non-transferable; a new grant from the + original owner is required. + +3. **Tenant admin override scope** *(applies to the deferred grant design, + Appendix A)*. Should a tenant admin be able to grant cross-user access to + any user's memory, or only tenant-shared items? This has significant + privacy implications for personal workspace content. + +4. **Channel membership freshness.** How frequently should the broker refresh + channel membership state from the transport? Real-time (webhook-driven) vs. + periodic polling vs. on-demand check at retrieval time. Each has different + latency/cost/consistency trade-offs. + +5. **Teams auto-binding policy.** §5.1's core-side token verification means a + bare envelope object ID can no longer bind an identity, but the policy + question remains: should first-interaction binding (from a verified Bot + Framework token) be per-tenant opt-in, and should step-up pairing be + required before an auto-bound identity can issue cross-user grants? + (See risk 4 in the Security Posture section.) + +## Prior Art + +- [#96883 — Scope agent cron operations to the calling agent](https://github.com/openclaw/openclaw/pull/96883) (merged): + Establishes the Gateway-side caller-identity plumbing pattern this RFC's + session-scoped attribution builds on. Introduces a runtime identity token + (`src/gateway/agent-runtime-identity-token.ts`), a caller-context extractor + for Gateway RPC (`src/agents/tools/gateway-caller-context.ts`), and + server-side scope enforcement that limits agent-tool calls to their own + resources while preserving unscoped operator access. The memory broker's + `SessionEnvelope` would extend this same `callerScope` pattern from + `agentId` to the session's resolved user/channel principal, and the + evaluator would consume it the way cron's server methods do — scoped for + agent/user sessions, unscoped for operator/admin paths. + +## Implementation + +[Implementation plan](0010/implementation-plan.md) — component breakdown, +dependency graph, phase mapping, and integration points with existing OpenClaw +code. + +## Security Posture & Risk Assessment + +This section records the honest security posture of the proposal as written: +what the design enforces, what it deliberately does not, and the residual +risks ranked by severity. Reviewers should evaluate the RFC against this +assessment rather than against an implied stronger guarantee. + +### What the design enforces + +- **Layered boundaries in the right order.** Structural placement first + (unmounted subtrees are never read; unmounted index partitions are never + attached), then index-level scope, then per-item ACL, with the postfilter + as the authoritative check — property-tested (the prefilter never excludes + what the evaluator allows; results ⊇ evaluator-allowed) and monitored + (postfilter-deny-after-prefilter-pass as a drift alarm). +- **Deny-by-default survives degradation.** Fail-closed group snapshots with + a bounded staleness window; no identity binding → no user principal; + autonomous sessions carry no user scope. +- **Attribution cannot be confused mid-run.** Session scope is stamped at + routing time and immutable; steering by another member never switches the + principal; a `user_id` field is unrepresentable in the envelope. +- **Cross-user flows only narrow.** The postbox narrows channel → one + member; there is no cross-user read path at all in this RFC (grants + deferred, Appendix A); derived items take the intersection of parent + labels; declassification is explicit and audited. +- **Limits are stated in-text** (§2.1): in-process enforcement with an + unsandboxed `exec` tool is policy hygiene, not an adversarial boundary. + +### Residual risks (ranked) + +| # | Risk | Severity | Status | +|---|------|----------|--------| +| 1 | **In-process exec ceiling.** All controls hold against retrieval-path bugs and ordinary model behavior; a deliberately prompt-injected `exec` can read unmounted subtrees. The adversarial boundary is the out-of-process broker + sandboxed agents (broker design §12 Option B), which is sketched, not designed. | High | Accepted for Phase 0; **Option B is a MUST for deployments with untrusted channel members**. Stated in §2.1. | +| 2 | **Session transcripts bypass the broker.** The group transcript is shared context, persisted and reloaded; compaction summaries of multi-user sessions would otherwise be derived memories with no lineage, ACL, or broker involvement. | High | **Addressed in §7.** Transcripts carry the session's scope under §2.1 file-view enforcement; compaction artifacts MUST be written through the broker as scoped `derive` operations with lineage; revocation scrub is enabled by audit linkage. | +| 3 | **Identity injection via the IdP adapter registry.** A malicious or compromised plugin that registers an adapter could resolve any sender to any principal, and core would faithfully enforce the forged identity. | High | **Addressed in §5.1.** Adapters return raw token + pinned JWKS material only; core verifies and constructs principals; operator-config provider allowlist; duplicate registrations refused; registry sealed at startup. | +| 4 | **Teams auto-binding.** Binding on first interaction makes the enterprise identity exactly as strong as the channel plugin's webhook validation; a forged activity envelope becomes a silent identity binding — the root credential for everything else. | Medium-High | **Open.** Should be per-tenant opt-in with an audit event; step-up pairing required before an auto-bound identity can issue grants. (§5.1's core-side token verification narrows this: the envelope's object ID alone is no longer sufficient to bind.) | +| 5 | **Postbox poisoning as persistent prompt injection.** A hostile channel member can steer the agent into postboxing crafted content into a member's store; it later hydrates into that user's private DM context, where the agent may hold broader tool permissions. | Medium-High | **Addressed in §1.3.** Postboxed items are a low-trust tier: provenance-labeled retrieval only, never system-prompt-level memory, never an instruction source; explicit promotion; first-contact notification + per-channel rate limits; `postbox.mode` posture knob — `review-required` quarantine is the enterprise-IdP default, and `off` disables the mechanism entirely. | +| 6 | **`dmScope` default undermines user-scoped sessions.** OpenClaw's default (`dmScope: main`) shares one session across all DM users; such a session has no coherent single user yet would be the one mounting a personal store. | Medium | **Addressed in §1.1.** The broker refuses to mount user scopes for shared-DM sessions unconditionally; multi-user deployments MUST set `per-channel-peer` or stricter; `openclaw doctor` flags violations. | +| 7 | **Index partitions are content-bearing at rest.** The catalog holds pointers, but `memory_fts` stores chunk text and embeddings are invertible — index files under `~/.openclaw/memory-acl/` contain recoverable content. | Low-Medium | Accepted under the OS-FDE baseline (§8 of the broker design); noted so the index directory is not treated as metadata-only. | +| 8 | **Membership freshness window** (Unresolved Q4): an ex-member can read channel memory until the membership snapshot refreshes. | Low-Medium | Open question; webhook-driven refresh narrows the window on Teams/Slack/Discord. | +| 9 | **Audit metadata sensitivity.** `memory_exposure_audit` records who saw what — itself sensitive; no access control on the audit surface is specified. Group snapshot "signing" has no specified key infrastructure. | Low | Open; to be specified with the admin/ops tooling. | +| 10 | **Migration misclaiming.** The `doctor --fix` interactive step that claims legacy unscoped files into a user scope can misclaim; defaulting to `shared/` preserves (never widens) the pre-migration audience. | Low | Accepted; misclaiming narrows audience, so the failure mode is availability, not disclosure. | + +### Explicitly out of scope (restated) + +Semantic laundering through the context window (Non-Goals §1), model-weight +memorization (Non-Goals §4), cross-instance federation (Non-Goals §6), and +content-level DLP (Non-Goals §2) are not addressed by this design; the +mitigations available for them are detection and audit, not prevention. + +### Bottom line + +For the core scenario — a multi-user channel in which no participant can read +another member's or the owner's private memory — the posture is sound at the +retrieval and tool layer and honestly labeled at the exec layer. Risks 2, 3, +5, and 6 are resolved in-document (§7, §5.1, §1.3, §1.1 respectively). Risk 1 +remains the deployment-mode requirement: installs with untrusted channel +members and `exec` enabled need the out-of-process broker tier. Risk 4 +(Teams auto-binding policy) is the remaining open design question of +consequence. + +## Appendix A — Deferred: Cross-User Access Grants + +*Non-normative. Preserved so the schema and evaluation semantics stay +forward-compatible with a future grant mechanism; see §4 for why this is +deferred.* + +The scenario: Alice ran weeks of research with the agent in `#project-x`; +Bob takes over the project and needs her project memory in his own sessions. +Alice grants Bob time-bounded retrieval over exactly that slice: + +``` +Tenant: contoso — one OpenClaw instance, one broker + Alice (entra:contoso:alice) Bob (entra:contoso:bob) + memory/users/alice/ memory/users/bob/ + │ │ + │ ┌───────────────────────────┐ │ + └──│ Access grant │─────┘ + │ grantor: entra:contoso:alice + │ grantee: entra:contoso:bob + │ scope: channel:project-x + │ perm: retrieve + │ expires: 2026-08-01 + └───────────────────────────┘ +``` + +Under such a grant, Bob's user-scoped sessions would mount a **read-only +view** of the granted slice of Alice's subtree (and its index partition), +prefiltered to `scope_channel_id` — Bob sees her project-x items, not her +personal memory. Both users resolve through the same instance's identity +bindings, and the same broker enforces both sides; nothing crosses a host +boundary (that constraint is load-bearing — the broker enforces grants by +opening files it already controls under identities it already verified; +cross-instance sharing is a different problem, Non-Goals §6). + +**Sketched grant structure:** + +```sql +CREATE TABLE memory_access_grant ( + grant_id TEXT PRIMARY KEY, + grantor_principal TEXT NOT NULL, -- who issued the grant + grantee_principal TEXT NOT NULL, -- who receives access + source_tenant_id TEXT NOT NULL, -- belt-and-braces: must equal both users' tenant + scope_channel_id TEXT, -- NULL = all grantor-owned items + perm TEXT NOT NULL CHECK (perm IN ('retrieve','read','derive')), + purpose TEXT, -- human-readable reason + granted_at INTEGER NOT NULL, + expires_at INTEGER NOT NULL, -- mandatory expiry + revoked_at INTEGER, + revoked_by TEXT +); +``` + +**Sketched rules:** +- Only the item owner (source user) or a tenant admin can issue a grant. +- Grants are always time-bounded (`expires_at` required). Permanent grants + accumulate and become unauditable; mandatory expiry forces periodic + re-evaluation, matching enterprise access-review practice. +- Grants are revocable at any time; revocation takes effect immediately for + retrieval. Content already exposed into the grantee's session transcripts + is not retroactively erased; the exposure audit records exactly what was + seen under the grant (§7.4), making the residual exposure enumerable and + scrub-able. +- Grants are exercised only from the grantee's **user-scoped sessions** — a + group session the grantee participates in cannot wield the grant, so a + grant never becomes channel-wide access by accident. +- The broker mounts the grantor's slice read-only; the grantee's sessions + never gain a write path into the grantor's subtree or index partition. +- Exposure is audited with the grant ID as provenance. +- `derive` permission in a grant allows the grantee's agent to produce + summaries, but the derived item's ACL inherits the intersection rule (the + grantee can read it; the grantor's deny rules still apply to further + propagation). + +**Open questions parked with this design:** grant issuance/review UX (how +Alice actually creates, sees, and reasons about her outstanding grants), +delegation depth (sketch: non-transferable), and tenant-admin override scope +(see Unresolved Questions 2–3). diff --git a/rfcs/0010/implementation-plan.md b/rfcs/0010/implementation-plan.md new file mode 100644 index 00000000..5b8e5c04 --- /dev/null +++ b/rfcs/0010/implementation-plan.md @@ -0,0 +1,991 @@ +# RFC 0010 — Implementation Plan: Component Breakdown + +This document maps the ACL-based memory partitioning RFC to concrete components +in the OpenClaw codebase. The architecture splits into **core enforcement** +(always present, cannot be disabled) and an **enterprise plugin** (optional +IdP adapters and operational tooling). + +--- + +## Architecture Split + +``` +┌──────────────────────────────────────────────────────────────────────┐ +│ CORE (src/memory-acl/) │ +│ Always loaded. Cannot be disabled. Security boundary lives here. │ +│ │ +│ ┌──────────────┐ ┌──────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ Principal │ │ ACL │ │ Memory │ │ Prefilter │ │ +│ │ Resolver │ │ Evaluator │ │ Broker │ │ SQL Builder │ │ +│ └──────────────┘ └──────────────┘ └─────────────┘ └─────────────┘ │ +│ ┌──────────────┐ ┌──────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ Lineage │ │ Exposure │ │ Channel │ │ Local IdP │ │ +│ │ Walker │ │ Auditor │ │ Membership │ │ Adapter │ │ +│ └──────────────┘ └──────────────┘ └─────────────┘ └─────────────┘ │ +│ ┌──────────────┐ ┌───────────────────────────────────────────────┐ │ +│ │ IdP Adapter │ │ Schema: acl_entry, lineage_edge, principal, │ │ +│ │ Contract │ │ channel_identity, channel_member, │ │ +│ │ (interface) │ │ exposure_audit (grants: deferred) │ │ +│ └──────────────┘ └───────────────────────────────────────────────┘ │ +└──────────────────────────────────────────────────────────────────────┘ + ▲ implements IdpAdapter contract + │ +┌───────┴──────────────────────────────────────────────────────────────┐ +│ PLUGIN (extensions/memory-acl-enterprise/) │ +│ Optional. Adds richer identity + operational tooling. │ +│ │ +│ ┌──────────────┐ ┌──────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ Entra ID │ │ Okta │ │ Google │ │ Cross-User │ │ +│ │ Adapter │ │ Adapter │ │ Workspace │ │ Grant Tools │ │ +│ └──────────────┘ └──────────────┘ └─────────────┘ └─────────────┘ │ +│ ┌──────────────┐ ┌──────────────┐ ┌─────────────┐ │ +│ │ Anomaly │ │ Group Sync │ │ Admin CLI │ │ +│ │ Detection │ │ (IdP→local) │ │ Commands │ │ +│ └──────────────┘ └──────────────┘ └─────────────┘ │ +└──────────────────────────────────────────────────────────────────────┘ +``` + +**Why this split:** + +| Concern | Core | Plugin | +|---------|------|--------| +| Can it be disabled? | No | Yes | +| What happens without it? | N/A — always present | Local identity only; no enterprise IdP adapters, group sync, or anomaly detection | +| Failure mode if removed | N/A | Graceful: falls back to local adapter; core enforcement unaffected | +| Security boundary? | Yes — broker is the only path to memory | No — adds identity sources, not enforcement | +| Deployment coupling | Ships with OpenClaw | Installed when enterprise IdP is needed | + +--- + +## Existing Seams (what we build on) + +| Seam | Location | What it provides | Used by | +|------|----------|------------------|---------| +| Memory index manager | `extensions/memory-core/src/memory/manager.ts` | Per-agent SQLite memory with FTS + vector search | Core: Broker wraps it | +| Memory DB schema | `src/state/openclaw-agent-schema.sql` | `memory_index_chunks`, `memory_index_sources` | Core: schema additions | +| Channel identity resolution | `src/channels/message-access/runtime-identity.ts` | Normalizes `channel_user_id` → stable identifiers | Core: Principal Resolver | +| Ingress subject model | `src/channels/message-access/types.ts` | `ChannelIngressIdentifierKind` | Core: Principal Resolver | +| Plugin state store | `src/plugin-state/plugin-state-store.sqlite.ts` | Kysely-backed KV with TTL | Plugin: anomaly-detection state (grant metadata when RFC Appendix A lands) | +| Shared state DB | `src/state/openclaw-state-db.ts` | Global `~/.openclaw/state/openclaw.db` | Core: audit, bindings, membership | +| Agent DB registry | `src/state/openclaw-state-db.ts` | `agent_databases` tracks paths | Core: DB file routing | +| Access groups | `src/config/types.access-groups.ts` | Static + dynamic group membership | Plugin: IdP group mapping | +| Plugin SDK | `src/plugin-sdk/plugin-entry.ts` | `definePluginEntry` | Plugin: tool registration | +| Memory host SDK | `packages/memory-host-sdk/src/host/memory-schema.ts` | Schema helpers | Core: schema migrations | +| Scoped file reader | `packages/memory-host-sdk/src/host/read-file.ts` | Symlink-safe containment checks for memory file reads | Core: session file-view enforcement (RFC §2.1), content resolution at exposure time | + +--- + +## Core Components (`src/memory-acl/`) + +### C1: Principal Resolver (`src/memory-acl/principals.ts`) + +**Purpose:** Resolves a **session envelope** into a `PrincipalSet` — the +identity used for all ACL evaluation for the session's lifetime. Attribution +is at **session granularity** (RFC §1.1): the gateway stamps the scope at +routing time; tool calls never carry or construct identity. + +**Builds on:** the gateway's session routing (`session.dmScope`, +group-per-channel isolation) and +`src/channels/message-access/runtime-identity.ts` for sender normalization. + +**Integration note:** memory tools today receive only `agentId` +(`extensions/memory-core/src/tools.ts`) — no per-message sender reaches the +tool layer, and it doesn't need to. The envelope is stamped once at session +creation and read from the session registry; this is deliberately *not* a +per-tool-call plumbing change. + +```typescript +export type PrincipalKind = + | 'user' | 'tenant' | 'group' | 'channel' | 'role' | 'agent' | 'source'; + +export type SessionScope = 'user' | 'channel' | 'agent'; + +export interface PrincipalSet { + readonly tenantId: string; + readonly sessionScope: SessionScope; + readonly userId: string | null; // user-scoped sessions only; NEVER from envelope assertion + readonly channelUserId: string | null; // last resolved sender; audit + postbox only + readonly principals: ReadonlySet; + readonly channelId: string; + readonly agentId: string; + readonly sensitivityCeiling: number; +} + +export interface SessionEnvelope { + // Stamped once by the gateway at session routing time; immutable. + tenantId: string; + sessionKey: string; + sessionScope: SessionScope; + channelId: string; + agentId: string; +} + +export interface SenderRef { + channelUserId: string; // transport-asserted sender (per message) + conversationId?: string; +} + +export interface PrincipalResolver { + resolve(envelope: SessionEnvelope, sender: SenderRef | null): PrincipalSet; +} +``` + +**Key behavior:** +- `user` scope (DM): looks up `channel_identity` to derive `userId` + from the sender; no binding → abort before any DB opens (`deny-identity`) +- `channel` scope (group): principal is `channel:{channelId}`; sender is + resolved for audit rows and postbox targeting only — never for retrieval +- `agent` scope (cron/webhook/sub-agent): `agent:{agentId}` + + `tenant:{tenantId}` only; nothing user-scoped is ever visible +- Delegates to registered `IdpAdapter` for canonical principal format +- Falls back to `LocalIdpAdapter` when no enterprise plugin is loaded +- Group principals come from the snapshot with **fail-closed bounded + staleness** (see P6): stale-within-window → honored + audited; beyond + window → dropped from the set +- **`dmScope` precondition (RFC §1.1):** a `user`-scope resolution is + rejected when the session is a shared DM (`session.dmScope: main`) — such + a session has no coherent single user. The broker refuses to mount user + scopes for shared-DM sessions unconditionally; `openclaw doctor` flags + multi-user installs that have not set `per-channel-peer` or stricter. + +--- + +### C2: ACL Evaluator (`src/memory-acl/evaluator.ts`) + +**Purpose:** Pure, stateless function. Given a `PrincipalSet`, an item header, +and ACL entries, returns a `Decision`. No I/O, no DB access — testable in +isolation. + +```typescript +export type Perm = 'retrieve' | 'read' | 'derive' | 'sync' | 'admin'; + +export type Decision = + | 'allow' + | 'deny-explicit' + | 'deny-default' + | 'deny-scope' + | 'deny-lineage' + | 'deny-partition' + | 'deny-stale' + | 'deny-identity' + | 'deny-membership'; + +export interface AclEntry { + aclId: string; + itemId: string; + principalId: string; + effect: 'allow' | 'deny'; + perm: Perm; + expiresAt: number | null; +} + +export interface ItemHeader { + itemId: string; + tenantId: string; + ownerUserId: string | null; + channelId: string | null; + derived: boolean; + sensitivity: number; + effectiveLabel: string; + deletedAt: number | null; + expiresAt: number | null; +} + +export interface AclEvaluator { + evaluate(ctx: PrincipalSet, item: ItemHeader, acl: AclEntry[], perm: Perm): Decision; + readableLabels(ctx: PrincipalSet): ReadonlySet; +} +``` + +**Permission lattice (normative — from RFC §3.1):** allows imply down the +chain `admin ⇒ derive ⇒ read ⇒ retrieve` (`sync` independent; `admin ⇒ sync`). +Each requested perm P has a required set (required(`read`) = +{retrieve, read}, etc.); a deny matching any member of required(P) denies, +and every member must be covered — by **placement** (item's scope == the +session's own scope; the common case, no ACL rows — RFC §2) or by an allow +entry. Denies are honored even on placement-allowed items. The evaluator, +the prefilter (C6), and the property tests all derive from this one table. + +**Evaluation order (normative — mirrors RFC §3.2 and broker design §4; +divergence between the three documents is a spec bug):** +1. Hard partition: `item.tenantId !== ctx.tenantId`, or item not in a + mounted subtree/partition, or `item.deletedAt` set, or + `item.expiresAt < now` → `deny-partition` +2. Sensitivity ceiling: `item.sensitivity > ctx.sensitivityCeiling` → `deny-scope` +3. Explicit deny: any deny entry matching any member of required(perm) → `deny-explicit` +4. Allow required: any member of required(perm) uncovered by placement or + an allow entry → `deny-default` +5. Channel scope: `item.channelId` set but ≠ `ctx.channelId` or no live membership → `deny-scope` / `deny-membership` +6. Pass: `allow` + +(Step 0 — session scope & identity resolution — happens in C1 before any +subtree or partition is mounted; step 6 of the RFC order — +`deny-lineage`/`deny-stale` — is produced by C4 and the staleness check in +the postfilter, outside this pure function.) + +**Decision producers** (every `Decision` variant has exactly one producer — +no dead policy inputs): +- `sensitivity`: set at `remember` time from per-channel/per-agent runtime + config defaults or trusted admin surfaces; `derive` takes `max()` over + parents. Never from model text. +- `deny-stale`: embedding staleness (`embedding_meta.item_rev ≠ + memory_item.rev`) drops vector candidates in the postfilter. +- `deny-identity` / `deny-membership`: C1 resolution and C7 read-time + membership check respectively. + +--- + +### C3: Memory Broker (`src/memory-acl/broker.ts`) + +**Purpose:** The ONLY surface agents use for memory access. Replaces direct +`MemoryIndexManager` usage. Wraps existing search with ACL enforcement. + +```typescript +export interface MemoryBroker { + query(req: MemoryQuery): Promise; + remember(req: MemoryWrite): Promise; + derive(req: DerivationRequest): Promise; + forget(req: ForgetRequest): Promise; +} + +export interface MemoryQuery { + envelope: SessionEnvelope; + queryText: string; + tokenBudget: number; + k?: number; +} + +export interface MemoryQueryResult { + contextBlock: string; + exposed: ReadonlyArray<{ itemId: string; rev: number }>; + batchId: string; +} +``` + +**Read pipeline:** +1. Resolve `PrincipalSet` (C1, session-scoped) +2. Resolve `SessionMounts` (C10): subtrees + index partitions for the + session's scope + shared (+ live members' projections for group + sessions). No cross-user read mount ships (deferred, RFC Appendix A). +3. Build prefilter CTE (C6, required(`retrieve`)) → fan out to FTS5 + vector +4. RRF merge + score +5. Postfilter: per-item ACL eval (C2, required(`read`)) + lineage check (C4) + + `trust_tier` tagging (C11) so postboxed items render provenance-labeled +6. Trim to token budget +7. Record audit (C5) +8. Return + +**Write path:** +- `remember`: owner = the session's resolved scope; reject any owner field + that disagrees with it. The only cross-scope write is the postbox (C11), + which **narrows** audience to one member — it can never widen scope. +- `derive`: check `derive` perm on every parent; compute effective label as + intersection; write with lineage edges. **Compaction summaries are + `derive` calls** (RFC §7) — they inherit the session's scope with lineage + back to the transcript segment; no summary is persisted outside this path. +- `forget`: tombstone + purge FTS/vec rows + delete item file + quarantine + descendants + +**Integration with existing memory-core:** +``` +Before: memory_search tool → MemoryIndexManager.search() → raw results +After: memory_search tool → MemoryBroker.query() + → PrincipalResolver.resolve() + → prefilter CTE + MemoryIndexManager.search() + → AclEvaluator.evaluate() per item + → ExposureAuditor.record() + → filtered results +``` + +The `MemoryIndexManager` is not removed — it remains the search/index engine. +The broker is a layer above it that adds policy enforcement. + +--- + +### C4: Lineage Walker (`src/memory-acl/lineage.ts`) + +**Purpose:** For derived items, walks the ancestor graph and denies if any +ancestor is tombstoned or fails ACL. + +```typescript +export interface LineageWalker { + checkAncestry(ctx: PrincipalSet, itemId: string, maxDepth?: number): Decision; +} +``` + +**Schema** (added to agent DB alongside existing memory tables): + +```sql +CREATE TABLE lineage_edge ( + child_id TEXT NOT NULL, + parent_id TEXT NOT NULL, + relation TEXT NOT NULL, + created_at INTEGER NOT NULL, + PRIMARY KEY (child_id, parent_id, relation) +); +CREATE INDEX ix_lineage_parent ON lineage_edge(parent_id); +``` + +**Walk:** Recursive CTE, depth-capped at 8. Any tombstoned or ACL-failing +ancestor → `deny-lineage`. + +--- + +### C5: Exposure Auditor (`src/memory-acl/audit.ts`) + +**Purpose:** Records what entered prompts. Enables revocation impact analysis +and write-after-exposure detection. + +**Schema:** `exposure_audit` in the per-tenant `state.db`. The canonical +definition is broker design §3.5; the columns this component relies on are +`exposure_id`, `batch_id` (the `MemoryContextResult.exposureBatchId`), +`agent_id`, `policy_ctx_hash`, `item_id`, `item_rev`, `decision`, `reason`, +and `grant_id` (NULL until the deferred grant design of RFC Appendix A +lands). Do not re-declare a divergent copy here — extend the canonical table. + +Retention: configurable; default 30 days. Compaction deletes `denied_*` rows +older than retention; `exposed` rows kept longer for revocation queries. + +--- + +### C6: Prefilter SQL Builder (`src/memory-acl/prefilter.ts`) + +**Purpose:** Generates a CTE that bounds candidate work before postfilter. +Optimization only — postfilter is the security boundary. + +```typescript +export function buildPrefilterCte(ctx: PrincipalSet, now: number): { + sql: string; + params: Record; +}; +``` + +Generates (candidate stage = required(`retrieve`) from the lattice; the +postfilter, not this CTE, evaluates required(`read`)). The allow stage is +satisfied by placement **or** an allow row — without the placement branch +the prefilter would exclude every ACL-row-free item the evaluator allows, +violating the recall invariant: +```sql +WITH ctx_principals(pid) AS (VALUES (?), (?), ...), +authorized AS ( + SELECT item_id FROM memory_item + WHERE deleted_at IS NULL + AND tenant_id = ? + AND (expires_at IS NULL OR expires_at > ?) + AND (channel_id IS NULL OR channel_id = ?) + -- deny: required(retrieve) = {retrieve}; honored even on placement items + AND NOT EXISTS (SELECT 1 FROM acl_entry d + WHERE ... d.effect='deny' AND d.perm='retrieve' ...) + AND ( + scope = :session_scope -- placement allow: session's own scope, no ACL rows (RFC §2) + OR EXISTS (SELECT 1 FROM acl_entry a + WHERE ... a.effect='allow' + AND a.perm IN ('retrieve','read','derive','admin') ...) + ) +) +``` + +--- + +### C7: Channel Membership (`src/memory-acl/membership.ts`) + +**Purpose:** Tracks live channel membership. Scope-narrowing in C2 queries +this to confirm the requester is a current member of the item's channel. + +**Schema:** `channel_member` in the per-tenant `state.db` — canonical +definition in broker design §3.2 (`channel_id`, `user_id`, `role`, +`joined_at`, `left_at`, PK `(channel_id, user_id)`). `left_at` is the +read-time authority: setting it revokes retrieval of the channel's items +immediately, with no re-labeling. + +**Refresh:** Webhook-driven for Teams/Slack/Discord; periodic poll fallback; +on-demand check as last resort. + +--- + +### C8: Local IdP Adapter (`src/memory-acl/idp-local.ts`) + +**Purpose:** Default adapter for single-user and self-hosted installs. No +enterprise IdP needed — the user owns everything. + +```typescript +// Implements the C9 contract in the degenerate way: there is no token to +// fetch, so the assertion path is skipped, not trusted-by-default — core +// constructs `local:{instanceId}:{user}` from its own pairing state. +export class LocalIdpAdapter implements IdpAdapter { + readonly providerId = 'local'; + async fetchIdentityAssertion(): Promise { + return null; + } + // Single-user: one principal owns all items — evaluator is a trivial pass +} +``` + +--- + +### C9: IdP Adapter Contract (`src/memory-acl/idp-contract.ts`) + +**Purpose:** The interface that enterprise plugins implement to provide richer +identity resolution. Core depends only on this contract, never on specific +IdP implementations. Per RFC §5.1, **adapters fetch raw verification +material; core verifies it and constructs principals** — an adapter can deny +service but cannot forge identity. + +```typescript +export interface IdentityAssertion { + rawToken: string; // token exactly as received from transport/OAuth + jwksUri: string; // from provider discovery, pinned in operator config + expectedIssuer: string; + expectedAudience: string; +} + +export interface IdpAdapter { + readonly providerId: string; + + /** Fetch raw material for a sender. NEVER returns a principal string. */ + fetchIdentityAssertion( + channelId: string, + channelUserId: string, + ): Promise; + + /** Raw provider group IDs; core namespaces them as + * group:{provider}:{tenant}:{gid} — an adapter can only influence group + * principals under its own allowlisted prefix. */ + refreshGroupMembership?(userId: string): Promise; +} + +export interface IdpAdapterRegistry { + /** Refuses unlisted providers (operator allowlist: + * memoryAcl.identity.providers), refuses duplicates, audit-logs every + * registration and refusal. Sealed before the first session is routed. */ + register(pluginId: string, adapter: IdpAdapter): void; + get(providerId: string): IdpAdapter | undefined; + active(): IdpAdapter; // configured adapter, or LocalIdpAdapter +} +``` + +Core performs OIDC verification (signature against the pinned JWKS, issuer, +audience, expiry) in `src/memory-acl/` and builds the canonical principal +from the verified claims. `LocalIdpAdapter` (C8) is the degenerate case: no +token exists, so core constructs `local:{instance}:{user}` from its own +pairing state — the assertion path is skipped, not trusted-by-default. + +The enterprise plugin registers adapters into this registry at startup. Core +always has at least `LocalIdpAdapter` available. + +--- + +### C10: Session Mounts & File-View Enforcement (`src/memory-acl/mounts.ts`) + +**Purpose:** Turns a resolved `PrincipalSet` into the concrete set of memory +subtrees and index partitions a session may touch (RFC §1.2/§2), and +enforces that `memory_search`/`memory_get` and file tools cannot escape it +(RFC §2.1). This is the structural boundary the RFC leans on — placement, +not row filters. + +**Builds on:** the symlink-safe containment checks in +`packages/memory-host-sdk/src/host/read-file.ts` (reused, not reimplemented). + +```typescript +export interface SessionMounts { + readonly writable: ReadonlyArray; // exactly one: the session's own scope + readonly readable: ReadonlyArray; // + shared; + live members' projections (group) + // NOTE: no cross-user read mount — deferred grant design (RFC Appendix A) +} + +export interface ScopeMount { + readonly scope: string; // 'user:{id}' | 'channel:{id}' | 'shared' | 'projection:{id}' + readonly subtreeDir: string; // absolute path under {workspace}/memory/ + readonly indexPartition: string; // absolute path under ~/.openclaw/memory-acl/.../index/ + readonly mode: 'rw' | 'ro'; +} + +export interface MountResolver { + resolve(ctx: PrincipalSet): SessionMounts; + /** Throws if `absPath` is outside every mounted subtree. Used by the + * memory tools and by file-tool path validation for memory/ paths. */ + assertPathMounted(mounts: SessionMounts, absPath: string): void; +} +``` + +**Key behavior:** +- Group sessions never receive a `user:{other}` mount — the personal subtree + is not in `readable`, so it is never indexed into the candidate set nor + resolvable via `memory_get`. +- `assertPathMounted` is wired into `memory_get`/`memory_search` path + resolution and into file-tool path validation for `memory/` paths, so an + unmounted memory subtree is treated exactly like a path outside the + workspace. +- **Honest limit (RFC §2.1):** with in-process, unsandboxed `exec`, this + boundary holds against retrieval-path bugs and model behavior, not against + a deliberately injected shell; the adversarial boundary is the + out-of-process broker (broker design §12 Option B). `openclaw doctor` + warns when a multi-user config leaves `exec` unconfined over `memory/`. + +--- + +### C11: Postbox Write Valve (`src/memory-acl/postbox.ts`) + +**Purpose:** The single write path from a group session into a member's +personal store (RFC §1.2/§1.3). Narrows audience; never widens or reads. + +```typescript +export interface PostboxWrite { + targetUserId: string; // must be a sender resolved in THIS session + content: string; + sourceChannelId: string; // provenance + sourceItemId: string; // lineage parent (channel-scoped source) +} +``` + +**Key behavior:** +- Writes a markdown item under `memory/users/{target}/postbox/` with + `allow read user:{target}` only, channel provenance in frontmatter, and a + `lineage_edge` back to the channel-scoped source — delivered through the + broker write queue, never a directly opened foreign handle. +- **Low-trust tier (RFC §1.3):** postboxed items are tagged so retrieval + renders them as provenance-labeled results only — never promoted into + system-prompt-level (`MEMORY.md`) memory, never treated as an instruction + source — until the owner explicitly promotes them out of `postbox/`. +- **Rate limits + notification:** per-channel cap (default 20/day; over-cap + writes dropped and audited); first postbox from a not-previously-seen + channel emits a notification. +- **Deployment posture (`memoryAcl.postbox.mode`, RFC §1.3):** `labeled` + (default for single-user/local), `review-required` (default when an + enterprise IdP is configured — items quarantined until owner approval), + or `off` (no postbox writes at all). Enterprise defaults conservative; + tenants opt down, not up. `openclaw doctor` reports the active mode in + multi-user installs. + +**Schema addition** (item catalog): + +```sql +ALTER TABLE memory_item ADD COLUMN trust_tier TEXT NOT NULL DEFAULT 'normal'; + -- 'normal' | 'postbox-unreviewed' | 'quarantined' +``` + +--- + +## Enterprise Plugin Components (`extensions/memory-acl-enterprise/`) + +### P1: Entra ID Adapter (`extensions/memory-acl-enterprise/src/entra.ts`) + +**Implements:** `IdpAdapter` contract from core (C9) + +- Fetches the Entra ID token + pinned OIDC discovery/JWKS material; **core** + validates signature/issuer/audience/expiry and extracts `tid` + `oid` + from the verified claims (RFC §5.1) +- Canonical format (constructed by core): `entra:{tenantId}:{objectId}` +- Teams auto-binding: requires a verified Bot Framework token, not a bare + envelope object ID; opt-in policy is Unresolved Q5 +- Group membership: calls Microsoft Graph `/me/memberOf` on refresh; core + namespaces results as `group:entra:{tid}:{gid}` + +--- + +### P2: Okta Adapter (`extensions/memory-acl-enterprise/src/okta.ts`) + +**Implements:** `IdpAdapter` + +- Fetches Okta session/access tokens + pinned introspection/JWKS material; + core validates and extracts `org_id` + `uid` (RFC §5.1) +- Canonical format (constructed by core): `okta:{orgId}:{userId}` +- Group membership: calls Okta `/api/v1/users/{id}/groups` + +--- + +### P3: Google Workspace Adapter (`extensions/memory-acl-enterprise/src/google.ts`) + +**Implements:** `IdpAdapter` + +- Fetches Google ID tokens + pinned OIDC discovery material; core validates + and extracts `hd` (hosted domain) + `sub` (subject) (RFC §5.1) +- Canonical format (constructed by core): `google:{domain}:{subject}` +- Group membership: calls Directory API + +--- + +### P4: Cross-User Grant Tools (`extensions/memory-acl-enterprise/src/grant-tools.ts`) — DEFERRED + +**Status: deferred with the grant design itself (RFC §4 / Appendix A).** +Nothing in this component ships until the grant design graduates from the +appendix; it is kept here so the plugin surface is planned for. + +**Purpose:** Agent-facing tools for managing cross-user grants (e.g., Alice +grants Bob `retrieve` over her `#project-x` items). Registered via plugin +SDK. Cross-instance federation is out of scope (RFC Non-Goals §6). + +Tools: +- `memory_grant` — create a time-bounded grant to another user +- `memory_revoke` — revoke an active grant +- `memory_grants_list` — list grants given/received +- `memory_grant_audit` — show what was accessed under a specific grant + +**Schema** (shared state DB, owned by core but populated by plugin tools): + +```sql +CREATE TABLE memory_access_grant ( + grant_id TEXT PRIMARY KEY, + grantor_principal TEXT NOT NULL, + grantee_principal TEXT NOT NULL, + source_tenant_id TEXT NOT NULL, -- must equal both users' tenant (same instance) + scope_channel_id TEXT, + perm TEXT NOT NULL, + purpose TEXT, + granted_at INTEGER NOT NULL, + expires_at INTEGER NOT NULL, + revoked_at INTEGER, + revoked_by TEXT +); +CREATE INDEX ix_grant_grantee ON memory_access_grant(grantee_principal) + WHERE revoked_at IS NULL; +CREATE INDEX ix_grant_grantor ON memory_access_grant(grantor_principal); +``` + +Grants would be exercised only from the grantee's **user-scoped sessions**; a +group session the grantee participates in could not wield the grant. + +When this design graduates (RFC Appendix A): the grant *schema* lives in +core (the broker queries it during the read path to determine which +additional subtrees/index partitions to mount read-only), while the grant +*management tools* live in the plugin. Until then neither the table nor the +tools exist, and the broker has no grant-aware read path. + +--- + +### P5: Anomaly Detection (`extensions/memory-acl-enterprise/src/anomaly.ts`) + +**Purpose:** Detects potential semantic laundering by comparing new writes +against recently-exposed cross-principal items. + +**How it works:** +1. Hooks into broker `remember` calls (via plugin hook) +2. Queries `exposure_audit` for recent cross-principal exposures +3. Computes embedding similarity between new write and exposed items +4. Flags if similarity > threshold (default 0.85) +5. Advisory only: logs metric + emits event. Does not block the write. + +**Why plugin, not core:** False positives are common (user legitimately writes +about the same topic). Blocking writes would break workflows. This is an +operational signal for security teams, not an enforcement boundary. + +--- + +### P6: Group Sync (`extensions/memory-acl-enterprise/src/group-sync.ts`) + +**Purpose:** Periodically refreshes IdP group membership and updates local +principal snapshots. + +- Calls `IdpAdapter.refreshGroupMembership()` on configurable schedule +- Updates a local `idp_group_membership` table +- Principal resolver includes group principals in the resolved set +- **Fail-closed with a bounded staleness window (default).** When the IdP is + unreachable, the last-known snapshot is honored only within + `maxStalenessHours` (default 24); every use of a stale snapshot is audited + with a `stale` flag. Beyond the window, group-derived principals are + dropped from the resolved set while `user:`/`tenant:` principals (anchored + by the locally stored verified binding) are retained — group-gated access + tightens, personal memory keeps working. A user removed from a group is + therefore locked out no later than the window, even if sync never + recovers. Deployments may shorten the window or drop stale groups + immediately. Unbounded trust in the stale snapshot is **break-glass, not + configuration**: the value is deliberately named `fail-open-unsafe` so it + cannot read as a peer of the defaults, and `openclaw doctor` reports it + as a security finding (a standing deny-by-default violation), not a + notice + +--- + +### P7: Plugin Entry (`extensions/memory-acl-enterprise/index.ts`) + +```typescript +export default definePluginEntry({ + id: 'memory-acl-enterprise', + register(api) { + const config = api.getConfig(); + const registry = api.getService('memory-acl:idp-registry'); + + // Register enterprise IdP adapters. The registry enforces the operator + // allowlist (memoryAcl.identity.providers → plugin id), refuses + // duplicates, and audit-logs the registration (RFC §5.1); a plugin + // cannot self-authorize a provider prefix. + if (config.identity?.provider === 'entra') + registry.register('memory-acl-enterprise', new EntraIdpAdapter(config.identity)); + else if (config.identity?.provider === 'okta') + registry.register('memory-acl-enterprise', new OktaIdpAdapter(config.identity)); + else if (config.identity?.provider === 'google') + registry.register('memory-acl-enterprise', new GoogleIdpAdapter(config.identity)); + + // P4 grant tools (memory_grant, memory_revoke, memory_grants_list, + // memory_grant_audit) are registered here once the deferred grant + // design (RFC Appendix A) graduates — no tool surface ships until then. + + // Register anomaly detection hook + api.registerHook('memory-write', anomalyDetectionHook(config.anomaly)); + + // Start group sync background job + if (config.groupSync?.enabled) + api.registerService('group-sync', new GroupSyncService(registry, config.groupSync)); + }, +}); +``` + +**Manifest** (`extensions/memory-acl-enterprise/openclaw.plugin.json`): + +```json +{ + "id": "memory-acl-enterprise", + "activation": { "onCapabilities": ["memory-acl"] }, + "contracts": { + "tools": [] + }, + "configSchema": { + "type": "object", + "properties": { + "identity": { + "type": "object", + "properties": { + "provider": { "enum": ["entra", "okta", "google"] }, + "tenantId": { "type": "string" }, + "clientId": { "type": "string" }, + "clientSecret": { "type": "string" } + }, + "required": ["provider", "tenantId", "clientId"] + }, + "anomaly": { + "type": "object", + "properties": { + "threshold": { "type": "number", "default": 0.85 } + } + }, + "groupSync": { + "type": "object", + "properties": { + "enabled": { "type": "boolean", "default": false }, + "intervalMinutes": { "type": "number", "default": 60 }, + "stalePolicy": { "enum": ["fail-closed", "fail-open-unsafe"], "default": "fail-closed" }, + "maxStalenessHours": { "type": "number", "default": 24 } + } + } + } + } +} +``` + +--- + +## Dependency Graph (revised with core/plugin split) + +``` +┌─────────────────────────────────────────────────────────────────────────┐ +│ PLUGIN: extensions/memory-acl-enterprise/ │ +│ │ +│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌──────────┐ ┌───────────┐ │ +│ │P1 Entra │ │P2 Okta │ │P3 Google│ │P4 Grant │ │P5 Anomaly │ │ +│ │ Adapter │ │ Adapter │ │ Adapter │ │ Tools │ │ Detection │ │ +│ └────┬────┘ └────┬────┘ └────┬────┘ └────┬─────┘ └─────┬─────┘ │ +│ └─────────────┼───────────┘ │ │ │ +│ │ implements │ │ │ +└─────────────────────┼─────────────────────────┼───────────────┼────────┘ + │ │ │ +══════════════════════╪═════════════════════════╪═══════════════╪════════════ + │ │ │ +┌─────────────────────┼─────────────────────────┼───────────────┼────────┐ +│ ▼ CORE: src/memory-acl/ │ │ │ +│ ┌──────────────┐ │ │ │ +│ │ C9: IdP │ │ │ │ +│ │ Contract │ │ │ │ +│ └──────┬───────┘ │ │ │ +│ │ used by │ │ │ +│ ┌──────▼───────┐ │ │ │ +│ │ C1: Principal│ │ │ │ +│ │ Resolver │ │ │ │ +│ └──────┬───────┘ │ │ │ +│ │ │ │ │ +│ ┌──────▼────────────────────────────────────────────────┐ │ +│ │ C3: Memory Broker │ │ +│ │ (the single entry point) │ │ +│ └─┬─────┬─────┬─────┬─────┬─────┬─────┬────────────────┘ │ +│ │ │ │ │ │ │ │ │ +│ ┌──────▼┐ ┌──▼──┐ ┌▼───┐ ┌▼────┐ ┌──▼───┐ ┌▼──────┐ ┌───────┐│ +│ │C2 ACL │ │C4 │ │C5 │ │C6 │ │C7 │ │C10 │ │C11 ││ +│ │Eval │ │Linge│ │Audt│ │Prflt│ │Membr │ │Mounts │ │Postbox││ +│ └───────┘ └─────┘ └────┘ └──┬──┘ └──────┘ └───┬───┘ └───────┘│ +│ │ │ │ +│ ┌──────────────────────▼─────────────────▼─────────┐ │ +│ │ C8: Local IdP Adapter (default, always available) │ │ +│ │ │ +│ ┌────────────────────────────▼─────────────────────┐ │ +│ │ Existing: MemoryIndexManager │ │ +│ │ (extensions/memory-core/ — unchanged engine) │ │ +│ └──────────────────────────────────────────────────┘ │ +└──────────────────────────────────────────────────────────────────────┘ +``` + +--- + +## Phase Mapping + +| Phase | Scope | Components | Exit criteria | +|-------|-------|-----------|---------------| +| **0 — Core enforcement** | Core | C1, C2, C3, C5, C6, C8, C10 | Property tests: prefilter never excludes evaluator-allowed items (results ⊇ evaluator-allowed); cross-channel blocked; spoofed identity rejected; personal subtree unreachable from a group session; shared-DM session refuses user-scope mount; single-user install works with zero config | +| **1 — Lineage & compaction** | Core | C4, schema: `lineage_edge`; compaction-as-derive integration (RFC §7) | Tombstoned ancestor poisons descendants; summary-leak red-team green; compaction summary inherits session scope with lineage | +| **2 — Channel membership** | Core | C7, schema: `channel_member` | Leave-channel → immediate retrieval denial; webhook-driven refresh for Teams/Slack/Discord | +| **3 — Postbox** | Core | C11, schema: `trust_tier` | Group session files into member's postbox; item invisible to the group; hydrates low-trust in owner's DM; rate limit + notification; review-required quarantine | +| **4 — Enterprise IdP** | Plugin | P1, P2, P3, P6, P7 | Entra/Okta/Google binding e2e (core-side token verification, §5.1); registry allowlist rejects unlisted/duplicate providers; group membership resolves to principals; stale-IdP fail-closed window enforced (removed user loses group access ≤ `maxStalenessHours`) | +| **5 — Anomaly detection** | Plugin | P5 | High-similarity write after cross-user exposure → flag raised; no false-positive blocking | +| **— Cross-user grants (DEFERRED)** | Core schema + Plugin tools | Deferred until the grant design (RFC Appendix A) graduates. Core: grant schema, broker grant-aware read path. Plugin: P4 | Grant lifecycle e2e; expired/revoked denied; grant unusable from group sessions; audit correlation works | + +**Single-user installs (no enterprise plugin):** Phases 0–2 give full +enforcement with `LocalIdpAdapter`. One principal owns everything; the evaluator +is a trivial pass. No performance overhead, no configuration. The security +invariant (all access through broker) is established from Phase 0. + +--- + +## Integration with Existing Code + +### How core replaces direct memory access + +The broker becomes the **mandatory** intermediary. The existing `memory_search` +and `memory_get` tools in `extensions/memory-core/` are re-pointed to go through +the broker rather than calling `MemoryIndexManager` directly: + +``` +Current flow: + memory_search tool → MemoryIndexManager.search() → results + +Phase 0 flow: + memory_search tool → MemoryBroker.query() + → C1: resolve principal set + → C6: build prefilter CTE + → MemoryIndexManager.search(with prefilter) + → C2: postfilter each candidate + → C5: audit + → results +``` + +The `MemoryIndexManager` remains the search/index engine, and **markdown +files remain the content store** (RFC §2): the broker's catalog rows are +pointers (`path` + span + `content_hash`), content is read at exposure time +through the symlink-safe scoped reader +(`packages/memory-host-sdk/src/host/read-file.ts`), and `memory_get` path +resolution is confined to the session's mounted subtrees (RFC §2.1). No +existing memory-core code is deleted — it's wrapped. + +### Channel identity bridge + +Existing `ChannelIngressSubject` from `src/channels/message-access/` carries +`stable-id` — the transport-asserted sender. The Principal Resolver (C1) takes +this and resolves through `channel_identity` to get the canonical +principal. Existing identity normalization is reused without modification. + +### Session transcripts & compaction (RFC §7) + +The memory store is not the only durable representation of a conversation; +transcripts and compaction summaries must not bypass the broker. + +- **Transcript placement.** Session transcript files live under the same + scoped layout as memory subtrees and fall under C10 file-view enforcement + — a group transcript is channel-scoped and unreadable from another user's + DM session; a DM transcript is never readable from a group session. +- **Compaction is derivation.** The runtime's compaction/summarization step + writes through the broker as a `derive` call (C3) whose parent is the + transcript segment; the summary inherits the session's scope with lineage. + This is the one integration point in the existing agent loop that must be + re-pointed at the broker — a summary written directly to disk would be an + unscoped, unlineaged laundering channel. +- **Revocation scrub.** Because C5 records which item revisions entered + which sessions, a revocation-triggered transcript scrub is a well-defined + optional job (not a retroactive guarantee). + +### Store layout (backward-compatible migration path) + +**Phase 0 (immediate):** memory files stay exactly where they are. For +single-user installs the whole existing `memory/` tree is one scope +(`user:{local}`), so there are **no file moves and no behavior change** — +the scope column and ACL/catalog tables are added alongside the existing +index tables in the per-agent DB +(`~/.openclaw/agents/{agentId}/agent/openclaw-agent.sqlite`), and the +broker treats that DB as the single scope's index partition. + +**Multi-user (doctor migration):** enabling multi-user memory runs +`openclaw doctor --fix`, which reorganizes the memory tree into the scoped +subtree layout (`memory/users/{id}/`, `memory/channels/{id}/`, +`memory/shared/`, `memory/projections/{id}/` — RFC §2), splits the index +into per-scope partitions +(`~/.openclaw/memory-acl/tenants/{tenantId}/index/...`), and enables session +file-view enforcement. Existing un-scoped files migrate to `shared/` by +default with an interactive option to claim them into a user scope — the +conservative default is the *wider* store here because pre-migration content +was already visible to every session, so `shared/` preserves, not widens, +its existing audience. The index is rebuilt from files, never trusted across +the migration. + +### Grant schema ownership (deferred with the grant design) + +When the grant design graduates (RFC Appendix A), the `memory_access_grant` +table lives in the **shared state DB** and is created by a core schema +migration. The broker reads it during the read path to determine which +additional subtrees/index partitions to mount read-only. The enterprise +plugin provides the *tools* to manage grants, but enforcement doesn't depend +on the plugin being loaded. Until then, no grant table exists and the broker +has no grant-aware code path. + +--- + +## Testing Strategy + +| Layer | Scope | Approach | +|-------|-------|----------| +| ACL evaluator | Core | Property-based: fuzz principal sets × ACL entries × items. Invariant: prefilter SQL never excludes what evaluator allows. | +| Prefilter SQL | Core | Generate random ACL state, run prefilter, assert results ⊇ evaluator decisions. | +| Lineage walker | Core | Depth graphs with tombstoned ancestors; verify poisoning propagates correctly. | +| Channel scope | Core | Leave-channel → immediately denied. Join → immediately allowed. | +| Broker integration | Core | End-to-end: multi-user scenario with channel isolation, personal items, shared items. | +| Single-user passthrough | Core | LocalIdpAdapter: confirm zero-config single-user has no regressions vs. current behavior. | +| Mounts & file-view (C10) | Core | Group session never mounts a `user:{other}` subtree; `assertPathMounted` rejects `memory_get` / file-tool paths outside the session's mounts (incl. symlink-escape attempts via the reused reader checks). | +| Postbox tiering (C11) | Core | Postboxed item invisible to the writing group session; hydrates low-trust (never `MEMORY.md`, never instruction) in owner's DM; over-cap writes dropped; `mode: review-required` quarantines until approval; `mode: off` → postbox writes rejected outright. | +| Compaction scoping | Core | Compaction summary of a group session is written via `derive`, inherits `channel:C` scope + lineage; no unscoped summary reaches disk. | +| Identity binding | Core + Plugin | Spoofed `channel_user_id` → denied. Revoked binding → denied. Valid Entra token → resolves. Adapter returning a forged principal string is impossible (contract returns raw material; core builds the principal); unlisted/duplicate provider registration refused. | +| Cross-user grants (deferred) | Core + Plugin | With the Appendix A design: grant → access from grantee's user-scoped session. From a group session → denied. Expire → denied. Revoke → denied. Re-grant attempt by grantee → rejected. Until then: assert no cross-user read path exists at all. | +| Session attribution | Core | Group session: retrieval principal is `channel:C` regardless of who triggered/steered the run. Steering by B mid-run never switches to B's user principal. Autonomous session → no user-scoped items. Postbox write lands with `user:A` allow only; unreadable from the writing session. | +| Anomaly detection | Plugin | High-similarity write after exposure → flag. Low-similarity → no flag. Advisory only. | +| E2E scenario | All | Multi-user Teams: shared channel memory visible to members; personal stores unreadable from the group session; postbox items land in the member's store and are invisible to the group; user leaves → channel memory revoked; user A's memory never retrievable from user B's sessions. | + +--- + +## File Layout Summary + +``` +src/memory-acl/ + index.ts -- exports, wires core components + principals.ts -- C1: PrincipalResolver + evaluator.ts -- C2: AclEvaluator (pure function) + broker.ts -- C3: MemoryBroker + lineage.ts -- C4: LineageWalker + audit.ts -- C5: ExposureAuditor + prefilter.ts -- C6: buildPrefilterCte + membership.ts -- C7: channel membership state + idp-local.ts -- C8: LocalIdpAdapter + idp-contract.ts -- C9: IdpAdapter interface + registry (core-side verification) + mounts.ts -- C10: SessionMounts + file-view enforcement + postbox.ts -- C11: postbox write valve + trust tiers + schema.sql -- all core ACL schema additions + evaluator.test.ts -- property-based tests + broker.test.ts -- integration tests + prefilter.test.ts -- recall invariant: prefilter results ⊇ evaluator-allowed + mounts.test.ts -- file-view escape + cross-user non-mount tests + +extensions/memory-acl-enterprise/ + index.ts -- P7: plugin entry + openclaw.plugin.json -- manifest + src/ + entra.ts -- P1: Entra ID adapter + okta.ts -- P2: Okta adapter + google.ts -- P3: Google Workspace adapter + grant-tools.ts -- P4: grant management tools (DEFERRED — RFC Appendix A) + anomaly.ts -- P5: anomaly detection hook + group-sync.ts -- P6: IdP group refresh + types.ts -- shared plugin types +``` diff --git a/rfcs/0010/openclaw-memory-broker-design.md b/rfcs/0010/openclaw-memory-broker-design.md new file mode 100644 index 00000000..1ffa24d5 --- /dev/null +++ b/rfcs/0010/openclaw-memory-broker-design.md @@ -0,0 +1,737 @@ +# OpenClaw Memory Broker — Local-First, ACL-Enforced Memory on Markdown Files with a SQLite Index + +## 1. Design Principles + +0. **Files are the store; SQLite is the index.** OpenClaw memory is markdown + the user can read, hand-edit, and git-sync. The broker does not replace + that store — it catalogs it. Every SQLite row about content is derived + state (pointer + hash + search structures) that the indexer rebuilds from + the files; policy state (ACL, lineage, audit, bindings — and grants once + the deferred grant design of RFC Appendix A lands) is authoritative in + SQLite. +1. **Deny by default.** A memory item is invisible unless an allow rule matches the resolved principal set *and* no deny rule matches. Absence of ACL rows = unreadable (for scopes where ACL rows apply; directory placement covers the rest). +2. **The model never queries the index directly.** All retrieval flows through the Memory Broker, which resolves policy context, filters at the SQL layer, re-filters post-retrieval, and audits exposure. There is no "raw query" tool surface. File access to memory subtrees is confined to the session's mounts — see RFC §2.1 for the enforcement requirements and their honest limits. +3. **Isolation by construction before isolation by policy.** Tenant, user, and channel isolation is achieved with **separate directory subtrees and per-scope index partitions**, not WHERE clauses. ACLs handle the finer grain (shared items, projections, postboxed and declassified items) *within* a mounted scope. A bug in ACL evaluation cannot cross a scope boundary — the unmounted subtree's files are never read and its index partition is never attached. +4. **Labels flow with data.** Derived memories (summaries, extractions, embeddings-of-summaries) inherit the **intersection** (most-restrictive meet) of their ancestors' authorizations. Lineage is a first-class table, and it is re-checked at read time, not only at write time. +5. **Two-phase enforcement.** Enforce ACLs in the candidate query (prefilter) *and* re-evaluate item-by-item after retrieval (postfilter). The prefilter is an optimization; the postfilter is the security boundary. This makes vector-index limitations a recall problem, never a disclosure problem. +6. **Every exposure is recorded.** If an item's content reached a prompt, there is an audit row saying which item revision, under which policy context, for which request. This makes revocation impact analysis ("who saw this after it should have been deleted?") a query, not forensics. + +--- + +## 2. Local File Layout & Isolation Model + +Content lives in the workspace memory tree (the store users already know); +broker state lives under `~/.openclaw/`: + +``` +{workspace}/memory/ # markdown source of truth (RFC §2) + users/{user_id}/ # personal store; mounted rw only in that + ... # user's DM sessions + postbox/ # machine-filed items from group sessions + channels/{channel_id}/ # channel stores; mounted rw only in that + # channel's group sessions + shared/ # tenant-shared; read-only everywhere + projections/{user_id}/ # opt-in shareable items + +~/.openclaw/memory-acl/ + broker.lock # exclusive-writer advisory lock (whole runtime) + tenants/ + {tenant_id}/ + index/ + users/{user_id}.db # per-scope index partition: item catalog, + channels/{channel_id}.db # ACL rows, lineage, FTS (at-rest: OS FDE, §8) + shared.db + vectors/ # sqlite-vec, split per scope (see §7) + state.db # bindings, membership, exposure audit + # (grants: deferred, RFC Appendix A) + sync-outbox.db # append-only op log for cloud reconciliation +``` + +**Why per-scope subtrees + per-scope index partitions (recommended), vs. alternatives:** + +| Option | Isolation | Ops complexity | Cross-scope queries | Failure blast radius | +|---|---|---|---|---| +| **A. Subtree + index partition per scope (user / channel / shared)** | Strong: a cross-scope leak requires mounting the wrong subtree, not a bad predicate | Moderate: broker fans out to the session's mounted partitions per request | Broker-side merge (cheap at local scale) | One corrupt partition loses one scope; rebuildable from files | +| B. One store, scopes as rows + WHERE clauses | ACL-only between scopes | Simple | Native SQL | Corruption or SQL-injection-class bug exposes all users | +| C. Partition per (user × channel) | Strongest | Partition explosion; lineage spans partitions; painful re-derivation | Expensive multi-partition merges | Small | + +Option A is the right trade-off for a local agent runtime: user and channel boundaries are the highest-consequence boundaries and get placement-level enforcement; finer distinctions (shared items, projections, postboxed items) are policy-shaped and belong in ACL rows within a partition. Option C makes derived memories that summarize across channels nearly impossible to manage. Option B is acceptable only for single-user installs — where it is also the current state of the world, which is why single-user Phase 0 is a no-op migration (one scope, one partition). + +A session mounts at most: its scope's subtree + index partition, the shared partition, and live members' projections (group sessions, read-only). There is no cross-user read mount in this design (read-only grant slices for user sessions are part of the deferred grant design, RFC Appendix A). It never attaches another scope's partition for writing. Because index partitions are derived from files, a lost or corrupt partition is rebuilt by reindexing — the blast radius of index corruption is availability, not confidentiality or data loss. + +--- + +## 3. Schema + +One schema, instantiated per index partition. All timestamps are Unix epoch +ms; all IDs are ULIDs (sortable, mergeable on sync). + +**The catalog is derived state.** `memory_item` rows point at markdown files; +the file is authoritative for content, the row is authoritative for policy +metadata (scope, ACL, lineage, tombstones). The indexer — memory-core's +existing watcher/sync machinery — rewrites rows when files change; a +`content_hash` mismatch between row and file marks the row stale (reindex, +and exclude from results until refreshed). `forget` tombstones the row *and* +removes the item file through the broker's write path in the same operation. + +### 3.1 Item catalog and provenance + +```sql +PRAGMA journal_mode = WAL; +PRAGMA foreign_keys = ON; +PRAGMA busy_timeout = 5000; +PRAGMA synchronous = NORMAL; -- FULL if durability > latency + +CREATE TABLE source ( + source_id TEXT PRIMARY KEY, -- ULID + kind TEXT NOT NULL, -- 'file' | 'chat' | 'tool' | 'web' | 'agent' + uri TEXT, -- file path, URL, tool invocation id + content_hash TEXT, -- sha256 of raw source at ingestion + channel_id TEXT, -- originating channel, NULL = user-global + conversation_id TEXT, + agent_id TEXT, -- which agent/runtime ingested it + created_at INTEGER NOT NULL, + deleted_at INTEGER, -- tombstone; NULL = live + delete_reason TEXT -- 'user' | 'retention' | 'revocation' +); + +CREATE TABLE memory_item ( + item_id TEXT PRIMARY KEY, -- ULID + rev INTEGER NOT NULL DEFAULT 1, -- bumped on content change + tenant_id TEXT NOT NULL, -- redundant with placement; belt-and-braces check + scope TEXT NOT NULL, -- 'user:{id}' | 'channel:{id}' | 'shared' | + -- 'projection:{user_id}'; derived from path at + -- indexing time; redundant with partition placement + path TEXT NOT NULL, -- memory-tree-relative markdown file (SOURCE OF TRUTH) + span_start INTEGER, -- chunk span within the file (lines); NULL = whole file + span_end INTEGER, + content_hash TEXT NOT NULL, -- sha256 of the chunk at indexing time; + -- mismatch with file ⇒ stale row ⇒ reindex, exclude + owner_user_id TEXT, -- NULL for shared-scope items owned by org + kind TEXT NOT NULL, -- 'observation' | 'fact' | 'summary' | 'preference' | 'task' + source_id TEXT REFERENCES source(source_id), + channel_id TEXT, -- originating channel; NULL = user-global + conversation_id TEXT, + agent_id TEXT, -- agent that authored it + sensitivity INTEGER NOT NULL DEFAULT 0, -- 0=default,1=confidential,2=restricted; monotonic max over lineage + derived INTEGER NOT NULL DEFAULT 0, -- 1 if produced from other items (see lineage) + effective_label TEXT NOT NULL, -- canonical hash of computed effective ACL (write-time cache, §5) + created_at INTEGER NOT NULL, + updated_at INTEGER NOT NULL, + expires_at INTEGER, -- TTL memories + deleted_at INTEGER -- tombstone (file removed via broker write path) +); +CREATE INDEX ix_item_scope ON memory_item(channel_id, conversation_id) WHERE deleted_at IS NULL; +CREATE INDEX ix_item_path ON memory_item(path) WHERE deleted_at IS NULL; +CREATE INDEX ix_item_source ON memory_item(source_id); +CREATE INDEX ix_item_label ON memory_item(effective_label) WHERE deleted_at IS NULL; +``` + +Note there is **no `content` column**: the row is a pointer (`path` + +`span` + `content_hash`). Content is read from the file at exposure time +through the symlink-safe scoped reader, after the postfilter allows it — +which also means a row that outlives its file dangles harmlessly instead of +leaking a stale copy. + +### 3.2 ACL + +```sql +CREATE TABLE principal ( + principal_id TEXT PRIMARY KEY, -- 'user:u_123', 'group:eng', 'tenant:t_1', + -- 'channel:c_ops', 'role:admin', 'agent:coder', + -- 'source:s_abc' (grants keyed to a source) + kind TEXT NOT NULL, -- 'user'|'group'|'tenant'|'channel'|'role'|'agent'|'source' + display_name TEXT +); + +CREATE TABLE acl_entry ( + acl_id TEXT PRIMARY KEY, + item_id TEXT NOT NULL REFERENCES memory_item(item_id) ON DELETE CASCADE, + principal_id TEXT NOT NULL REFERENCES principal(principal_id), + effect TEXT NOT NULL CHECK (effect IN ('allow','deny')), + perm TEXT NOT NULL CHECK (perm IN ('read','retrieve','derive','sync','admin')), + -- 'retrieve' = may appear in candidate search; 'read' = content may enter a prompt; + -- 'derive' = may be summarized/combined; 'sync' = may leave the device. + granted_by TEXT, -- provenance of the grant + created_at INTEGER NOT NULL, + expires_at INTEGER +); +CREATE INDEX ix_acl_item ON acl_entry(item_id, effect, perm); +CREATE INDEX ix_acl_principal ON acl_entry(principal_id, effect, perm); + +-- Channel identity binding: maps the sender ID asserted BY the channel transport +-- (Discord user id, WhatsApp JID, Slack member id, Telegram id) to the canonical user. +-- This table lives in the tenant state DB (state.db); written only by the pairing flow. +CREATE TABLE channel_identity ( + channel_id TEXT NOT NULL, + channel_user_id TEXT NOT NULL, -- transport-asserted sender identity + user_id TEXT NOT NULL, -- canonical OpenClaw user + tenant_id TEXT NOT NULL, + verified_method TEXT NOT NULL, -- 'pairing-code' | 'oauth' | 'admin-link' + verified_at INTEGER NOT NULL, + revoked_at INTEGER, -- unlink / account compromise + PRIMARY KEY (channel_id, channel_user_id) +); +CREATE INDEX ix_chan_ident_user ON channel_identity(user_id) WHERE revoked_at IS NULL; + +-- Live channel membership (group channels, DMs). Read-time authority for channel scope. +CREATE TABLE channel_member ( + channel_id TEXT NOT NULL, + user_id TEXT NOT NULL, + role TEXT NOT NULL DEFAULT 'member', -- 'member' | 'admin' + joined_at INTEGER NOT NULL, + left_at INTEGER, -- leaving = immediate retrieval revocation + PRIMARY KEY (channel_id, user_id) +); +``` + +Separating `retrieve` from `read` matters for derived flows: an agent may be allowed to *know an item exists and cite its ID* (retrieve) without its content entering the prompt (read), and `derive` gates whether summarizers may consume it at all. The perms form an implication lattice (`admin ⇒ derive ⇒ read ⇒ retrieve`; `sync` independent) with per-request required sets — see §4 and RFC §3.1 for the normative rules the prefilter and postfilter both derive from. + +### 3.3 Lineage / graph + +```sql +CREATE TABLE lineage_edge ( + child_id TEXT NOT NULL REFERENCES memory_item(item_id), + parent_id TEXT NOT NULL REFERENCES memory_item(item_id), + relation TEXT NOT NULL, -- 'summarizes' | 'extracts' | 'merges' | 'refutes' | 'links' + created_at INTEGER NOT NULL, + PRIMARY KEY (child_id, parent_id, relation) +); +CREATE INDEX ix_lineage_parent ON lineage_edge(parent_id); +``` + +### 3.4 FTS5 and embeddings + +```sql +-- FTS holds the indexed chunk text, keyed to the catalog row. Populated by +-- the indexer (memory-core's existing file watcher/sync), not by triggers — +-- the catalog has no content column to trigger on. Catalog row + FTS row + +-- vector row are written/removed in one indexer transaction. +CREATE VIRTUAL TABLE memory_fts USING fts5( + content, + tokenize='porter unicode61' +); +-- rowid of memory_fts == rowid of the corresponding memory_item row. + +CREATE TABLE embedding_meta ( + item_id TEXT PRIMARY KEY REFERENCES memory_item(item_id), + item_rev INTEGER NOT NULL, -- rev embedded; staleness check = rev mismatch + model TEXT NOT NULL, -- 'text-embedding-3-small' etc. + dim INTEGER NOT NULL, + created_at INTEGER NOT NULL +); + +-- sqlite-vec (vectors.db, optionally ATTACHed): +CREATE VIRTUAL TABLE vec_memory USING vec0( + item_id TEXT PRIMARY KEY, + embedding float[768], + channel_id TEXT, -- aux/partition columns for coarse prefilter + effective_label TEXT +); +``` + +Soft-deletes (`deleted_at` set, row retained) must **also** remove the FTS row and the vector row immediately, and delete the item file (or chunk span) through the broker's write path — a tombstoned row that still matches FTS, or an orphaned file a session can still `memory_get`, is the classic deleted-source leak. Do the index removals in one transaction with the tombstone, the file removal in the same broker operation, then rely on the postfilter as backstop. Conversely, a file edited or deleted *out of band* (hand-edit, git pull) is reconciled by the indexer: hash mismatch → stale → excluded until reindexed. Out-of-band edits are a feature (it's the user's store), not a bypass — the file's placement still determines its scope. + +### 3.5 Sync metadata and audit + +```sql +CREATE TABLE sync_op ( + op_seq INTEGER PRIMARY KEY AUTOINCREMENT, -- local monotonic + hlc TEXT NOT NULL, -- hybrid logical clock for merge ordering + op_kind TEXT NOT NULL, -- 'upsert_item'|'acl_add'|'acl_remove'|'tombstone'|'lineage_add' + entity_id TEXT NOT NULL, + payload BLOB NOT NULL, -- canonical JSON incl. ACL + lineage snapshot + synced_at INTEGER -- NULL = pending +); + +-- Canonical definition of the exposure audit table (the implementation plan's +-- C5 shows a subset view of the same table). Lives in the per-tenant state.db, +-- not a per-scope partition. +CREATE TABLE exposure_audit ( + exposure_id TEXT PRIMARY KEY, + batch_id TEXT NOT NULL, -- MemoryContextResult.exposureBatchId (one query = one batch) + request_id TEXT, -- runtime request correlation, when available + agent_id TEXT NOT NULL, + policy_ctx_hash TEXT NOT NULL, -- hash of resolved principal set + scope + item_id TEXT NOT NULL, + item_rev INTEGER NOT NULL, + decision TEXT NOT NULL, -- 'exposed'|'retrieved_not_read'|'denied_prefilter'|'denied_postfilter'|'denied_lineage' + reason TEXT, + prompt_hash TEXT, -- hash of assembled context block + grant_id TEXT, -- provenance under deferred grants (RFC Appendix A); NULL until it lands + created_at INTEGER NOT NULL +); +CREATE INDEX ix_audit_item ON exposure_audit(item_id, created_at); +CREATE INDEX ix_audit_batch ON exposure_audit(batch_id); +``` + +Auditing *denials* (sampled, if volume matters) is deliberate: postfilter denials with prefilter passes are your leading indicator that the prefilter has drifted from the policy — i.e., a latent leak you caught. + +--- + +## 4. ACL Model & Evaluation Semantics + +**Policy context resolution — session-granularity, channel identity is the input, user identity is derived.** The policy context is resolved **once, at session creation**, by the gateway's routing layer — not per tool call. A session's scope is fixed at routing time (group chats route to channel-scoped sessions; DMs route to user-scoped sessions under `session.dmScope`), and every memory operation in the session inherits it. A mid-run steering message from a different member does not mutate the session principal; per-message sender identity is resolved for audit and postbox targeting only. This avoids the shared-session attribution trap: a run triggered by Alice and steered by Bob has no single "requesting user," and pretending otherwise yields either leaks or mid-run denials. + +For a **user-scoped (DM) session**, the resolver: + +1. Verifies transport authenticity at the channel adapter (webhook signature / bot session) — `channel_user_id` is only as trustworthy as the transport that asserted it. +2. Resolves `channel_user_id → user_id` via `channel_identity` (must be verified, not revoked). **An envelope-supplied `user_id` is never trusted on channel-origin requests** — the binding table is the sole authority. No binding ⇒ the session runs as an *anonymous channel principal* with access to nothing user-scoped (or is rejected, per tenant policy). +3. Only then mounts that user's subtree + index partition and builds the user-scoped context. + +For a **channel-scoped (group) session**, the principal is `channel:{channel_id}`; no member's personal DB is ever mounted for reading. Live membership (`channel_member.left_at IS NULL`) is re-confirmed at read time for each requesting member's visibility into channel-scoped items. For **autonomous sessions** (cron, webhook, sub-agent) the principal is `agent:{agent_id}` — no user principal exists. + +``` +PolicyContext = { + tenant_id, + session_scope, -- 'user' | 'channel' | 'agent'; stamped at routing, immutable + user_id, -- user-scoped sessions only; derived from channel_identity, never asserted + channel_user_id, -- last resolved sender; retained for audit + postbox targeting + principals: { scope-derived: user:u | channel:c | agent:a, plus tenant:t, group:g*, role:r* }, + scope: { conversation_id, channel_id, purpose: 'retrieve'|'derive'|'sync' } +} +``` + +Group/role membership comes from a local, signed snapshot of the identity provider state (refreshed on sync), never from model-supplied claims. Snapshot staleness is **fail-closed with a bounded window**: within the window (default 24h) cached group principals are honored and audited as stale; beyond it, group principals are dropped while `user:`/`tenant:` principals are retained. The agent identity is asserted by the runtime, not the prompt. + +**Permission lattice (normative, shared with RFC §3.1):** allows imply down the chain `admin ⇒ derive ⇒ read ⇒ retrieve` (`sync` independent; `admin ⇒ sync`). Each requested perm P has a required set — required(`retrieve`) = {retrieve}, required(`read`) = {retrieve, read}, required(`derive`) = {retrieve, read, derive} — and a deny matching **any** member of required(P) denies, while every member must be covered by an allow. The prefilter evaluates required(`retrieve`); the postfilter evaluates required(`read`) before content enters a prompt. + +**Evaluation order (per item, per permission — normative, identical in RFC §3.2 and the implementation plan; one evaluator implements it):** + +0. **Session scope & identity** (precede everything): the session envelope is stamped at routing time. For user-scoped sessions, the `(channel_id, channel_user_id) → user_id` binding is verified and unrevoked. Failure here aborts before any DB file is opened — there is no context to evaluate items against. (`deny-identity`) +1. **Hard partition checks** (non-ACL, cannot be overridden): `item.tenant_id == ctx.tenant_id`; the item's file lives in a subtree — and its row in an index partition — that the session's mount set includes; `deleted_at IS NULL`; `expires_at` not passed. (`deny-partition`) +2. **Sensitivity ceiling.** `item.sensitivity > ctx.sensitivityCeiling` → `deny-scope`. Sensitivity has exactly three trusted producers: per-channel/per-agent runtime config defaults at `remember` time, explicit values from admin surfaces, and `max()` over parents at `derive` time. Model text never sets sensitivity. +3. **Explicit deny wins.** Any `deny` entry matching any principal in the context, for any member of required(P), → `deny-explicit`. No allow can override. +4. **Allow required.** Every member of required(P) must be satisfied — by **placement** (the item's scope is the session's own scope; the common case, which carries no ACL rows — RFC §2) or by a matching, unexpired `allow` (directly or via lattice implication) → allowed. Otherwise `deny-default`. Deny entries are honored even on placement-allowed items. +5. **Scope narrowing.** Even when allowed, channel-scoped items (`channel_id NOT NULL`) are only returned when `ctx.channel_id == item.channel_id` **and** the requesting member has live membership in that channel (`channel_member.left_at IS NULL`, checked at read time), or an explicit `allow` exists for `channel:{ctx.channel_id}`. A grant to a *user* does not silently make a channel-scoped item visible in other channels, and leaving a channel revokes retrieval of its items immediately without any re-labeling. (`deny-scope` / `deny-membership`) +6. **Lineage & staleness.** Derived items re-check ancestry (§5) — `deny-lineage`; rev-mismatched vector candidates are dropped — `deny-stale`. + +**Precedence summary:** session scope/identity → hard partition → sensitivity ceiling → deny → allow → scope narrowing (channel match + live membership) → lineage/staleness → default deny. + +**Group-channel semantics (the mount model, RFC §1.2).** In a multi-member channel C, the session principal is `channel:C` and the visibility rules follow from what the session mounts, not from per-message classification: (a) *channel memory* — facts observed in C's shared conversation — is written with `allow read` for `channel:C`, so any live member retrieves it while in C; (b) *personal stores are never mounted for reading in channel-scoped sessions* — there is no code path that opens member A's DB inside C's session, so "what do you know about @A" resolves against channel memory only; (c) the **postbox** lets C's session file an observation into member A's personal store as a write-only operation (`allow read user:A`, lineage back to the channel source, delivered through the broker's write queue — never a directly opened foreign handle). The postbox narrows audience, never widens it; a hostile channel can pollute, not read — and postboxed items land in a **low-trust tier** (RFC §1.3): they hydrate only as provenance-labeled retrieval results, never into system-prompt-level memory and never as an instruction source, are rate-limited per source channel, and gain normal trust only by explicit owner promotion — with the whole mechanism deployment-controlled via `postbox.mode` (`labeled` | `review-required` | `off`; RFC §1.3); (d) a member may opt specific personal items into a **shareable projection** readable by channels they are a live member of — an explicit, audited, revocable declassification, never a default. This is what prevents user B in the same group chat from pulling A's inferred preferences merely by sharing a channel — same-channel presence grants channel memory plus A's deliberate projections, never A's personal store. + +**Why explicit-deny-wins rather than most-specific-wins:** most-specific-wins (à la NTFS canonical ordering) is friendlier for admins but requires a total specificity ordering over heterogeneous principal kinds (is `channel:x` more specific than `group:y`?). In a system where grants are frequently machine-written by agents, a simple, order-independent rule is easier to prove correct and to replicate identically on the sync server. The cost — you can't allow a subgroup under a group-level deny — is acceptable; model those cases by removing the deny and writing narrower allows. + +--- + +## 5. Derived Memories & Label Propagation + +The most dangerous leak path in agent memory is not raw retrieval; it's **laundering through derivation**: restricted content → summary → summary indexed with looser scope → retrieved everywhere. + +**Rules:** + +1. **Derivation is gated by `derive` permission.** The summarizer agent's policy context must be allowed `derive` on every input item. Inputs failing the check are excluded *before* the summarization prompt is built (and audited as `denied`). +2. **Write-time label computation.** A derived item's authorization is the **meet (intersection)** of its parents: + - allows: intersect (a principal must be allowed on *all* parents to be allowed on the child); + - denies: union (any parent's deny applies to the child); + - `sensitivity`: max over parents; + - `channel_id`: if parents span multiple channels, the child gets *no* single channel — it gets explicit allows for exactly those channels, and is invisible elsewhere. + The result is materialized as `acl_entry` rows on the child plus `effective_label` (a canonical hash) for cheap prefiltering and vector partitioning. +3. **Read-time lineage re-check.** Write-time labels go stale when a parent's ACL tightens or a parent is tombstoned. On retrieval of any `derived=1` item, walk ancestors (recursive CTE, depth-capped) and deny if any ancestor is tombstoned or now fails the context's `read` check: + +```sql +WITH RECURSIVE anc(id, depth) AS ( + SELECT parent_id, 1 FROM lineage_edge WHERE child_id = :item + UNION + SELECT le.parent_id, anc.depth + 1 + FROM lineage_edge le JOIN anc ON le.child_id = anc.id + WHERE anc.depth < 8 +) +SELECT EXISTS ( + SELECT 1 FROM anc JOIN memory_item mi ON mi.item_id = anc.id + WHERE mi.deleted_at IS NOT NULL -- tombstoned ancestor poisons descendants +); +``` + +4. **Quarantine + re-derivation.** When a source or item is deleted/revoked, a background job marks descendants `quarantined` (denied at retrieval), then re-runs derivation *without* the revoked parent where possible, producing a new item with a fresh label. This preserves utility ("weekly summary minus the revoked doc") without leaking. +5. **Trade-off to make explicit:** strict intersection makes broadly-useful summaries over mixed-scope inputs nearly unreachable. The escape hatch is *deliberate declassification*: a human (or a policy with a named approver) writes a new allow on the derived item, recorded with `granted_by`, auditable. Never automatic. +6. **Compaction is derivation (RFC §7).** Session-compaction summaries are derived memories of the transcript: they MUST be written through the broker as `derive` operations that inherit the session's scope (channel-scoped for group sessions) with lineage back to the summarized transcript segment. No code path outside the broker persists a summary — otherwise compaction is a laundering channel that bypasses every rule above. + +--- + +## 6. Retrieval Flow + +```mermaid +flowchart TD + A[Agent runtime request\nquery + conversation ctx] --> B[Resolve PolicyContext\nprincipals, scope, purpose] + B --> C1[FTS5 candidates\nACL-prefiltered CTE] + B --> C2[Vector candidates\npartition/label prefilter + overfetch] + C1 --> D[Merge & score\nRRF fusion, recency/importance boost] + C2 --> D + D --> E[Postfilter: per-item ACL re-eval\n+ lineage re-check + rev/staleness check] + E --> F[Trim: token budget, dedupe,\nsensitivity-aware ordering] + F --> G[Build prompt context block] + G --> H[Write exposure_audit rows] + H --> I[Return context to runtime → model] +``` + +**Prefilter CTE** (the optimization layer — bounds candidate work). The +candidate stage evaluates required(`retrieve`) from the permission lattice +(RFC §3.1): the deny check matches denies on `retrieve` itself, and the +allow stage is satisfied by **placement** (the item's scope is the session's +own scope — the common case, which carries no ACL rows, RFC §2) or by any +allow that *implies* retrieve (`retrieve`, `read`, `derive`, `admin`). +Without the placement branch the prefilter would exclude every ACL-row-free +item the evaluator allows — violating the recall invariant below: + +```sql +WITH ctx_principals(pid) AS (VALUES (:p1),(:p2),(:p3) /* resolved set */), +authorized AS ( + SELECT mi.item_id + FROM memory_item mi + WHERE mi.deleted_at IS NULL + AND mi.tenant_id = :tenant + AND (mi.expires_at IS NULL OR mi.expires_at > :now) + AND (mi.channel_id IS NULL OR mi.channel_id = :channel) + -- deny stage: required(retrieve) = {retrieve}; honored even on + -- placement-allowed items + AND NOT EXISTS (SELECT 1 FROM acl_entry d + WHERE d.item_id = mi.item_id AND d.effect='deny' + AND d.perm = 'retrieve' + AND d.principal_id IN (SELECT pid FROM ctx_principals)) + AND ( + -- placement allow: item lives in the session's own scope (RFC §2); + -- no ACL row exists or is needed + mi.scope = :session_scope + -- row allow: shared / projection / postbox / declassified items + OR EXISTS (SELECT 1 FROM acl_entry a + WHERE a.item_id = mi.item_id AND a.effect='allow' + AND a.perm IN ('retrieve','read','derive','admin') + AND (a.expires_at IS NULL OR a.expires_at > :now) + AND a.principal_id IN (SELECT pid FROM ctx_principals)) + ) +) +SELECT mi.item_id, bm25(memory_fts) AS score +FROM memory_fts +JOIN memory_item mi ON mi.rowid = memory_fts.rowid +JOIN authorized au ON au.item_id = mi.item_id +WHERE memory_fts MATCH :query +ORDER BY score LIMIT :k; +``` + +**Postfilter (the security boundary):** for each surviving candidate, re-run the full ACL evaluator for required(`read`) = {retrieve, read} — a deny on either rung, or a missing allow for either rung, blocks content — plus group snapshot freshness, lineage walk for derived items, embedding-rev staleness check, and the sensitivity ceiling for the requesting agent. Only then does content leave the broker. Items that pass `retrieve` but fail `read` may be cited by ID (audited `retrieved_not_read`) but their content never enters the prompt. The postfilter never trusts the prefilter; deliberate defense in depth against predicate drift between the SQL and the evaluator. + +**Vector candidates and the prefilter-leak problem.** ANN indexes generally can't apply arbitrary ACL predicates during traversal. Naive "top-k then filter" has two failure modes: recall collapse (all k neighbors denied) and, worse, implementations that surface neighbor IDs/distances before filtering. Mitigations, layered: + +- **Coarse partitioning for free:** per-scope index partitions mean a session's vector search never even sees other scopes' vectors. +- **In-index metadata filter:** store `channel_id` and `effective_label` as sqlite-vec aux/partition columns; filter to the context's channel + the set of labels the context can read (labels are enumerable because they're hashes of ACL sets — the broker maintains a `label → principal-set` map and computes readable labels per context). +- **Adaptive over-fetch:** request `k × f` (f starting at 4), postfilter, widen f and retry if results < k, hard cap on total scanned. Recall problem solved iteratively; disclosure impossible because nothing bypasses the postfilter. +- Distances/IDs of denied items never leave the broker — even "item X exists with similarity 0.93" is an inference channel. + +**Trim stage:** token budget packing (importance × recency × score), near-duplicate collapse by `content_hash`/simhash, and a per-request **sensitivity ceiling** (e.g., a web-browsing agent may be capped at sensitivity 0 even where the user could read level 1). Items cut at trim are audited `retrieved_not_read` — they were authorized but not exposed, which matters when reconstructing what a model actually saw. + +--- + +## 7. Vector / FTS Abstraction + +``` +ICandidateSource (n) + ├─ Fts5CandidateSource — always available, zero extra deps + ├─ SqliteVecCandidateSource — sqlite-vec loaded as extension, vectors.db + └─ (future) RemoteAnnCandidateSource — only over 'sync'-permitted items +``` + +Keep vectors in a **separate `vectors.db`** ATTACHed at open: (a) sqlite-vec tables are large and rebuildable — excluding them from backup/sync is trivial; (b) an index rebuild (embedding model upgrade) doesn't churn the WAL of the authoritative DB; (c) if you choose not to encrypt vectors identically, the trust boundary is explicit (note: embeddings are invertible enough to be treated as content — encrypt them the same way). + +Staleness contract: `embedding_meta.item_rev != memory_item.rev` ⇒ the vector is for old content. The postfilter drops stale-vector candidates from vector search (they may still arrive via FTS, which is trigger-synced and always fresh); a background embedder repairs the gap. + +--- + +## 8. Encryption & Key Management + +| Option | What it protects | Cost | Notes | +|---|---|---|---| +| **A. OS full-disk / per-file (BitLocker, FileVault, EFS)** | Offline theft | Free; already present on managed devices | **Baseline for Phases 0–2.** No protection from other local processes running as the same user — but neither is SQLCipher when the broker runs in-process with the agent runtime (the keys live in the same process the model steers). | +| B. SQLCipher (whole-file) | DB at rest incl. FTS shadow tables, WAL, freelist | ~5–15% CPU on I/O; **requires a different SQLite driver** | Deferred. OpenClaw's memory stack is `node:sqlite`, which links vanilla SQLite and cannot open SQLCipher databases; adopting SQLCipher means shipping a second native driver on every platform. Worth it only once the broker runs **out-of-process** (Option B in §12), where key isolation is real. FTS5 shadow tables contain plaintext tokens — if/when app-level encryption lands, it must be whole-file, not column-level. | +| C. App-level column encryption | Selected columns | Breaks FTS and vec entirely | Only for narrow secret fields, not memory content. | + +**Honest threat accounting:** with an in-process broker, application-level encryption defends against offline disk access — the same threat OS FDE already covers — while adding a native-driver dependency and key-management surface. The marginal value appears only with process separation (broker holds keys, agent processes never see them). Therefore: Phases 0–2 rely on OS FDE plus file permissions (`0700` on `~/.openclaw/memory-acl/` and the workspace `memory/` tree); the key-hierarchy design below is specified now so the out-of-process broker can adopt it without schema changes. + +**Key hierarchy (for the out-of-process phase):** OS keystore root key (DPAPI/Keychain/TPM-backed) → wraps per-tenant KEK (`tenant.kek` file) → wraps per-DB DEK. Rotation of KEK = rewrap DEKs (cheap, no data rewrite). Rotation of a DEK = `VACUUM INTO` a re-keyed file (offline, per-user granularity — another payoff of per-user files). Keys are held in memory only inside the broker process; child agent processes never receive them. + +--- + +## 9. Concurrency Model + +- **Single broker process, single logical writer.** `broker.lock` acquired exclusively at startup (`FileStream` with `FileShare.None`, plus a named mutex on Windows); a second runtime instance gets a clear "broker already running — connecting as client" path (local IPC: UDS/named pipe) rather than a second SQLite writer. +- **WAL mode** with `busy_timeout=5000`: readers never block the writer, writer never blocks readers. Broker keeps a small read-connection pool and exactly **one** write connection. +- **Write serialization in-process:** all mutations flow through a `Channel` consumed by one loop that batches ops into transactions (amortizes fsync, guarantees ordering, gives natural backpressure). Item write + ACL rows + FTS trigger effects + `sync_op` append are one transaction — memory can never exist without its ACL. +- **Checkpointing:** passive auto-checkpoint plus a periodic `wal_checkpoint(TRUNCATE)` during idle; monitor WAL size as an operational signal. +- **Multi-process defense in depth:** even with the lock, run with `PRAGMA locking_mode=NORMAL` and treat `SQLITE_BUSY` as retryable; corruption from a rogue second writer is prevented by the lock, not by hope. + +--- + +## 10. Sync Model (Eventual Cloud Reconciliation) + +Goals: reconcile devices/cloud **without ever widening local access** and without shipping content the item's ACL forbids leaving the device. + +- **Outbox, not state diffing.** `sync_op` is an append-only op log with HLC timestamps; the sync engine ships ops whose subject items carry an `allow sync` grant. Items without `sync` permission (or `deny sync`) simply never leave — local-only memory is a first-class concept, not a flag on the transport. +- **Payloads carry policy.** Each op includes the item, its ACL rows, and lineage edges as a unit. The server **re-validates**: it recomputes derived labels from lineage and rejects ops whose claimed ACL is broader than the recomputed one (defense against a compromised client widening scope for everyone else). +- **Merge rules (order-independent, same as local semantics):** deny ∪ deny; allow ∩ allow on conflicting concurrent ACL edits (tighten on conflict — availability suffers, confidentiality doesn't); content conflicts resolve LWW by HLC with both revisions retained; **tombstones dominate** any concurrent content update and propagate with a grace window before physical purge so late writers converge on deletion. +- **Inbound ops are re-filtered locally**: applying a remote op never bypasses the broker's write path, so a malicious/buggy server cannot inject an item into a scope the local policy forbids (e.g., a cross-tenant item is rejected at the hard-partition check). +- Embeddings are not synced; each device re-embeds (keeps model/dim heterogeneity sane and avoids shipping invertible vectors). + +--- + +## 11. Failure Modes & Mitigations + +| Failure mode | Mechanism | Mitigations (primary / backstop) | +|---|---|---| +| **Cross-channel leakage** | Channel-scoped item retrieved in another channel via user-level grant | Scope-narrowing rule in evaluator; `channel_id` predicate in prefilter *and* postfilter; vec partition on channel | +| **Deleted-source leakage** | Tombstoned source's items still in FTS/vec, or alive through derived summaries | Same-transaction FTS/vec removal on delete; lineage re-check poisons descendants; quarantine + re-derivation job; audit query for post-deletion exposures | +| **Stale embeddings** | Vector reflects old (possibly more sensitive or since-redacted) content | `item_rev` in `embedding_meta`; postfilter drops rev-mismatched vector hits; background re-embedder; redaction bumps `rev` | +| **Vector prefilter leaks** | ANN returns neighbors before ACL filtering; IDs/distances observable | Postfilter is the boundary; per-user files + label/channel partitions; adaptive over-fetch; denied candidates never serialized out of the broker | +| **Summary leakage** | Restricted content laundered into broadly-scoped derived items | `derive` perm gate on inputs; write-time label intersection; read-time lineage re-check; declassification only via explicit audited grant | +| **Multi-process writes** | Two runtimes writing one DB → lost writes/corruption | Exclusive `broker.lock` + named mutex; second instance becomes IPC client; WAL + busy_timeout; single write connection | +| **Prefilter/postfilter drift** | SQL predicate diverges from evaluator logic after a policy change | Postfilter authoritative; audit `denied_postfilter`-after-prefilter-pass as an alerting signal; property-based tests asserting the recall invariant (prefilter never excludes what the evaluator allows; results ⊇ evaluator-allowed) | +| **Spoofed channel sender** | Attacker asserts another member's `channel_user_id` to inherit their memory | Transport auth at the adapter (webhook signatures, bot session); `channel_identity` binding is sole user-resolution authority; unbound senders get anonymous (empty) scope; bindings revocable on compromise | +| **Envelope user impersonation** | Compromised agent/tool passes an arbitrary `user_id` in the request envelope | Channel-origin requests ignore envelope `user_id` entirely — user is always derived from `(channel_id, channel_user_id)`; non-channel (API) origins require an authenticated session token instead | +| **Stale channel membership** | User left/was removed from channel but still retrieves channel-scoped items | `channel_member.left_at` checked at read time in rule 0 and scope narrowing — no cached grants; membership snapshot refreshed from channel adapter events; sync propagates leaves as ACL-tightening ops | +| **Cross-member personal-memory leak in group channels** | Member B retrieves inferred personal facts about member A via shared channel scope | Mount model (RFC §1.2): personal stores are never mounted for reading in channel-scoped sessions — structural placement, not a filter; personal facts reach A's store only via the write-only postbox; channel principal never satisfies allow on personal items; red-team suite covers "what do you know about @A" from B's session and from the group session | +| **Steering-attribution confusion** | Run triggered by user A, steered mid-run by user B — tool calls attributed to the wrong user | Session-granularity attribution: the retrieval principal is the session's scope, stamped at routing time, immutable for the session; per-message senders are audit/postbox inputs only, never retrieval principals | +| **Postbox pollution** | Hostile group session files junk or poisoned content into a member's personal store | Postbox can only narrow audience (write to `user:A` with channel provenance + lineage), never read; items are reviewable/purgeable by owner, bulk-purge by source channel; rate limits per channel | +| **Prompt-injected exfiltration** | Model instructs agent to "recall everything about user X" | Broker only accepts scope from runtime-asserted context, never model text; sensitivity ceilings per agent; rate/volume anomaly alerts on exposure_audit | +| **Sync widening** | Client or server merges ACLs upward | Tighten-on-conflict merge; server label recomputation; inbound ops re-validated locally | + +--- + +## 12. Broker Runtime + +OpenClaw is a TypeScript/Node runtime. The broker is implemented as a **core +module** (`src/memory-acl/`) with an optional enterprise plugin for IdP adapters +and operational tooling. See [implementation-plan.md](implementation-plan.md) +for the full component breakdown. + +| Option | Pros | Cons | +|---|---|---| +| **A. TS in-process broker** (`node:sqlite` + sqlite-vec, the existing memory-core stack) | One runtime, one deploy, zero new native deps; zero IPC latency; Node's single-threaded event loop makes the single-writer invariant nearly free; `DatabaseSync`'s synchronous API keeps txn scopes trivially correct | Broker shares a failure domain and memory space with the agent runtime (a prompt-injected agent is *in the same process* as the broker); CPU-bound embedding/re-derivation needs `worker_threads` | +| B. TS broker in the gateway process, agents as child processes | Process isolation *and* single runtime — agents get an IPC client, never the DB | Requires OpenClaw's process model to support it | + +**Recommendation: A for single-machine local-first installs, moving toward B as OpenClaw's process model allows.** The schema, evaluation semantics, and pipeline are identical in both; only the host boundary changes. + +### 12.1 TypeScript contracts + +```typescript +// ---- Identity & policy ------------------------------------------------------ +export type SessionScope = 'user' | 'channel' | 'agent'; + +export interface SessionEnvelope { + // Stamped ONCE by the gateway at session routing time; immutable for the + // session's lifetime. Tool calls never construct or modify this. + tenantId: string; + sessionKey: string; + sessionScope: SessionScope; // 'user' (DM), 'channel' (group), 'agent' (cron/webhook/subagent) + channelId: string; + agentId: string; + // NOTE: no userId field — deliberately unrepresentable. For user-scoped + // sessions the user is derived from the binding table at resolution time. +} + +export interface SenderRef { + // Per-message, used ONLY for audit rows and postbox targeting — never as + // the retrieval principal of a channel-scoped session. + channelUserId: string; // transport-asserted; verified by the channel adapter + conversationId?: string; +} + +export interface PolicyContext { + readonly tenantId: string; + readonly sessionScope: SessionScope; // stamped at routing, immutable + readonly userId: string | null; // user-scoped sessions only; derived via channel_identity + readonly channelUserId: string | null; // last resolved sender; audit + postbox only + readonly principals: ReadonlySet; + readonly channelId: string; + readonly conversationId?: string; + readonly agentId: string; + readonly purpose: 'retrieve' | 'derive' | 'sync'; + readonly sensitivityCeiling: number; +} + +export type Perm = 'retrieve' | 'read' | 'derive' | 'sync' | 'admin'; +export type Decision = + | 'allow' | 'deny-explicit' | 'deny-default' | 'deny-scope' + | 'deny-lineage' | 'deny-partition' | 'deny-stale' + | 'deny-identity' | 'deny-membership'; + +export interface PolicyContextResolver { + /** Called once at session creation (and on sender change, to refresh the + * audit/postbox SenderRef). For user-scoped sessions, throws + * IdentityBindingError before any DB file is opened if + * (channelId, channelUserId) has no live verified binding. */ + resolve(envelope: SessionEnvelope, sender: SenderRef | null, + purpose: PolicyContext['purpose']): Promise; +} + +export interface AclEvaluator { + evaluate(ctx: PolicyContext, item: MemoryItemHeader, acl: AclEntry[], perm: Perm): Decision; + readableLabels(ctx: PolicyContext): ReadonlySet; // for vec label prefilter +} + +// ---- Broker (the only surface agents see) ----------------------------------- +export interface MemoryBroker { + buildContext(q: MemoryQuery): Promise; + remember(w: MemoryWrite): Promise; + derive(r: DerivationRequest): Promise; // gated summarization + forget(r: ForgetRequest): Promise; // tombstone + cascade + FTS/vec purge + analyzeRevocation(itemId: string): Promise; +} + +export interface MemoryQuery { + envelope: SessionEnvelope; + queryText: string; + tokenBudget: number; + k?: number; +} + +export interface MemoryContextResult { + contextBlock: string; + exposed: ReadonlyArray<{ itemId: string; rev: number }>; + exposureBatchId: string; +} + +// ---- Retrieval pipeline ------------------------------------------------------- +export interface CandidateSource { + readonly name: 'fts5' | 'sqlite-vec'; + search(ctx: PolicyContext, query: string, k: number): Promise; +} + +export interface VectorIndex { + upsert(itemId: string, rev: number, emb: Float32Array, + channelId: string | null, label: string): void; + delete(itemId: string): void; + nearest(q: Float32Array, k: number, + channelFilter: string | null, + labelFilter: ReadonlySet): Array<{ itemId: string; dist: number }>; +} + +export interface LineageWalker { + checkAncestry(ctx: PolicyContext, itemId: string, maxDepth: number): Decision; +} + +export interface ExposureAuditor { + recordBatch(batchId: string, ctx: PolicyContext, + rows: Array<{ itemId: string; rev: number; decision: Decision; reason?: string }>, + promptHash?: string): void; +} +``` + +### 12.2 TS implementation notes + +The broker uses **the stack memory-core already ships**: `node:sqlite` +(`DatabaseSync`) with the `sqlite-vec` extension — no new native +dependencies, no second SQLite driver. `DatabaseSync` is synchronous, so +transactions compose without async pitfalls and Node's event loop is the +single-writer serializer; this is the same pattern +`extensions/memory-core/src/memory/manager-db.ts` uses today. + +```typescript +import { DatabaseSync } from 'node:sqlite'; + +// Same driver + extension loading path as memory-core's manager-db.ts. +const db = new DatabaseSync(scopeIndexDbPath, { allowExtension: true }); +db.loadExtension(sqliteVecPath); // sqlite-vec, already a dependency +db.exec(` + PRAGMA journal_mode = WAL; + PRAGMA foreign_keys = ON; + PRAGMA busy_timeout = 5000; +`); + +// Single-writer invariant across processes still needs the lock: +// broker.lock acquired exclusively at startup; losers connect as IPC clients. +// In-process, all mutations flow through one serial queue so multi-agent +// concurrency can't interleave partial writes; each op runs inside a single +// transaction: item + ACL + lineage + sync_op atomic. +const writeQueue = createSerialQueue(); // concurrency: 1 +export const enqueueWrite = (op: (db: DatabaseSync) => T) => + writeQueue.add(() => runInTransaction(db, op)); + +// Off-loop work: embeddings and re-derivation run in worker_threads — +// the same split memory-core uses for local embedding work; workers receive +// content + item ids, never a DB handle. +``` + +Stack: `node:sqlite` (`DatabaseSync`, already the memory-core driver); +`sqlite-vec` (already a dependency); `ulid`. Zod schemas at the broker +boundary so malformed agent-supplied envelopes fail closed. At-rest +encryption is OS FDE in this phase (§8) — `node:sqlite` links vanilla SQLite +and cannot open SQLCipher files; an encrypted-driver swap is an +out-of-process-broker concern, deliberately isolated behind the broker +interface so it never touches callers. + +### 12.3 Session scope resolution (TS) + +```typescript +export class SessionScopeResolver implements PolicyContextResolver { + constructor(private readonly stateDb: DatabaseSync, // tenant state.db owns bindings + private readonly groups: GroupSnapshot) {} // fail-closed, bounded staleness (§4) + + async resolve(env: SessionEnvelope, sender: SenderRef | null, + purpose: PolicyContext['purpose']): Promise { + const base = { + tenantId: env.tenantId, + sessionScope: env.sessionScope, + channelId: env.channelId, + agentId: env.agentId, + purpose, + sensitivityCeiling: agentCeiling(env.agentId), + }; + + if (env.sessionScope === 'user') { + // DM session: the retrieval principal IS the verified user. + if (!sender) throw new IdentityBindingError('deny-identity', env); + const binding = this.stateDb.prepare(` + SELECT user_id FROM channel_identity + WHERE channel_id = ? AND channel_user_id = ? AND revoked_at IS NULL + `).get(env.channelId, sender.channelUserId) as { user_id: string } | undefined; + if (!binding) throw new IdentityBindingError('deny-identity', env); + + return { ...base, userId: binding.user_id, channelUserId: sender.channelUserId, + principals: new Set([ + `user:${binding.user_id}`, `tenant:${env.tenantId}`, `agent:${env.agentId}`, + ...this.groups.principalsFor(binding.user_id), // dropped when snapshot too stale + ]) }; + } + + if (env.sessionScope === 'channel') { + // Group session: channel principal; sender resolved for audit/postbox only. + const senderUser = sender ? this.resolveSenderForAudit(env.channelId, sender) : null; + return { ...base, userId: null, channelUserId: sender?.channelUserId ?? null, + principals: new Set([ + `channel:${env.channelId}`, `tenant:${env.tenantId}`, `agent:${env.agentId}`, + ]) }; + } + + // Autonomous (cron/webhook/subagent): agent principal only, nothing user-scoped. + return { ...base, userId: null, channelUserId: null, + principals: new Set([`agent:${env.agentId}`, `tenant:${env.tenantId}`]) }; + } +} +``` + +Live channel membership is not resolved here — it is checked **at read time** +in the scope-narrowing step (§4, rule 5) per requesting member, so leaving a +channel revokes visibility immediately regardless of session lifetime. + +### 12.4 Alternative runtime hosts + +Option B (gateway-hosted broker) uses identical +schema and evaluation semantics. The TS interfaces in §12.1 are the canonical +contract; alternative hosts implement the same surface over UDS/named-pipe IPC. +See the [implementation plan](implementation-plan.md) for the component +breakdown and core vs. plugin architecture split. + +--- + +## 13. Phased Implementation Plan + +**Phase 0 — Secure core (2–3 wks).** Scoped memory subtree layout + per-scope index partitions (single-user: one scope, no file moves); schema (item catalog, source, ACL, `channel_identity`, `channel_member`, audit); session file-view enforcement for `memory_search`/`memory_get` (RFC §2.1); pairing flow for channel-identity binding; broker with session-scope resolver → FTS5 prefilter → postfilter → trim → audit; single-writer queue; exclusive lock. *Exit:* property-based tests proving the prefilter never excludes evaluator-allowed items (results ⊇ evaluator-allowed; postfilter authoritative for anything it over-includes); adversarial matrix green for cross-channel retrieval, spoofed `channel_user_id`, and envelope `user_id` injection. + +**Phase 1 — Derivation & lineage (2 wks).** `lineage_edge`, `derive` gating, write-time label intersection, read-time ancestry re-check, tombstone quarantine + re-derivation worker. *Exit:* deleted-source and summary-leak red-team suites green. + +**Phase 2 — Vector search (1–2 wks).** `vectors.db` + sqlite-vec behind `IVectorIndex`; label/channel partition filters; adaptive over-fetch; staleness contract + background embedder. *Exit:* recall benchmarks vs. FTS-only; leak tests confirming denied neighbors never serialize. + +**Phase 3 — Hardening (1–2 wks).** At-rest posture on the current stack: OS FDE verification in `openclaw doctor`, `0700` permissions on `~/.openclaw/memory-acl/` (index partitions are content-bearing — risk 7) and the workspace `memory/` tree; audit retention/compaction; anomaly alerts on exposure volume. App-level encryption (SQLCipher + OS-keystore key hierarchy, §8) is deliberately **not** in this phase — it requires a driver change and only pays off with an out-of-process broker; the schema and key-hierarchy design are ready for it when that lands. + +**Phase 4 — Sync (3–4 wks).** Outbox shipping of `sync`-permitted ops; HLC merge with tighten-on-conflict; server-side label recomputation; tombstone propagation with grace window; inbound re-validation through the local write path. + +Each phase ships behind the same `IMemoryBroker` surface — the runtime integration is written once in Phase 0 and never changes.