feat(extension): report editable element ids alongside the a11y snapshot

[miguelg719]

Stack

Stack: #2951 snapshot editable ids → #2952 library → #2953 act pipeline → #2954 observe + cache check → #2955 extract. This is 1/5 (#2951). Each PR targets its predecessor.

Why

The a11y outline tells you an element's role, not whether you can type into it. A contenteditable div shows up as generic, and rich-text editors inside iframes look like plain containers. The Jev act path (next PRs) needs to know which nodes accept text to build its "input" candidate view; Chrome already reports this on each AX node as the editable property.

What

• a11yForFrame collects the encoded ids of nodes whose AX editable property is plaintext or richtext and returns them as editableIds.
• captureHybridSnapshot / tryScopedSnapshot merge them into combinedEditableIds (and per-frame editableIds).
• It is a side channel on the private snapshot type: the outline text, xpath map and url map are byte-for-byte unchanged, so nothing the LLM sees (or any cache key) moves.
• No consumer yet; the embedded Go extension zip is regenerated because extension source changed.

Testing

• New a11y-editable-ids.test.ts: editable nodes are reported by encoded id, ignored nodes are skipped, the outline is untouched.
• Extension unit tests, protocol tests, root parity tests, typecheck, oxfmt --check, extensionpack --check pass locally.

[changeset-bot]

⚠️ No Changeset found

Latest commit: fb0c254

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

2 issues found across 5 files

Confidence score: 3/5

• packages/extension/types/private/snapshot.ts drops the new per-frame editable IDs before consumers can access them, so editable-element behavior may not work across frames; add editableIds to PerFrameSnapshot and both per-frame snapshot object literals.
• packages/extension/understudy/a11y/snapshot/capture.ts lacks regression coverage for cross-frame aggregation and the scoped snapshot path, leaving future propagation regressions undetected; add capture-level tests for both paths.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/extension/types/private/snapshot.ts">

<violation number="1" location="packages/extension/types/private/snapshot.ts:74">
P2: The new per-frame editable IDs are dropped before consumers can read them. Add `editableIds` to `PerFrameSnapshot` and copy it into both per-frame snapshot object literals.</violation>
</file>

<file name="packages/extension/understudy/a11y/snapshot/capture.ts">

<violation number="1" location="packages/extension/understudy/a11y/snapshot/capture.ts:836">
P2: Add capture-level regression tests for aggregation across frames and for the scoped snapshot path. The current feature test leaves this wiring unverified, so a future propagation regression can ship while the editable-node test remains green.

(Based on your team's feedback about unit tests for new behavior.)</violation>
</file>

Architecture diagram

sequenceDiagram
    participant Capture as Snapshot Capture
    participant CDP as CDP Session
    participant AX as Chrome AX Tree
    participant A11y as a11yForFrame
    participant Maps as Per-Frame Maps
    participant Merge as Snapshot Merger
    participant Snapshot as HybridSnapshot

    Note over Capture,Snapshot: A11y snapshot carries editable element IDs as a side channel

    Capture->>CDP: Accessibility.getFullAXTree(frameId)
    CDP->>AX: Fetch full accessibility tree
    AX-->>CDP: AX nodes with role, name, properties
    CDP-->>Capture: AX nodes

    Capture->>A11y: Build frame snapshot
    A11y->>A11y: Filter ignored backend DOM nodes
    A11y->>A11y: Decorate roles and build outline + urlMap
    loop Each unignored node
        A11y->>A11y: Inspect AX "editable" property
        alt editable is plaintext or richtext
            A11y->>A11y: Encode backendDOMNodeId and append to editableIds
        end
    end
    A11y-->>Capture: outline, urlMap, editableIds

    alt Full hybrid snapshot
        Capture->>Maps: Store frame maps including editableIds
        Capture->>Merge: Merge per-frame maps
        Merge->>Snapshot: combinedEditableIds from flattened per-frame editableIds
    else Scoped single-frame snapshot
        Capture->>Snapshot: combinedEditableIds from this frame's editableIds
    end

    Note over Snapshot: Outline text, xpath map, and url map are unchanged<br/>LLM-visible content and cache keys are not affected

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: The new per-frame editable IDs are dropped before consumers can read them. Add editableIds to PerFrameSnapshot and copy it into both per-frame snapshot object literals.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/extension/types/private/snapshot.ts, line 74:

<comment>The new per-frame editable IDs are dropped before consumers can read them. Add `editableIds` to `PerFrameSnapshot` and copy it into both per-frame snapshot object literals.</comment>

<file context>
@@ -66,6 +71,7 @@ export type FrameDomMaps = {
   xpathMap: Record<string, string>;
   scrollableMap: Record<string, boolean>;
   urlMap: Record<string, string>;
+  editableIds?: string[];
 };
 
</file context>

[miguelg719]

Fixed on the branch: PerFrameSnapshot.editableIds added and copied into both per-frame literals (scoped and merged paths).

[cubic-dev-ai]

P2: Add capture-level regression tests for aggregation across frames and for the scoped snapshot path. The current feature test leaves this wiring unverified, so a future propagation regression can ship while the editable-node test remains green.

(Based on your team's feedback about unit tests for new behavior.)

View Feedback

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/extension/understudy/a11y/snapshot/capture.ts, line 836:

<comment>Add capture-level regression tests for aggregation across frames and for the scoped snapshot path. The current feature test leaves this wiring unverified, so a future propagation regression can ship while the editable-node test remains green.

(Based on your team's feedback about unit tests for new behavior.) </comment>

<file context>
@@ -828,6 +833,7 @@ export function mergeFramesIntoSnapshot(
     combinedTree,
     combinedXpathMap,
     combinedUrlMap,
+    combinedEditableIds: [...perFrameMaps.values()].flatMap((maps) => maps.editableIds ?? []),
     perFrame: perFrameOutlines.map(({ frameId, outline }) => {
       const maps = perFrameMaps.get(frameId);
</file context>

[miguelg719]

Not changing this — the cross-frame aggregation is a one-line spread of per-frame results, and the a11yForFrame boundary (where the ids are computed) has its own test; capture-level tests need full CDP session mocks that this stack does not introduce. The editable ids are exercised end to end by the pipeline tests in #2953 (contenteditable targets).