feat(extension): experimental Jev observe() and cached-action check - #2954
Open
miguelg719 wants to merge 1 commit into
Open
miguelg719 wants to merge 1 commit into
miguelg719 wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: 6205907 The changes in this PR will be included in the next version bump. This PR includes changesets to release 20 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
This was referenced Sep 17, 2026
miguelg719
added this pull request to stack #2966
September 17, 2026 19:20
This was referenced Sep 18, 2026
miguelg719
marked this pull request as ready for review
September 19, 2026 21:20
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
11 issues found across 18 files
Confidence score: 2/5
packages/extension/services/actService.tsreplays every cached action after validating only the first, allowing stale selectors to drive later actions;packages/extension/services/jevAct/cacheCheck.tsalso makes cached press/scroll actions miss the document node and be ignored. Validate each action immediately before replay and align the document XPath.packages/extension/services/observeService.tscan invoke Jev even whenexperimentalJevAct.enabledis false, changing baseline observe behavior; the same Jev completion path also needsflowLoggerinstrumentation for its public methods. Honor the shared disable flag and instrument the affected interfaces.packages/extension/services/jevAct/observe.tsmay report success after its timeout when the final snapshot or Jev request overruns, masking incomplete observation. RecheckensureTimeRemaining()infinish()before returning success.- Default observation in
packages/extension/services/jevAct/observe.tsomits native selects and labels contenteditable regions asclick, so controls can be missed or receive unusable actions. Include the select view and treat editable nodes as fill targets; pin the cache stale/match boundary inpackages/extension/tests/jevCacheCheck.test.tsto prevent threshold regressions.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/extension/services/jevAct/README.md">
<violation number="1" location="packages/extension/services/jevAct/README.md:49">
P3: When a cached selector is missing from the snapshot, `checkCachedAction` returns `unknown` without sending the Jev request, so this is not true for every cache hit. Document that the request only occurs when the selector resolves, while the snapshot is captured for the enabled check.</violation>
</file>
<file name="packages/extension/services/jevAct/observe.ts">
<violation number="1" location="packages/extension/services/jevAct/observe.ts:71">
P2: When the final snapshot or Jev request runs past `observe()`'s timeout, `runJevObserve()` can still return success because `finish()` never checks the timeout guard. Check `deps.ensureTimeRemaining()` in `finish()` before returning the outcome.</violation>
<violation number="2" location="packages/extension/services/jevAct/observe.ts:89">
P2: With no instruction, `observe()` omits native `<select>` elements because it never includes the `select` view. Include native selects and assign them a usable observation action so default observation does not miss interactive form controls.</violation>
<violation number="3" location="packages/extension/services/jevAct/observe.ts:274">
P2: When no instruction is provided, contenteditable regions are returned as `click` actions instead of `fill` actions. Include `node.editable` in the role-free fill condition so the action matches the input view.</violation>
</file>
<file name="packages/extension/services/observeService.ts">
<violation number="1" location="packages/extension/services/observeService.ts:82">
P2: When `experimentalJevAct.enabled` is false, this branch still runs Jev if `observe` is true. Honor the shared Jev disable switch so baseline configurations do not invoke TypeSafe or alter observe results.</violation>
<violation number="2" location="packages/extension/services/observeService.ts:105">
P1: Custom agent: **Ensure all public methods added to the stagehand class, agent, or understudy (page, locator, etc.) interfaces are properly instrumented with the flowLogger**
When Jev completes through this branch, its `systemOne` LLM requests bypass the required LLM instrumentation and are reported as zero usage. Route Jev requests through the flow-logger LLM middleware or add manual `logLlmRequest` records, and propagate their usage instead of returning zeros.</violation>
</file>
<file name=".changeset/jev-observe-cache-check.md">
<violation number="1" location=".changeset/jev-observe-cache-check.md:8">
P3: The release note does not tell users how to enable these experimental features and uses the internal term “Jev path.” Name the `experimentalJevAct.observe` and `experimentalJevAct.cacheCheck` flags and state that they are off by default, so the generated changelog describes the user-visible configuration accurately.
(Based on your team's feedback about user-visible changesets.) .</violation>
</file>
<file name="packages/extension/services/actService.ts">
<violation number="1" location="packages/extension/services/actService.ts:382">
P1: When a cached act contains multiple actions, `cacheCheck` validates only the first selector and then blindly replays the rest. Validate each action immediately before replaying it, recapturing the page after earlier actions so later targets are checked against the DOM state in which they will run.</violation>
</file>
<file name="packages/extension/services/jevAct/cacheCheck.ts">
<violation number="1" location="packages/extension/services/jevAct/cacheCheck.ts:27">
P2: Cached `press` and scroll actions use `xpath=/html`, but the snapshot map records the document element as `/html[1]`, so this lookup returns `unknown` for every such cache hit. Because replay ignores `unknown`, the `cacheCheck` opt-in provides no stale-target protection for these actions; canonicalize the document-root selector before comparing paths.</violation>
</file>
<file name="packages/extension/tests/jevCacheCheck.test.ts">
<violation number="1" location="packages/extension/tests/jevCacheCheck.test.ts:79">
P2: The stale/match boundary (STALE_BELOW = 0.35 in cacheCheck.ts) is unpinned: the tests only assert scores far from it (0.04 and 0.93). A regression moving the threshold to 0.1 or 0.5 would pass the entire suite silently. Since no eval covers the cache path end to end, add a probe near the boundary, e.g. still_matches at 0.3 (stale) and 0.4 (match), ideally at exactly 0.35 to lock in the strict `<` comparison.</violation>
</file>
<file name="packages/extension/tests/act.test.ts">
<violation number="1" location="packages/extension/tests/act.test.ts:633">
P3: The test can pass even if the cache layer silently stops reading hits: both the intended stale-cache path (`missMetadata(getResponse, "replay_failed")` in cacheService.withCache) and a failed/absent cache read (`missMetadata(null, "read_failed")`) yield `status: "MISS"` and then run the identical LLM re-inference, ending in the same selector. Assert `result.metadata.cache.missReason` toBe "replay_failed" (and optionally that `get`/`set` were called) so the test actually pins the hit-then-stale re-check path it describes.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| // control gets acted on with no model in the loop. One Jev yes/no on the | ||
| // first action catches that; throwing sends the act through full inference. | ||
| if (context.jevAct?.cacheCheck && context.jevAct.enabled !== false) { | ||
| const verdict = await validateCachedAction(actions[0]!, instruction, context, variables).catch( |
Contributor
There was a problem hiding this comment.
P1: When a cached act contains multiple actions, cacheCheck validates only the first selector and then blindly replays the rest. Validate each action immediately before replaying it, recapturing the page after earlier actions so later targets are checked against the DOM state in which they will run.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/extension/services/actService.ts, line 382:
<comment>When a cached act contains multiple actions, `cacheCheck` validates only the first selector and then blindly replays the rest. Validate each action immediately before replaying it, recapturing the page after earlier actions so later targets are checked against the DOM state in which they will run.</comment>
<file context>
@@ -373,6 +375,23 @@ async function replayCachedActions(
+ // control gets acted on with no model in the loop. One Jev yes/no on the
+ // first action catches that; throwing sends the act through full inference.
+ if (context.jevAct?.cacheCheck && context.jevAct.enabled !== false) {
+ const verdict = await validateCachedAction(actions[0]!, instruction, context, variables).catch(
+ (error: unknown) => {
+ if (error instanceof TimeoutError) throw error;
</file context>
| metadata: { usage: zeroStagehandResultUsage(), cache: disabledCacheMetadata() }, | ||
| }, | ||
| cacheValue: outcome.actions.length > 0 ? outcome.actions : undefined, | ||
| llmUsage: { inputTokens: 0, outputTokens: 0, llmDurationMs: 0 }, |
Contributor
There was a problem hiding this comment.
P1: Custom agent: Ensure all public methods added to the stagehand class, agent, or understudy (page, locator, etc.) interfaces are properly instrumented with the flowLogger
When Jev completes through this branch, its systemOne LLM requests bypass the required LLM instrumentation and are reported as zero usage. Route Jev requests through the flow-logger LLM middleware or add manual logLlmRequest records, and propagate their usage instead of returning zeros.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/extension/services/observeService.ts, line 105:
<comment>When Jev completes through this branch, its `systemOne` LLM requests bypass the required LLM instrumentation and are reported as zero usage. Route Jev requests through the flow-logger LLM middleware or add manual `logLlmRequest` records, and propagate their usage instead of returning zeros.</comment>
<file context>
@@ -74,6 +79,45 @@ export async function observe({
+ metadata: { usage: zeroStagehandResultUsage(), cache: disabledCacheMetadata() },
+ },
+ cacheValue: outcome.actions.length > 0 ? outcome.actions : undefined,
+ llmUsage: { inputTokens: 0, outputTokens: 0, llmDurationMs: 0 },
+ };
+ }
</file context>
| deps: JevObserveDeps, | ||
| ): Promise<JevObserveOutcome> { | ||
| const trace: TraceEntry[] = []; | ||
| const finish = (outcome: JevObserveOutcome): JevObserveOutcome => { |
Contributor
There was a problem hiding this comment.
P2: When the final snapshot or Jev request runs past observe()'s timeout, runJevObserve() can still return success because finish() never checks the timeout guard. Check deps.ensureTimeRemaining() in finish() before returning the outcome.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/extension/services/jevAct/observe.ts, line 71:
<comment>When the final snapshot or Jev request runs past `observe()`'s timeout, `runJevObserve()` can still return success because `finish()` never checks the timeout guard. Check `deps.ensureTimeRemaining()` in `finish()` before returning the outcome.</comment>
<file context>
@@ -0,0 +1,307 @@
+ deps: JevObserveDeps,
+): Promise<JevObserveOutcome> {
+ const trace: TraceEntry[] = [];
+ const finish = (outcome: JevObserveOutcome): JevObserveOutcome => {
+ deps.logger.info("Jev observe pipeline finished", {
+ category: "jev",
</file context>
| if (!deps.instruction) { | ||
| const everything = uniqueById([ | ||
| ...buildView(snap.nodes, "pointer"), | ||
| ...buildView(snap.nodes, "input"), |
Contributor
There was a problem hiding this comment.
P2: With no instruction, observe() omits native <select> elements because it never includes the select view. Include native selects and assign them a usable observation action so default observation does not miss interactive form controls.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/extension/services/jevAct/observe.ts, line 89:
<comment>With no instruction, `observe()` omits native `<select>` elements because it never includes the `select` view. Include native selects and assign them a usable observation action so default observation does not miss interactive form controls.</comment>
<file context>
@@ -0,0 +1,307 @@
+ if (!deps.instruction) {
+ const everything = uniqueById([
+ ...buildView(snap.nodes, "pointer"),
+ ...buildView(snap.nodes, "input"),
+ ]).sort((a, b) => a.index - b.index);
+ trace.push({ node: "all_interactive", ms: 0, options: everything.length });
</file context>
| args = option ? [option] : []; | ||
| } else if ( | ||
| family === "fill" || | ||
| (!family && /\b(textbox|searchbox|spinbutton|textarea)\b/.test(role)) |
Contributor
There was a problem hiding this comment.
P2: When no instruction is provided, contenteditable regions are returned as click actions instead of fill actions. Include node.editable in the role-free fill condition so the action matches the input view.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/extension/services/jevAct/observe.ts, line 274:
<comment>When no instruction is provided, contenteditable regions are returned as `click` actions instead of `fill` actions. Include `node.editable` in the role-free fill condition so the action matches the input view.</comment>
<file context>
@@ -0,0 +1,307 @@
+ args = option ? [option] : [];
+ } else if (
+ family === "fill" ||
+ (!family && /\b(textbox|searchbox|spinbutton|textarea)\b/.test(role))
+ ) {
+ method = "fill";
</file context>
Suggested change
| (!family && /\b(textbox|searchbox|spinbutton|textarea)\b/.test(role)) | |
| (!family && | |
| (node.editable === true || /\b(textbox|searchbox|spinbutton|textarea)\b/.test(role))) |
| action: Action, | ||
| ): Promise<CacheVerdict> { | ||
| const wanted = normalizeXpath(action.selector); | ||
| const id = Object.entries(snap.xpathMap).find( |
Contributor
There was a problem hiding this comment.
P2: Cached press and scroll actions use xpath=/html, but the snapshot map records the document element as /html[1], so this lookup returns unknown for every such cache hit. Because replay ignores unknown, the cacheCheck opt-in provides no stale-target protection for these actions; canonicalize the document-root selector before comparing paths.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/extension/services/jevAct/cacheCheck.ts, line 27:
<comment>Cached `press` and scroll actions use `xpath=/html`, but the snapshot map records the document element as `/html[1]`, so this lookup returns `unknown` for every such cache hit. Because replay ignores `unknown`, the `cacheCheck` opt-in provides no stale-target protection for these actions; canonicalize the document-root selector before comparing paths.</comment>
<file context>
@@ -0,0 +1,65 @@
+ action: Action,
+): Promise<CacheVerdict> {
+ const wanted = normalizeXpath(action.selector);
+ const id = Object.entries(snap.xpathMap).find(
+ ([, xpath]) => normalizeXpath(xpath) === wanted,
+ )?.[0];
</file context>
Suggested change
| const id = Object.entries(snap.xpathMap).find( | |
| const wanted = normalizeXpath(action.selector).replace(/^\/html$/i, "/html[1]"); |
| cached, | ||
| ); | ||
|
|
||
| expect(verdict).toEqual({ |
Contributor
There was a problem hiding this comment.
P2: The stale/match boundary (STALE_BELOW = 0.35 in cacheCheck.ts) is unpinned: the tests only assert scores far from it (0.04 and 0.93). A regression moving the threshold to 0.1 or 0.5 would pass the entire suite silently. Since no eval covers the cache path end to end, add a probe near the boundary, e.g. still_matches at 0.3 (stale) and 0.4 (match), ideally at exactly 0.35 to lock in the strict < comparison.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/extension/tests/jevCacheCheck.test.ts, line 79:
<comment>The stale/match boundary (STALE_BELOW = 0.35 in cacheCheck.ts) is unpinned: the tests only assert scores far from it (0.04 and 0.93). A regression moving the threshold to 0.1 or 0.5 would pass the entire suite silently. Since no eval covers the cache path end to end, add a probe near the boundary, e.g. still_matches at 0.3 (stale) and 0.4 (match), ideally at exactly 0.35 to lock in the strict `<` comparison.</comment>
<file context>
@@ -0,0 +1,105 @@
+ cached,
+ );
+
+ expect(verdict).toEqual({
+ verdict: "stale",
+ score: 0.04,
</file context>
| | `llmFallback` | `true` | `false` fails the act when Jev abstains: the fastest way to see what Jev alone gets wrong. | | ||
| | `argumentLlm` | `true` | The argument-only LLM call for unquoted text. Independent of `llmFallback`; turn both off for an LLM-free run. The typed text is always the instruction's own characters, never the model's re-cased copy. | | ||
| | `pageState` | `true` | Page-state request when Jev leans toward "not on this page". | | ||
| | `cacheCheck` | `false` | One Jev yes/no before replaying a cached action; stale ones are re-inferred. Adds a snapshot and a request to every cache hit. | |
Contributor
There was a problem hiding this comment.
P3: When a cached selector is missing from the snapshot, checkCachedAction returns unknown without sending the Jev request, so this is not true for every cache hit. Document that the request only occurs when the selector resolves, while the snapshot is captured for the enabled check.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/extension/services/jevAct/README.md, line 49:
<comment>When a cached selector is missing from the snapshot, `checkCachedAction` returns `unknown` without sending the Jev request, so this is not true for every cache hit. Document that the request only occurs when the selector resolves, while the snapshot is captured for the enabled check.</comment>
<file context>
@@ -43,14 +46,17 @@ deliberately not a field of the public create config. Evals build that variable
| `llmFallback` | `true` | `false` fails the act when Jev abstains: the fastest way to see what Jev alone gets wrong. |
| `argumentLlm` | `true` | The argument-only LLM call for unquoted text. Independent of `llmFallback`; turn both off for an LLM-free run. The typed text is always the instruction's own characters, never the model's re-cased copy. |
| `pageState` | `true` | Page-state request when Jev leans toward "not on this page". |
+| `cacheCheck` | `false` | One Jev yes/no before replaying a cached action; stale ones are re-inferred. Adds a snapshot and a request to every cache hit. |
+| `observe` | `false` | Resolve `observe()` through Jev first. "Find all" is answered exhaustively or handed to the LLM (over 600 candidates; over 400 elements with no instruction), never truncated. |
| `retryNoEffect` | `false` | Click the runner-up when an ambiguous click provably changed nothing. Off: effects the outline cannot show (aria-pressed, copy, play) look like "nothing". Never cached. |
</file context>
Suggested change
| | `cacheCheck` | `false` | One Jev yes/no before replaying a cached action; stale ones are re-inferred. Adds a snapshot and a request to every cache hit. | | |
| | `cacheCheck` | `false` | One Jev yes/no when the cached selector resolves; stale ones are re-inferred. Adds a snapshot to every cache hit and a request when the selector resolves. | |
| "@browserbasehq/stagehand-python": patch | ||
| --- | ||
|
|
||
| experimental Jev path: opt-in `observe` resolution and a `cacheCheck` that validates cached actions against the page before replay |
Contributor
There was a problem hiding this comment.
P3: The release note does not tell users how to enable these experimental features and uses the internal term “Jev path.” Name the experimentalJevAct.observe and experimentalJevAct.cacheCheck flags and state that they are off by default, so the generated changelog describes the user-visible configuration accurately.
(Based on your team's feedback about user-visible changesets.) .
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At .changeset/jev-observe-cache-check.md, line 8:
<comment>The release note does not tell users how to enable these experimental features and uses the internal term “Jev path.” Name the `experimentalJevAct.observe` and `experimentalJevAct.cacheCheck` flags and state that they are off by default, so the generated changelog describes the user-visible configuration accurately.
(Based on your team's feedback about user-visible changesets.) .</comment>
<file context>
@@ -0,0 +1,8 @@
+"@browserbasehq/stagehand-python": patch
+---
+
+experimental Jev path: opt-in `observe` resolution and a `cacheCheck` that validates cached actions against the page before replay
</file context>
Suggested change
| experimental Jev path: opt-in `observe` resolution and a `cacheCheck` that validates cached actions against the page before replay | |
| add experimental, off-by-default `experimentalJevAct.observe` resolution for `observe()` and `experimentalJevAct.cacheCheck` validation before replaying cached actions |
| jevAct: { apiKey: "test", cacheCheck: true }, | ||
| }); | ||
|
|
||
| expect(result.metadata.cache.status).toBe("MISS"); |
Contributor
There was a problem hiding this comment.
P3: The test can pass even if the cache layer silently stops reading hits: both the intended stale-cache path (missMetadata(getResponse, "replay_failed") in cacheService.withCache) and a failed/absent cache read (missMetadata(null, "read_failed")) yield status: "MISS" and then run the identical LLM re-inference, ending in the same selector. Assert result.metadata.cache.missReason toBe "replay_failed" (and optionally that get/set were called) so the test actually pins the hit-then-stale re-check path it describes.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/extension/tests/act.test.ts, line 633:
<comment>The test can pass even if the cache layer silently stops reading hits: both the intended stale-cache path (`missMetadata(getResponse, "replay_failed")` in cacheService.withCache) and a failed/absent cache read (`missMetadata(null, "read_failed")`) yield `status: "MISS"` and then run the identical LLM re-inference, ending in the same selector. Assert `result.metadata.cache.missReason` toBe "replay_failed" (and optionally that `get`/`set` were called) so the test actually pins the hit-then-stale re-check path it describes.</comment>
<file context>
@@ -567,6 +567,77 @@ describe("act service", () => {
+ jevAct: { apiKey: "test", cacheCheck: true },
+ });
+
+ expect(result.metadata.cache.status).toBe("MISS");
+ expect(performAction).toHaveBeenCalledTimes(1);
+ expect(performAction.mock.calls[0]?.[3]).toBe("xpath=/html/body/form/button");
</file context>
Suggested change
| expect(result.metadata.cache.status).toBe("MISS"); | |
| expect(result.metadata.cache.status).toBe("MISS"); | |
| expect(result.metadata.cache.missReason).toBe("replay_failed"); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack
Stack: #2951 snapshot editable ids → #2952 library → #2953 act pipeline → #2954 observe + cache check → #2955 extract. This is 4/5 (#2954). Each PR targets its predecessor.
Why
Two more places where the question is "which of these elements" or "is this still the same element", reusing the picking library from #2952:
observe()is the same selection problem as act without the action.Both are separate opt-ins on
experimentalJevAct(observe,cacheCheck), default off.What
jevAct/observe.ts+observeService.ts— no instruction → every interactive element (LLM above 400); otherwise intent + cardinality: "one" uses the act picker, "several" asks a per-candidate yes/no in batches of 60 (LLM above 600 candidates). "Find all" is answered exhaustively or handed to the LLM, never truncated. Method comes from explicit intent first, else the element's role. Any abstention falls through to the existing LLM observe.jevAct/cacheCheck.ts+actService.replayCachedActions— one yes/no ("does the element this selector now resolves to still match the instruction / cached description?") before replay; a stale verdict throws into the normal re-inference path. Errors and timeouts never block replay. Costs one snapshot + one request per cache hit, hence opt-in.Results (gemini-3.8-flash, Browserbase, local)
observe suite: 9/12 baseline → 10/12; 11/16 observes answered by Jev in 0.18 s (baseline 1.99 s), 5/16 went to the LLM; 7/12 with the LLM disabled. The cache check threshold (0.35) comes from direct API probes; no eval exercises the cache path end to end.
Testing
jevObserve.test.ts,jevCacheCheck.test.ts, a replay test inact.test.ts(stale cached action is re-inferred instead of clicked). Full suites, typecheck, lint, fmt,extensionpack --checkpass locally.