Headless deployments: HITL tools cannot open a visible window — working CDP-relay workaround (remote web viewer)

[jsapede]

Summary

When cortex-scout runs on a headless server (no DISPLAY, no X server), the HITL tools (hitl_web_fetch, scout_agent_profile_auth) cannot open a visible browser window. They exist in the binary (full --all-features build) but fail at runtime because there is no display to present the interactive window to a human.

This issue documents a working workaround: a small CDP relay that exposes the existing headless Chromium session (the one cortex-scout already manages via chromiumoxide) as a web viewer, giving a human operator remote interactive control (click, type, scroll, navigate) from any browser — effectively a portable HITL channel for headless deployments.

Environment

• cortex-scout v3.3.7 built with cargo build --release --all-features on Debian 13 (trixie), x86_64
• Headless VM: no DISPLAY, no X11/Wayland session
• Chromium 150 system package
• Observed behavior of hitl_web_fetch on this host: fails with user cancelled (the visible window cannot be created; no human can interact with it)

Why noVNC-style VNC is the wrong path

A classic VNC setup (Xvfb + x11vnc + websockify + noVNC) would require launching a second Chromium in non-headless mode. That means:

• a different browser instance than the one cortex-scout is actually using → the CAPTCHA/auth wall being solved is not the one the agent is stuck on;
• profile lock conflicts if the same --user-data-dir is reused;
• cookies/session state divergence.

The interesting fact is that cortex-scout already runs a headless Chromium with a live CDP endpoint (chromiumoxide launches it with --remote-debugging-port=0, which binds a random localhost port). That endpoint is enough to build a faithful remote HITL channel with zero new browser instances.

The workaround: CDP relay → web viewer

Architecture:

[headless Chromium (cortex-scout)]
        │ CDP (ws://127.0.0.1:<random port>, loopback only)
        ▼
[relay: Node.js, ws@8]
   - discovers the CDP port via `ss -tlnp` (the chromium argv still says
     `--remote-debugging-port=0`, so /proc parsing does NOT give the real port)
   - connects to the page target
   - Page.startScreencast (JPEG frames)  ──► viewers
   - Input.dispatchMouseEvent / Input.dispatchKeyEvent / Page.navigate ◄── viewers
        │ HTTP :9092 + ?token= (auth)
        ▼
[browser of the human operator: static page, <canvas> + WebSocket]

Validation performed on the real system:

• CDP discovery: ss -tlnp reliably finds the chromium listener (argv keeps --remote-debugging-port=0);
• Page.startScreencast streams JPEG frames continuously (confirmed 3+ frames on visual change);
• Input.dispatchMouseEvent (mouseMoved/mousePressed/mouseReleased) and Input.dispatchKeyEvent are accepted with no error;
• Page.navigate works from the viewer;
• the session is the same browser cortex-scout uses: cookies/profile untouched, same process PID observed from both the MCP tool and the relay;
• scout_browser_automate and visual_scout keep working normally after a reconnect.

This turns the existing headless session into a remote "visible browser" for the duration of a wall, without touching cortex-scout's code.

Remaining gaps / not yet solved

• Port discovery across restarts: the CDP port changes on every cortex-scout restart; the relay currently re-discovers it with ss each time a viewer connects. A stable channel (e.g. a documented fixed port or a handshake file) would be cleaner.
• Multiple concurrent clients: Chrome tolerates several CDP clients, but concurrent viewer sessions were not stress-tested against an active scout_browser_automate run.
• Auth/TLS: the PoC uses a static query-string token; production use should use TLS + stronger auth, and never expose the raw CDP port (full browser control) directly.
• Walls that need a real human click with device-level trust (e.g. strict reCAPTCHA v3 scoring) may still flag CDP-dispatched input; this is inherent to any CDP-based approach.

Possible project-level improvements (nice to have)

1. Document the headless + HITL limitation and this relay pattern in the README.
2. Consider a CORTEX_SCOUT_CDP_PORT env var to pin --remote-debugging-port (instead of 0) for stable external attach.
3. Consider a built-in "remote HITL" transport (e.g. serve a small viewer next to the HTTP server when CORTEX_SCOUT_REMOTE_HITL=1) so no external relay is needed.

Links

• Prior context: issue dependancies for HITL #4 "dependancies for HITL" (build deps for the full build on Debian) — closed; this issue is about the headless runtime limitation, not the build.

[jsapede]

Operational update — solution validated in real conditions (2026-08-17)

What was validated

1. Full build (--all-features) on Linux x86_64

• Rust 1.97.1 + protoc + GTK/X11/ALSA dev libraries
• Required fix: cargo update -p ethnum --precise 1.5.3 (transmute broken by Rust 1.97)
• Build ~100 min (6 cores), binary 256 MB
• non_robot_search feature confirmed active (checked via strings <bin> | grep -c non_robot_search ≈ 700+)

2. Headless limitation confirmed

• hitl_web_fetch with the full build → user cancelled: the binary works, but no visible window can open without DISPLAY. Structural limitation confirmed.

3. The CDP relay is the operational solution

• Architecture: CDP port discovery via ss (chromium keeps --remote-debugging-port=0 in argv but listens on a real port) → WebSocket connection → web viewer (canvas + inputs)
• Critical implementation points (lessons learned):
  • Continuous screencast = bottleneck: Page.startScreencast at 1280x800 saturated the software renderer (Page.captureScreenshot went from ~34ms to ~2.6s). Solution: on-demand captures (immediate JPEG + delayed 1s after each action)
  • Serialized captures: only one Page.captureScreenshot in flight, recapture if requested meanwhile (avoids lost clicks)
  • Auto re-scan: port re-discovery every 8s, including after initial failure (the relay re-attaches by itself after a cortex-scout restart)
  • Keyboard: Input.dispatchKeyEvent must include text for printable keys (without it, nothing gets typed into fields)
  • Viewer: canvas resized to the real resolution of each frame + clearRect before drawing
• Real-world tests:
  • CAPTCHA: Google reCAPTCHA v2 demo — the challenge displays in the viewer, clicks are relayed (reCAPTCHA validation itself may loop: headless fingerprint detected, out of the relay's scope)
  • Login: practicetestautomation.com (student/Password123) — login succeeded via the viewer
  • Cookies: the viewer session = the same chromium as scout_browser_automate (profile ~/.cortex-scout/agent_profile) → identifications made in the viewer are reusable by subsequent calls

Key code: CDP discovery (Node.js)

// cortex-scout launches chromium with --remote-debugging-port=0:
// argv keeps 0, but chromium listens on a real port — find it via `ss`.
const { execFileSync } = require('child_process');
function findCdpPort() {
  let out;
  try { out = execFileSync('ss', ['-tlnp'], { encoding: 'utf8' }); } catch { return null; }
  for (const line of out.split('\n')) {
    if (!/chromium/.test(line)) continue;
    const m = line.match(/[\d.]+:(\d+)\s+/);
    const pid = line.match(/pid=(\d+)/);
    if (m && pid) return { port: +m[1], pid: +pid[1] };
  }
  return null;
}

Key code: on-demand serialized capture

let captureBusy = false, capturePending = false;
async function captureNow(state) {
  if (captureBusy) { capturePending = true; return; }
  captureBusy = true;
  try {
    const r = await send(state, 'Page.captureScreenshot',
                         { format: 'jpeg', quality: 45, captureBeyondViewport: false });
    if (r && r.data) broadcast({ type: 'frame', data: r.data });
  } catch (e) { console.error('capture:', e.message); }
  captureBusy = false;
  if (capturePending) { capturePending = false; setTimeout(() => captureNow(state), 50); }
}
// after each action: immediate capture + delayed capture (let the challenge open)
function captureAction(state) {
  captureNow(state);
  setTimeout(() => captureNow(state), 1000);
}

Key code: keyboard events need text for printable keys

// viewer side (keydown handler)
const params = { type: 'keyDown', key: e.key, code: e.code, windowsVirtualKeyCode: e.keyCode };
if (e.key.length === 1) params.text = e.key;   // REQUIRED for printable chars
ws.send(JSON.stringify({ type: 'key', params }));

Deployment (systemd service)

/root/cortex-deport/           # server.js, viewer.html, README.md, cortex-deport.service
cortex-deport.service          # port 9092, token via CORTEX_DEPORT_TOKEN env
URL: http://<host>:9092/?token=<TOKEN>

Follow-up requests (nice to have, non-blocking)

1. CDP port pinning: CORTEX_SCOUT_CDP_PORT (instead of --remote-debugging-port=0) for stable attach without ss
2. Built-in HITL: serve a viewer directly from the HTTP server when CORTEX_SCOUT_REMOTE_HITL=1 (removes the external relay)
3. Headless documentation: state in the README that hitl_web_fetch requires a desktop session, and that the headless fallback is the CDP relay

Pitfall for maintainers

The release pipeline does not provide linux-x64 binaries on GitHub (v3.3.7 only ships arm64/macos/windows) — any x64 binary must be compiled from source. Adding a linux-x64 asset to the release pipeline would ease adoption.

---

Issue #12: Headless deployments — HITL tools cannot open a visible window.