Skip to content

fix(recording): export touch overlay at capture size, never black (#2707) - #2767

Open
thymikee wants to merge 1 commit into
mainfrom
t3code/fix-record-touch-overlay-export
Open

thymikee wants to merge 1 commit into
mainfrom
t3code/fix-record-touch-overlay-export

Conversation

@thymikee

Copy link
Copy Markdown
Member

Summary

A record clip collapsed to ~220x480 (and went all-black on long captures) whenever the touch overlay drew, while exiting 0. The burn-in re-encoded through AVAssetExportPresetMediumQuality, a fixed-canvas preset that caps the long edge at 480px. The trigger was the overlay path, not rot90 inner-panel tracks.

  • The overlay now exports through the geometry-preserving preset, so a touched clip matches the raw capture size.
  • The Swift tool verifies its own output — resolved size equals the capture, frames decode, the clip is not truncated, and it did not go all-black — and throws otherwise. The finalizer then keeps the raw recording and reports overlayWarning on record stop instead of returning a broken file at exit 0.
  • --quality no longer changes Apple output (it never affected the raw capture and no longer picks a resolution preset); CLI help and website docs corrected.
  • ADR 0025 and device-verification.md rot90 claims corrected.
  • New @agent-device/capture-kit/recording-mp4-track-size reader backs the regression test.

Closes #2707. 12 files.

Validation

Commit 3e404ec35. pnpm check:affected --run passed all runnable checks (fallow complexity, full unit suite 11085, layering, package, bundle-owner, packaged-runner-swift, command-docs).

An offline AVFoundation harness reproduced the collapse (1206x2622220x480, landscape 1310x602480x220) and confirms the fix preserves geometry and rejects black/truncated output. Unit tests cover the MP4 reader (v0/v1 display-vs-coded, rot90, coded fallback, truncated) and the finalizer drop-and-report path. test/integration/recording-overlay.test.ts compares a touched export to a --hide-touches control, asserting equal track size and that the overlay drew.

Open risk: re-measuring the completion-condition matrix on a booted non-rot90 simulator and the iPhone Duo inner panel (AGENT_DEVICE_RECORDING_E2E) is the device-lane step and has not been run here.

@github-actions

github-actions Bot commented Sep 22, 2026

Copy link
Copy Markdown
PR Preview Action v1.8.1

QR code for preview link

🚀 View preview at
https://callstack.github.io/agent-device/pr-preview/pr-2767/

Built to branch gh-pages at 2026-09-22 20:21 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

@github-actions

github-actions Bot commented Sep 22, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
Installed (including dependencies) 4.78 MB 4.78 MB +3.9 kB
Package (unpacked) 4.78 MB 4.78 MB +3.9 kB
Package (download) 1.43 MB 1.43 MB +960 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 27.0 ms 27.5 ms +0.4 ms
CLI --help 79.6 ms 78.4 ms -1.2 ms

)

The overlay burn-in re-encoded through AVAssetExportPresetMediumQuality, a
fixed-canvas preset that caps the long edge at 480px, collapsing any touch
recording to ~220x480 (480x220 landscape) and going all-black on long clips,
always with exit 0. --hide-touches and an empty gesture list copy the raw
capture through, so only the composited path was affected.

Both quality tiers now export through the one geometry-preserving preset
(HighestQuality); the hardware encoder makes the full-resolution re-encode
cheap. The compositor now verifies its own output against the raw before the
caller adopts it — resolved track size must match, and a uniformly black track
whose raw had visible content is rejected — and throws on failure so the
overlay is dropped, the raw kept, and the choice reported as overlayWarning
rather than a broken file. Adds an mp4 transform-aware track-size reader and a
device-lane assertion that a touched export equals its --hide-touches control.
@thymikee
thymikee force-pushed the t3code/fix-record-touch-overlay-export branch from 3e404ec to 717f9f3 Compare September 22, 2026 20:19
@thymikee

Copy link
Copy Markdown
Member Author

Reviewed at 717f9f3. The change is not proven on the route it touches: nobody has run a real record start, tap, record stop sequence at this head, the #2707 completion matrix in the PR body is still open, and the new e2e case in test/integration/recording-overlay.test.ts#L16 only runs under AGENT_DEVICE_RECORDING_E2E. So two things stay unproven: whether the touched export now matches the --hide-touches size, and whether the long-clip all-black case is actually gone rather than just detected after the fact, with its cause never shown. Can you run the gated e2e, or the CLI sequence by hand, at this head on a non-rot90 iOS simulator (for example iPhone 17) and on the iPhone Duo inner panel, and attach: a record stop --json output for a short tap clip with no overlayWarning and a track size equal to a --hide-touches control of the same tap, plus a clip of about 90s with taps and scroll showing no overlayWarning, the same size as the control, and a non-zero mean luma?

packages/capture-kit/src/recording/mp4-track-size.ts#L182 nearly duplicates childAtoms in packages/capture-kit/src/recording/mp4-atoms.ts — same MAX_SIBLINGS_PER_LEVEL, same 64-bit size==1 branch, same safe-integer and container-end checks, just walking a Buffer instead of an fd — and it disagrees with the original on size==0 (extend-to-end vs stop-here). This adds about 290 lines of test-only parser, fixtures, and a new package subpath to production capture-kit, to serve one gated integration test, when test/integration/support/recording-inspect.swift already computes the same transform-applied size via AVFoundation (resolvedRenderSize, lines 78 and 184) without outputting it. Could recording-inspect.swift just add renderWidth/renderHeight to its manifest, have recordTap read the size through runRecordingInspect, and let mp4-track-size.ts, its test, its fixtures, the package export and the boundary allow-list entry be deleted, keeping AVFoundation as the one owner of track geometry that the overlay tool already uses?

Not blocking: the #2707-labeled tests in src/platform-runtime-screen-recording-finalizer-host.test.ts#L84 exercise a catch-to-overlayWarning path already on main, not the new Swift checks, and the "unplayable" test's title says "keeping the raw recording" even though overlay.ts#L98 has already renamed the composite over the raw file, so these could use retitling or a check that's red on the old preset, but that's optional either way.

All 21 CI checks are green, but none of them runs the AGENT_DEVICE_RECORDING_E2E device lane, so this green run says nothing about the changed export path. I did not reproduce the claim that the MediumQuality preset collapses the track to 220x480; that rests on an offline harness. The 32px size tolerance and the luma thresholds are not calibrated against real encoder output in any test, and a moov with no tkhd falls back to the stsd coded size without applying the transform, so a rot90 track in that case would come back transposed rather than undefined. Please attach the live device run showing the touched size equals the control with no overlayWarning and the 90s clip is not black, and replace the TS MP4 parser with the size recording-inspect.swift can already provide, before this is ready to merge.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Recording touch overlay exports at ~220x480 or all-black whenever touches are drawn

1 participant