Skip to content

Commit fbf3d03

Browse files
committed
fix: report live video as macOS-only on Windows and Linux builds
Non-macOS builds link native_stubs.c instead of the macOS native bridge, so neither iOS simulator nor Android emulator H.264 streaming can work there. The browser previously showed the raw JSON stub error in the stream area and kept retrying WebRTC offers that could never succeed. Server: - Add platform.rs as the single source of truth for the live-video capability, its user-facing reason, and the CLI note. - Report `hostOs` and a `liveVideo` block from /api/health and /api/stream-quality. - Reject WebRTC offers early with 501 and the platform explanation via a new AppError::Unsupported variant. - Print a "Live video:" note after the service URLs on non-macOS hosts. - Share one clear message across the H.264 encoder stubs; drop the unused WIP helper and the duplicated #[test] attribute. Client: - Parse `{"error": ...}` bodies from failed offer and stream-config posts. - Read `liveVideo` from the stream-quality response, pause the live stream, and show the reason in the viewport instead of the reconnect loop. - Disable the stream transport, encoder, frame rate, and resolution controls with a "Requires macOS" note when the server cannot stream. Docs: platform support table, health/REST field docs, video guide, and a troubleshooting entry for the message.
1 parent f331672 commit fbf3d03

22 files changed

Lines changed: 459 additions & 33 deletions

‎AGENTS.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,13 @@ The native side should own anything that depends on macOS frameworks, `xcrun sim
2020
Defines REST routes for simulator control, health, metrics, and chrome assets.
2121
- `packages/server/src/transport/webrtc.rs`
2222
Exposes the H.264 WebRTC offer/answer endpoint for browser live video.
23+
- `packages/server/src/platform.rs`
24+
Single source of truth for host platform capabilities. Live H.264 video
25+
(iOS simulator and Android emulator) exists only in the macOS build; the
26+
Windows and Linux builds compile `packages/server/native_stubs.c` in place
27+
of the native bridge. Health, stream-quality, the WebRTC offer error, the
28+
CLI banner, and the browser client all read this module's `liveVideo`
29+
capability instead of hard-coding platform checks.
2330
- `packages/server/src/webkit.rs`
2431
Discovers simulator WebKit Remote Inspector targets and bridges WebInspectorUI
2532
WebSocket traffic to the simulator `webinspectord` binary-plist socket.

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ view inside the editor.
3939

4040
## Features
4141

42-
- Supports native H.264 streaming for both iOS simulators and Android emulators
42+
- Supports native H.264 streaming for both iOS simulators and Android emulators (live video requires the macOS build; the Windows and Linux CLIs manage Android emulators without a browser stream)
4343
- Full simulator control & inspection using private iOS accessibility APIs and Android UIAutomator - available using `simdeck` CLI
4444
- Real-time screen `describe` command using accessibility view tree - available in token-efficient format for agents
4545
- Profiling built-in: CPU, memory, disk writes, network throughput, hang signals, and stack sampling

‎docs/api/health.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@ Example:
2121
"serverKind": "launchAgent",
2222
"timestamp": 1714094761.234,
2323
"videoCodec": "auto",
24+
"hostOs": "macos",
25+
"liveVideo": { "supported": true, "requires": "macos" },
2426
"androidGpu": "host",
2527
"lowLatency": false,
2628
"realtimeStream": true,
@@ -47,10 +49,19 @@ Important fields:
4749
| `httpPort` | Port serving UI and API |
4850
| `serverKind` | `launchAgent` or `standalone` |
4951
| `videoCodec` | Requested codec mode: `auto`, `hardware`, or `software` |
52+
| `hostOs` | Server build target: `macos`, `windows`, or `linux` |
53+
| `liveVideo` | Whether this build can encode the live H.264 stream |
5054
| `androidGpu` | Android emulator renderer mode for SimDeck-owned boots |
5155
| `streamQuality` | Active stream profile and limits |
5256
| `webRtc` | ICE settings the browser should use |
5357

58+
`liveVideo` is `{ "supported": true, "requires": "macos" }` on macOS. Windows
59+
and Linux builds return `supported: false` plus a `reason` string, because live
60+
H.264 encoding for both iOS simulators and Android emulators lives in the macOS
61+
native bridge. Clients should show that reason instead of opening a WebRTC
62+
offer; the offer endpoint answers `501 Not Implemented` with the same message.
63+
`GET /api/stream-quality` includes the same `liveVideo` block.
64+
5465
When auth is required, the `401` JSON body still includes `serverId`, `advertiseHost`, `hostId`, `hostName`, `httpPort`, and `serverKind` so native clients can group endpoints before pairing.
5566

5667
## Metrics

‎docs/api/rest.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,10 @@ curl -X POST \
5252
| `GET` | `/api/stream-quality` | Current stream quality settings |
5353
| `POST` | `/api/stream-quality` | Update stream quality settings |
5454

55-
See [Health and metrics](/api/health) for details.
55+
See [Health and metrics](/api/health) for details. Both `/api/health` and
56+
`/api/stream-quality` include a `liveVideo` block; check `liveVideo.supported`
57+
before opening a WebRTC offer, because Windows and Linux builds cannot encode
58+
the live stream and answer offers with `501` and the `liveVideo.reason` text.
5659

5760
## Devices
5861

‎docs/guide/installation.md‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,28 @@
22

33
## Requirements
44

5-
- macOS on Apple Silicon.
5+
- macOS on Apple Silicon for the full experience.
66
- Xcode with the simulator runtimes you want to use.
77
- Node.js 18 or newer.
88

9+
## Platform support
10+
11+
The npm package installs a native CLI for macOS, Windows, and Linux, but only
12+
the macOS build includes the native simulator bridge and H.264 encoder.
13+
14+
| Capability | macOS | Windows / Linux |
15+
| -------------------------------------------------- | ----- | ----------------------- |
16+
| iOS simulator control, inspection, and streaming | Yes | No |
17+
| Android emulator control and inspection | Yes | Yes |
18+
| Live H.264 browser stream (iOS and Android) | Yes | No |
19+
| `--video-codec`, stream quality, and encoder menus | Yes | Reported as unavailable |
20+
21+
On Windows and Linux the CLI prints a `Live video:` note after the service
22+
URLs, `GET /api/health` and `GET /api/stream-quality` report
23+
`liveVideo.supported: false` with the reason, the browser shows that reason in
24+
place of the device screen, and the WebRTC offer endpoint answers `501` with
25+
the same message. See [Video and streaming](./video.md) for the encoder details.
26+
927
Check Xcode selection if you have multiple installs:
1028

1129
```sh

‎docs/guide/troubleshooting.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,20 @@ Android IDs in SimDeck use `android:<avd-name>`.
107107

108108
## Stream is black or stuck
109109

110+
### Live video requires macOS
111+
112+
```text
113+
Live H.264 video streaming requires macOS. This SimDeck build for Windows can manage devices but does not include the native H.264 encoder...
114+
```
115+
116+
This is expected on the Windows and Linux builds, for both iOS simulators and
117+
Android emulators. Those builds link a stub instead of the macOS native bridge
118+
that owns VideoToolbox and x264 encoding, so the browser stream, the encoder
119+
menu, and `--video-codec` cannot work there. Device control, `describe`,
120+
screenshots, and the inspectors still run. Use a Mac to see the live screen, or
121+
pair a Windows or Linux browser with a SimDeck service running on a Mac. Check
122+
`liveVideo.supported` in `GET /api/health` when scripting against the service.
123+
110124
### Timed out waiting for the first frame
111125

112126
Try software encoding:

‎docs/guide/video.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,18 @@ SimDeck streams live device video to the browser. Local iOS sessions default to
55
iOS simulator H.264 uses VideoToolbox for hardware encoding and x264 for software encoding.
66
Android emulator H.264 uses the emulator gRPC `streamScreenshot` API when SimDeck owns the boot. SimDeck receives raw RGBA frames, pads odd dimensions for H.264, and encodes them on the Mac. If the gRPC endpoint is unavailable, SimDeck falls back to the emulator `-share-vid` display surface and reads BGRA frames from the `videmulator<console-port>` shared memory region.
77

8+
## Platform requirement
9+
10+
Live video needs the macOS build. Both encoders above live in the macOS native
11+
bridge, so the Windows and Linux CLIs link a stub instead and cannot stream iOS
12+
simulators or Android emulators to the browser. Those builds still manage
13+
Android emulators, but `--video-codec`, stream quality, and the encoder menu
14+
have no effect there. The service reports this through `liveVideo` in
15+
`GET /api/health` and `GET /api/stream-quality`, prints a `Live video:` note
16+
when it starts, and answers WebRTC offers with `501 Not Implemented` and the
17+
same explanation. The browser client hides the retry loop and shows that
18+
message in place of the device screen.
19+
820
## When encoding runs
921

1022
SimDeck starts encoding when a browser stream needs H.264 frames. For iOS, the

‎packages/client/src/api/types.ts‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -187,12 +187,25 @@ export interface ChromeDevToolsTargetDiscovery {
187187
warnings: string[];
188188
}
189189

190+
/**
191+
* Whether the server build can encode the live H.264 browser stream. Only the
192+
* macOS build links the native encoder; Windows and Linux builds report
193+
* `supported: false` with a user-facing `reason`.
194+
*/
195+
export interface LiveVideoCapability {
196+
supported: boolean;
197+
requires?: string;
198+
reason?: string | null;
199+
}
200+
190201
export interface HealthResponse {
191202
ok: boolean;
192203
serverId?: string;
193204
advertiseHost?: string;
194205
hostId?: string;
195206
hostName?: string;
207+
hostOs?: string;
208+
liveVideo?: LiveVideoCapability;
196209
httpPort?: number;
197210
serverKind?:
198211
| "launchAgent"

‎packages/client/src/app/AppShell.tsx‎

Lines changed: 26 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@ import type {
3838
AccessibilitySourcePreference,
3939
AccessibilityTreeResponse,
4040
ChromeProfile,
41+
LiveVideoCapability,
4142
SimulatorMetadata,
4243
SimulatorStateResponse,
4344
TouchPhase,
@@ -55,6 +56,7 @@ import {
5556
simulatorUsesInsetChromeButtons,
5657
} from "../features/simulators/simulatorDisplay";
5758
import { useSimulatorList } from "../features/simulators/useSimulatorList";
59+
import { liveVideoUnavailableReason } from "../features/stream/liveVideoCapability";
5860
import { sendWebRtcControlMessage } from "../features/stream/streamWorkerClient";
5961
import type {
6062
StreamConfig,
@@ -174,6 +176,7 @@ clearLegacyVolatileUiState();
174176

175177
interface StreamQualityResponse {
176178
ok?: boolean;
179+
liveVideo?: LiveVideoCapability;
177180
quality?: {
178181
fps?: number;
179182
maxEdge?: number;
@@ -413,9 +416,9 @@ function simulatorDisplayReady(simulator: SimulatorMetadata): boolean {
413416
const display = simulator.privateDisplay;
414417
return Boolean(
415418
simulator.isBooted &&
416-
display?.displayReady &&
417-
display.displayWidth > 0 &&
418-
display.displayHeight > 0,
419+
display?.displayReady &&
420+
display.displayWidth > 0 &&
421+
display.displayHeight > 0,
419422
);
420423
}
421424

@@ -625,6 +628,10 @@ export function AppShell({
625628
);
626629
const [streamConfigApplyKey, setStreamConfigApplyKey] = useState(0);
627630
const [streamConfigReady, setStreamConfigReady] = useState(false);
631+
// Non-empty when the connected server build cannot encode live video (for
632+
// example the Windows or Linux CLI). The stream stays paused and the reason
633+
// is shown instead of retrying WebRTC offers that can never succeed.
634+
const [liveVideoUnavailable, setLiveVideoUnavailable] = useState("");
628635
const [touchIndicators, setTouchIndicators] = useState<TouchIndicator[]>([]);
629636
const [selectedSimulatorState, setSelectedSimulatorState] =
630637
useState<SimulatorStateResponse | null>(null);
@@ -847,6 +854,7 @@ export function AppShell({
847854
if (requestId !== streamConfigRequestIdRef.current) {
848855
return;
849856
}
857+
setLiveVideoUnavailable(liveVideoUnavailableReason(response.liveVideo));
850858
if (
851859
!options?.ignoreUserGrace &&
852860
Date.now() - streamConfigUserChangeAtRef.current <
@@ -930,7 +938,7 @@ export function AppShell({
930938
streamCanvasKey,
931939
} = useLiveStream({
932940
canvasElement: streamCanvasElement,
933-
paused: !streamConfigReady,
941+
paused: !streamConfigReady || Boolean(liveVideoUnavailable),
934942
remote: remoteStream,
935943
simulator: selectedSimulator,
936944
streamConfig: effectiveStreamConfig,
@@ -1035,8 +1043,8 @@ export function AppShell({
10351043
selectedSimulator != null && shouldRenderNativeChrome(selectedSimulator);
10361044
const deviceChromeToggleActive = Boolean(
10371045
selectedSupportsChrome &&
1038-
deviceChromeVisible &&
1039-
!selectedChromeAssetsFailed,
1046+
deviceChromeVisible &&
1047+
!selectedChromeAssetsFailed,
10401048
);
10411049
const shouldRenderChrome = deviceChromeToggleActive;
10421050
const viewportChromeProfile = shouldRenderChrome ? chromeProfile : null;
@@ -1127,8 +1135,8 @@ export function AppShell({
11271135
: recordingOverlayLabel || captureStatus?.label || "";
11281136
const captureOverlayBusy = Boolean(
11291137
isInstallingApp ||
1130-
captureStatus?.busy ||
1131-
screenRecording?.phase === "stopping",
1138+
captureStatus?.busy ||
1139+
screenRecording?.phase === "stopping",
11321140
);
11331141
const autoViewportOffsetY =
11341142
viewMode === "manual" ? 0 : -zoomDockReservedHeight / 2;
@@ -2153,13 +2161,15 @@ export function AppShell({
21532161
const viewportStatusOverlayLabel =
21542162
(providerDisconnected ? NOT_CONNECTED_MESSAGE : "") ||
21552163
simulatorStatusOverlayLabel ||
2164+
liveVideoUnavailable ||
21562165
streamStatusMessage ||
21572166
(selectedSimulator ? visibleListError : "");
21582167
const viewportHasStreamError = Boolean(
21592168
providerDisconnected ||
2160-
streamStatus.state === "error" ||
2161-
visibleStreamError ||
2162-
(selectedSimulator && visibleListError),
2169+
liveVideoUnavailable ||
2170+
streamStatus.state === "error" ||
2171+
visibleStreamError ||
2172+
(selectedSimulator && visibleListError),
21632173
);
21642174
const deviceTransform = `translate(${pan.x}px, ${pan.y + autoViewportOffsetY}px) scale(${effectiveZoom})`;
21652175
const chromeScreenRect = computeChromeScreenRect(
@@ -3812,6 +3822,7 @@ export function AppShell({
38123822
recordingActive={screenRecording?.phase === "recording"}
38133823
recordingStopping={screenRecording?.phase === "stopping"}
38143824
remoteStream={remoteStream}
3825+
liveVideoUnavailableReason={liveVideoUnavailable}
38153826
search={search}
38163827
selectedSimulator={selectedSimulator}
38173828
selectedSimulatorIdentifier={selectedSimulatorDetail}
@@ -3821,8 +3832,8 @@ export function AppShell({
38213832
}}
38223833
showBootButton={Boolean(
38233834
selectedSimulator &&
3824-
!selectedSimulator.isBooted &&
3825-
!selectedSimulatorTransitionKind,
3835+
!selectedSimulator.isBooted &&
3836+
!selectedSimulatorTransitionKind,
38263837
)}
38273838
streamConfig={effectiveStreamConfig}
38283839
streamTransport={streamTransport}
@@ -4325,8 +4336,8 @@ function normalizeMaxEdge(
43254336
function isAndroidSimulator(simulator: SimulatorMetadata | null): boolean {
43264337
return Boolean(
43274338
simulator?.platform === "android-emulator" ||
4328-
simulator?.deviceTypeIdentifier === "android-emulator" ||
4329-
simulator?.udid.startsWith("android:"),
4339+
simulator?.deviceTypeIdentifier === "android-emulator" ||
4340+
simulator?.udid.startsWith("android:"),
43304341
);
43314342
}
43324343

‎packages/client/src/features/simulators/SimulatorMenu.tsx‎

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,8 @@ interface SimulatorMenuProps {
4949
recordingActive: boolean;
5050
recordingStopping: boolean;
5151
remoteStream?: boolean;
52+
/** Set when the server build cannot stream video; disables stream controls. */
53+
liveVideoUnavailableReason?: string;
5254
selectedSimulator: SimulatorMetadata | null;
5355
showBootButton: boolean;
5456
showStopButton: boolean;
@@ -94,6 +96,7 @@ export function SimulatorMenu({
9496
recordingActive,
9597
recordingStopping,
9698
remoteStream = false,
99+
liveVideoUnavailableReason = "",
97100
selectedSimulator,
98101
showBootButton,
99102
showStopButton,
@@ -123,6 +126,7 @@ export function SimulatorMenu({
123126
const canRotateSelectedSimulator =
124127
selectedSimulator != null &&
125128
!simulatorHasFixedOrientation(selectedSimulator);
129+
const streamControlsDisabled = liveVideoUnavailableReason !== "";
126130
return (
127131
<div className="menu-wrap" ref={menuRef}>
128132
<button
@@ -146,13 +150,24 @@ export function SimulatorMenu({
146150
<div className="menu-section-heading">
147151
<span className="menu-section-title">Stream</span>
148152
<span className="menu-section-meta">
149-
{formatStreamConfigSummary(streamConfig, streamTransport)}
153+
{streamControlsDisabled
154+
? "Requires macOS"
155+
: formatStreamConfigSummary(
156+
streamConfig,
157+
streamTransport,
158+
)}
150159
</span>
151160
</div>
161+
{streamControlsDisabled ? (
162+
<p className="menu-note" title={liveVideoUnavailableReason}>
163+
{LIVE_VIDEO_UNAVAILABLE_NOTE}
164+
</p>
165+
) : null}
152166
<label className="menu-field">
153167
<span className="menu-field-label">Transport</span>
154168
<select
155169
className="menu-select"
170+
disabled={streamControlsDisabled}
156171
onChange={(event) =>
157172
onStreamTransportChange(
158173
event.currentTarget.value as StreamTransport,
@@ -171,8 +186,14 @@ export function SimulatorMenu({
171186
{STREAM_ENCODERS.map((option) => (
172187
<button
173188
className={`menu-option ${streamConfig.encoder === option.value ? "active" : ""}`}
189+
disabled={streamControlsDisabled}
174190
key={option.value}
175191
onClick={() => onStreamEncoderChange(option.value)}
192+
title={
193+
streamControlsDisabled
194+
? liveVideoUnavailableReason
195+
: undefined
196+
}
176197
type="button"
177198
>
178199
{option.label}
@@ -183,6 +204,7 @@ export function SimulatorMenu({
183204
{[...activeFpsOption, ...fpsOptions].map((option) => (
184205
<button
185206
className={`menu-option ${streamConfig.fps === option.value ? "active" : ""}`}
207+
disabled={streamControlsDisabled}
186208
key={option.value}
187209
onClick={() => onStreamFpsChange(option.value)}
188210
type="button"
@@ -195,6 +217,7 @@ export function SimulatorMenu({
195217
<span className="menu-field-label">Resolution</span>
196218
<select
197219
className="menu-select"
220+
disabled={streamControlsDisabled}
198221
onChange={(event) =>
199222
onStreamQualityChange(
200223
event.currentTarget.value as StreamQualityPreset,
@@ -397,6 +420,9 @@ export function SimulatorMenu({
397420
);
398421
}
399422

423+
const LIVE_VIDEO_UNAVAILABLE_NOTE =
424+
"Live H.264 streaming requires macOS. Device control still works from this host.";
425+
400426
const STREAM_ENCODERS: Array<{ label: string; value: StreamEncoder }> = [
401427
{ label: "Auto", value: "auto" },
402428
{ label: "Hardware", value: "hardware" },

0 commit comments

Comments
 (0)