feat: stream Android emulators on Windows and Linux with OpenH264

Non-macOS builds link native_stubs.c instead of the macOS simulator bridge,
so the Android emulator WebRTC source had no H.264 encoder and every stream
failed with the stub error. Frame capture (emulator gRPC screenshots) and the
WebRTC transport were already cross-platform Rust; only the encoder was
missing.

Server:
- Add transport/software_h264.rs: an OpenH264 encoder (openh264 crate,
  compiled from source, BSD) that turns RGBA/BGRA emulator frames into
  Annex B baseline H.264. Bitrate follows the macOS budget (bits per pixel
  with a minimum), keyframes honor RTCP requests, and the encoder rebuilds
  on size or quality changes. Unit tests cover the budget math, frame
  validation, keyframe forcing, and reinitialization.
- Route the Android WebRTC source through it on non-macOS builds; macOS
  keeps the native VideoToolbox/x264 path behind the C ABI. Frame
  publishing is shared between both paths.
- Reframe platform.rs around iOS simulator support: liveVideo now reports
  supported: true everywhere with encoder ("native" or "openh264"),
  androidEmulator, iosSimulator, and an iosSimulatorReason off macOS.
  WebRTC offers for iOS simulators on non-macOS answer 501; Android offers
  proceed.
- CLI banner note now says Android streams with software H.264 and iOS
  simulators require macOS.

CI: add a Linux and Windows cargo clippy/test job so the OpenH264 module and
the native stubs are linted and tested where they are actually compiled.

Docs: per-platform encoder table, health/REST field docs, troubleshooting
entries for the iOS message and slow software encoding, AGENTS pointer.

Cargo.lock is not updated here (no Rust toolchain on the authoring machine);
the first cargo build adds the openh264 entries.

Author: NathanWalker

.github/workflows/ci.yml

@@ -50,6 +50,40 @@ jobs:
       - name: Rust unit tests
         run: cargo test --manifest-path packages/server/Cargo.toml
 
+  rust-non-macos:
+    name: Rust unit tests (${{ matrix.os }})
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: dtolnay/rust-toolchain@stable
+        with:
+          components: clippy
+
+      - name: Cache Rust build outputs
+        uses: actions/cache@v4
+        with:
+          path: |
+            ~/.cargo/git
+            ~/.cargo/registry
+            packages/server/target
+          key: rust-${{ runner.os }}-${{ hashFiles('packages/server/Cargo.lock', 'packages/server/Cargo.toml', 'packages/server/build.rs', 'packages/server/src/**/*.rs', 'packages/server/native_stubs.c') }}
+          restore-keys: |
+            rust-${{ runner.os }}-
+
+      # The macOS job cannot compile the OpenH264 software encoder or the
+      # native stubs, so lint and test them where they are actually built.
+      - name: Clippy
+        run: cargo clippy --manifest-path packages/server/Cargo.toml --all-targets -- -D warnings
+
+      - name: Rust unit tests
+        run: cargo test --manifest-path packages/server/Cargo.toml
+
   client:
     name: Client lint, build, and tests
     runs-on: ubuntu-latest

AGENTS.md

@@ -21,12 +21,18 @@ The native side should own anything that depends on macOS frameworks, `xcrun sim
 - `packages/server/src/transport/webrtc.rs`
   Exposes the H.264 WebRTC offer/answer endpoint for browser live video.
 - `packages/server/src/platform.rs`
-  Single source of truth for host platform capabilities. Live H.264 video
-  (iOS simulator and Android emulator) exists only in the macOS build; the
-  Windows and Linux builds compile `packages/server/native_stubs.c` in place
-  of the native bridge. Health, stream-quality, the WebRTC offer error, the
-  CLI banner, and the browser client all read this module's `liveVideo`
-  capability instead of hard-coding platform checks.
+  Single source of truth for host platform capabilities. iOS simulators exist
+  only in the macOS build; the Windows and Linux builds compile
+  `packages/server/native_stubs.c` in place of the native bridge. Health,
+  stream-quality, the WebRTC offer error, the CLI banner, and the browser
+  client all read this module's `liveVideo` capability instead of hard-coding
+  platform checks.
+- `packages/server/src/transport/software_h264.rs`
+  OpenH264 software encoder (the `openh264` crate, compiled from source) used
+  for Android emulator WebRTC streams on non-macOS builds. It emits Annex B
+  baseline H.264 so `transport/webrtc.rs` packetizes it exactly like the macOS
+  native encoder output. macOS keeps encoding Android frames through
+  `XCWH264Encoder` behind the C ABI.
 - `packages/server/src/webkit.rs`
   Discovers simulator WebKit Remote Inspector targets and bridges WebInspectorUI
   WebSocket traffic to the simulator `webinspectord` binary-plist socket.

README.md

@@ -39,7 +39,7 @@ view inside the editor.
 
 ## Features
 
-- Supports native H.264 streaming for both iOS simulators and Android emulators (live video requires the macOS build; the Windows and Linux CLIs manage Android emulators without a browser stream)
+- Supports native H.264 streaming for both iOS simulators and Android emulators (iOS simulators need the macOS build; Windows and Linux stream Android emulators with a built-in OpenH264 software encoder)
 - Full simulator control & inspection using private iOS accessibility APIs and Android UIAutomator - available using `simdeck` CLI
 - Real-time screen `describe` command using accessibility view tree - available in token-efficient format for agents
 - Profiling built-in: CPU, memory, disk writes, network throughput, hang signals, and stack sampling

docs/api/health.md

@@ -22,7 +22,12 @@ Example:
   "timestamp": 1714094761.234,
   "videoCodec": "auto",
   "hostOs": "macos",
-  "liveVideo": { "supported": true, "requires": "macos" },
+  "liveVideo": {
+    "supported": true,
+    "encoder": "native",
+    "iosSimulator": true,
+    "androidEmulator": true
+  },
   "androidGpu": "host",
   "lowLatency": false,
   "realtimeStream": true,
@@ -50,16 +55,19 @@ Important fields:
 | `serverKind`    | `launchAgent` or `standalone`                           |
 | `videoCodec`    | Requested codec mode: `auto`, `hardware`, or `software` |
 | `hostOs`        | Server build target: `macos`, `windows`, or `linux`     |
-| `liveVideo`     | Whether this build can encode the live H.264 stream     |
+| `liveVideo`     | Live stream encoder and per-platform device support     |
 | `androidGpu`    | Android emulator renderer mode for SimDeck-owned boots  |
 | `streamQuality` | Active stream profile and limits                        |
 | `webRtc`        | ICE settings the browser should use                     |
 
-`liveVideo` is `{ "supported": true, "requires": "macos" }` on macOS. Windows
-and Linux builds return `supported: false` plus a `reason` string, because live
-H.264 encoding for both iOS simulators and Android emulators lives in the macOS
-native bridge. Clients should show that reason instead of opening a WebRTC
-offer; the offer endpoint answers `501 Not Implemented` with the same message.
+`liveVideo.encoder` is `native` on macOS (VideoToolbox or x264 through the
+simulator bridge) and `openh264` on Windows and Linux, where Android emulator
+frames are encoded in software. `androidEmulator` is `true` everywhere.
+`iosSimulator` is `true` only on macOS; other builds add an
+`iosSimulatorReason` string, and the WebRTC offer endpoint answers iOS
+simulator offers there with `501 Not Implemented` and the same message.
+`supported` is `true` for every shipped build; a client should treat
+`supported: false` plus `reason` as "do not open a WebRTC offer".
 `GET /api/stream-quality` includes the same `liveVideo` block.
 
 When auth is required, the `401` JSON body still includes `serverId`, `advertiseHost`, `hostId`, `hostName`, `httpPort`, and `serverKind` so native clients can group endpoints before pairing.

docs/api/rest.md

@@ -53,9 +53,10 @@ curl -X POST \
 | `POST` | `/api/stream-quality`      | Update stream quality settings                             |
 
 See [Health and metrics](/api/health) for details. Both `/api/health` and
-`/api/stream-quality` include a `liveVideo` block; check `liveVideo.supported`
-before opening a WebRTC offer, because Windows and Linux builds cannot encode
-the live stream and answer offers with `501` and the `liveVideo.reason` text.
+`/api/stream-quality` include a `liveVideo` block. Android emulators stream on
+every platform (`encoder` is `native` on macOS and `openh264` elsewhere); check
+`liveVideo.iosSimulator` before opening an iOS simulator WebRTC offer, because
+Windows and Linux builds answer those with `501` and `iosSimulatorReason`.
 
 ## Devices
 

docs/guide/installation.md

@@ -8,21 +8,22 @@
 
 ## Platform support
 
-The npm package installs a native CLI for macOS, Windows, and Linux, but only
-the macOS build includes the native simulator bridge and H.264 encoder.
+The npm package installs a native CLI for macOS, Windows, and Linux. Only the
+macOS build includes the private simulator bridge, so iOS simulators need a
+Mac. Android emulators work everywhere, including the live browser stream.
 
-| Capability                                         | macOS | Windows / Linux         |
-| -------------------------------------------------- | ----- | ----------------------- |
-| iOS simulator control, inspection, and streaming   | Yes   | No                      |
-| Android emulator control and inspection            | Yes   | Yes                     |
-| Live H.264 browser stream (iOS and Android)        | Yes   | No                      |
-| `--video-codec`, stream quality, and encoder menus | Yes   | Reported as unavailable |
+| Capability                                       | macOS                | Windows / Linux        |
+| ------------------------------------------------ | -------------------- | ---------------------- |
+| iOS simulator control, inspection, and streaming | Yes                  | No                     |
+| Android emulator control and inspection          | Yes                  | Yes                    |
+| Android emulator live H.264 browser stream       | VideoToolbox or x264 | OpenH264 software      |
+| `--video-codec hardware`                         | VideoToolbox         | Falls back to software |
 
 On Windows and Linux the CLI prints a `Live video:` note after the service
-URLs, `GET /api/health` and `GET /api/stream-quality` report
-`liveVideo.supported: false` with the reason, the browser shows that reason in
-place of the device screen, and the WebRTC offer endpoint answers `501` with
-the same message. See [Video and streaming](./video.md) for the encoder details.
+URLs, and `GET /api/health` reports `liveVideo.encoder: "openh264"` with
+`liveVideo.iosSimulator: false`. Selecting an iOS simulator through the API
+there returns `501` with the same explanation. See
+[Video and streaming](./video.md) for the encoder details.
 
 Check Xcode selection if you have multiple installs:
 

docs/guide/troubleshooting.md

@@ -107,19 +107,32 @@ Android IDs in SimDeck use `android:<avd-name>`.
 
 ## Stream is black or stuck
 
-### Live video requires macOS
+### iOS simulators require macOS
 
 ```text
-Live H.264 video streaming requires macOS. This SimDeck build for Windows can manage devices but does not include the native H.264 encoder...
+iOS simulators require macOS. This SimDeck build for Windows can boot, control, and stream Android emulators, but the iOS simulator bridge is only available on macOS.
 ```
 
-This is expected on the Windows and Linux builds, for both iOS simulators and
-Android emulators. Those builds link a stub instead of the macOS native bridge
-that owns VideoToolbox and x264 encoding, so the browser stream, the encoder
-menu, and `--video-codec` cannot work there. Device control, `describe`,
-screenshots, and the inspectors still run. Use a Mac to see the live screen, or
-pair a Windows or Linux browser with a SimDeck service running on a Mac. Check
-`liveVideo.supported` in `GET /api/health` when scripting against the service.
+This is expected on the Windows and Linux builds. They link a stub instead of
+the macOS native bridge that owns iOS simulator control and the VideoToolbox
+and x264 encoders. Android emulators still boot, stream, and accept input on
+those hosts through the built-in OpenH264 software encoder. Use a Mac for iOS
+simulators, or pair a Windows or Linux browser with a SimDeck service running on
+a Mac. Check `liveVideo.iosSimulator` in `GET /api/health` when scripting
+against the service.
+
+### Android stream is slow on Windows or Linux
+
+Those builds encode with OpenH264 in software. Lower the encoded size first,
+then the frame rate:
+
+```sh
+simdeck service restart --stream-quality low
+```
+
+`GET /api/metrics` lists `androidEncoders[].encoder.native` with
+`latestEncodeLatencyUs` and `skippedFrames`; if encode latency stays above the
+frame interval, the host CPU cannot keep up with the selected profile.
 
 ### Timed out waiting for the first frame
 

docs/guide/video.md

@@ -5,17 +5,26 @@ SimDeck streams live device video to the browser. Local iOS sessions default to
 iOS simulator H.264 uses VideoToolbox for hardware encoding and x264 for software encoding.
 Android emulator H.264 uses the emulator gRPC `streamScreenshot` API when SimDeck owns the boot. SimDeck receives raw RGBA frames, pads odd dimensions for H.264, and encodes them on the Mac. If the gRPC endpoint is unavailable, SimDeck falls back to the emulator `-share-vid` display surface and reads BGRA frames from the `videmulator<console-port>` shared memory region.
 
-## Platform requirement
-
-Live video needs the macOS build. Both encoders above live in the macOS native
-bridge, so the Windows and Linux CLIs link a stub instead and cannot stream iOS
-simulators or Android emulators to the browser. Those builds still manage
-Android emulators, but `--video-codec`, stream quality, and the encoder menu
-have no effect there. The service reports this through `liveVideo` in
-`GET /api/health` and `GET /api/stream-quality`, prints a `Live video:` note
-when it starts, and answers WebRTC offers with `501 Not Implemented` and the
-same explanation. The browser client hides the retry loop and shows that
-message in place of the device screen.
+## Encoders per platform
+
+| Host            | iOS simulator stream          | Android emulator stream                  |
+| --------------- | ----------------------------- | ---------------------------------------- |
+| macOS           | VideoToolbox hardware or x264 | VideoToolbox hardware or x264            |
+| Windows / Linux | Not available                 | OpenH264 software, built into the binary |
+
+Both macOS encoders live in the macOS native bridge, which also owns iOS
+simulator control. The Windows and Linux CLIs link a stub in its place, so they
+cannot drive iOS simulators at all. Android emulator frames on those hosts are
+encoded in Rust with Cisco's OpenH264 (baseline profile, Annex B) and go
+through the same WebRTC path as on macOS. Stream quality profiles and the
+frame-rate and resolution controls apply there too; `--video-codec hardware`
+has no hardware encoder to select and behaves like `software`.
+
+The service reports this through `liveVideo` in `GET /api/health` and
+`GET /api/stream-quality` (`encoder` is `native` or `openh264`, `iosSimulator`
+is `false` off macOS), prints a `Live video:` note when it starts on Windows or
+Linux, and answers iOS simulator WebRTC offers there with `501 Not Implemented`
+and the same explanation.
 
 ## When encoding runs
 

packages/client/src/api/types.ts

@@ -188,12 +188,18 @@ export interface ChromeDevToolsTargetDiscovery {
 }
 
 /**
- * Whether the server build can encode the live H.264 browser stream. Only the
- * macOS build links the native encoder; Windows and Linux builds report
- * `supported: false` with a user-facing `reason`.
+ * What the server build can stream. Every shipped build encodes the live H.264
+ * browser stream (`native` VideoToolbox/x264 on macOS, `openh264` elsewhere),
+ * but only macOS can drive iOS simulators; Windows and Linux builds report
+ * `iosSimulator: false` with a user-facing `iosSimulatorReason`. A server that
+ * cannot stream at all would report `supported: false` with a `reason`.
  */
 export interface LiveVideoCapability {
   supported: boolean;
+  encoder?: string;
+  iosSimulator?: boolean;
+  androidEmulator?: boolean;
+  iosSimulatorReason?: string | null;
   requires?: string;
   reason?: string | null;
 }

packages/client/src/app/AppShell.tsx

@@ -628,9 +628,9 @@ export function AppShell({
   );
   const [streamConfigApplyKey, setStreamConfigApplyKey] = useState(0);
   const [streamConfigReady, setStreamConfigReady] = useState(false);
-  // Non-empty when the connected server build cannot encode live video (for
-  // example the Windows or Linux CLI). The stream stays paused and the reason
-  // is shown instead of retrying WebRTC offers that can never succeed.
+  // Non-empty when the connected server reports that it cannot encode live
+  // video at all. The stream stays paused and the reason is shown instead of
+  // retrying WebRTC offers that can never succeed.
   const [liveVideoUnavailable, setLiveVideoUnavailable] = useState("");
   const [touchIndicators, setTouchIndicators] = useState<TouchIndicator[]>([]);
   const [selectedSimulatorState, setSelectedSimulatorState] =