feat(examples): add modal mock inference example#64
Open
pham-tuan-binh wants to merge 9 commits into
Open
Conversation
Runs a Portal operator on Modal while the robot runs locally, and measures glass-to-glass latency by round-tripping a timestamp through the real H264 video path. - robot.py paints a (seq, capture_us) QR code into each frame, sends it plus a matching state sample, and reports latency, codec lag, and dropped frames on its own clock. - policy.py is a mock LeRobot-shaped policy that decodes the QR and echoes the token back. It also carries the Modal app, so it deploys with `modal run policy.py`. - portal.yaml holds the shared wire contract, loaded by both sides. - README.md is a step-by-step Modal setup and run guide. The policy is CPU-only; a real model drops in by adding a GPU and swapping the Policy class.
The policy runs on Modal, where the secret sets LIVEKIT_* as env vars, so loading a .env there was a no-op. Drop load_dotenv (and python-dotenv from the Modal image) from policy.py and read the environment directly. robot.py still loads .env since it runs locally.
- robot.py: drain in-flight replies before the final tally so tail frames are not counted as dropped. - policy.py: guard the codec-lag lookup with 'is not None' instead of a truthy check. - policy.py: move the Modal deploy block below main() so the file reads top to bottom: helpers, Policy, main, deploy.
Tested end to end against a local livekit-server and hit PortalError.NoPeer: the policy connected and immediately called set_active_operator, which RPCs the robot, before the robot was visible in the room. - policy.py: wait until robot_identity() is set before claiming control. - robot.py: wait until active_operator() is set before streaming, so frames sent before a policy is controlling are not counted as dropped. With both waits, 450 frames sent -> 438 round-trips, 12 dropped (6 QR-decode misses plus 6 frames lost in the video pipe), 99% decode rate.
A ~500ms glass-to-glass with a ~50ms codec_lag showed the latency was added after the frame arrived: the policy decoded every observation inline, so when decode could not keep up a standing queue built and every reply carried a stale capture time. That defeats the point of Portal. Adopt the inference example's pattern: keep only the freshest observations in a bounded deque and always decode the newest, dropping the backlog. Latency now tracks the network round trip. Codec lag is measured at arrival so the number stays meaningful.
…latency glass2glass much larger than rtt means the media pipeline (encode, jitter buffer, geography of the extra cloud hop) is the cost. glass2glass close to rtt means it is just the network round trip. This tells the two apart.
The background drifted a shade every frame, so every pixel changed every frame and H264 could not inter-frame compress: each frame was effectively a keyframe. On a real link that balloons the bitrate, inflates codec_lag, and congests the transport (latency spikes, burst drops). Make the background a constant so only the QR region changes, which is also how a real camera feed behaves.
This is an illustrative example, not a robust data channel. Revert the static background (it starved the QR under H264's rate control and broke decoding) and add a README note: QR over H264 is lossy so some frames drop, and latency is mostly the distance between robot and policy. Both are called out with the fix (byte-stream codec for pixel-exact data, co-locate for low latency).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A new example under
examples/python/modal-mock-inference/that runs a Portal policy on Modal while the robot runs locally, and measures glass-to-glass latency.The trick: the robot paints a
(seq, capture_us)token into each frame as a QR code, so the timestamp travels through the real H264 encoder and jitter buffer. The cloud policy reads the QR back out and echoes the token, and the robot closes the loop on its own clock. Because the timestamp rode the video pixels, the number is true glass-to-glass, not a metadata ping.The policy is a mock (it decodes a QR instead of running a model), so the example stays focused on the deployment pattern rather than weights or GPUs.
Layout
robot.pypolicy.pymodal run policy.py.portal.yamlREADME.mdpyproject.toml,.env.exampleWhat it measures
metrics.policy.e2e_us_p50/p95(fed the QR capture time throughin_reply_to_ts_us).seq.Extending to a real policy
CPU-only as shipped. A real model drops in by adding
gpu="A10G"to the Modal function and swapping thePolicyclass, which already has the LeRobotselect_actioncall shape. The operator loop does not change.Testing
Validated locally without a live room: both modules import, both configs load from
portal.yaml, a frame fromrobot.make_framedecodes back throughPolicy.get_action, an unreadable frame returnsNone, the drop and codec-lag arithmetic hold, and the Modalconfig_path()fallback resolves. The live LiveKit connection and the real H264 pipe are exercised by following the README (needs a LiveKit Cloud project and a Modal account).