Skip to content

Run inference on ExecuTorch - #308

Open
shoumikhin wants to merge 2 commits into
amazon-far:mainfrom
shoumikhin:executorch-backend
Open

shoumikhin wants to merge 2 commits into
amazon-far:mainfrom
shoumikhin:executorch-backend

Conversation

@shoumikhin

Copy link
Copy Markdown

This adds ExecuTorch, PyTorch's solution for deploying models on devices, as one more backend behind the ModelBackend seam that #277 added to OnnxModel.

Why

  • The .pte path is built on torch.export, with no ONNX conversion step.
  • A .pte carries the tensor names, gains and joint names. The TensorRT backend reads them from the .onnx.
  • One export function gives a CPU program (XNNPACK) or a GPU program, where TensorRT runs what it supports and ExecuTorch's CUDA backend runs the rest.
  • The TensorRT engine is built at export time, not at first load. It still runs only on one GPU model and TensorRT version.

Changes

  • export_policy_as_pte(wrapper, pte_file_path, example_obs_dict, metadata, backend) next to export_policy_as_onnx, with the same actor_obs input and action output. Metadata goes into a signature method of the program; CUDA backend weights go to a .ptd beside it.
  • runtime.backend: executorch, with ExecuTorch's Module, on one thread. It applies to every model in the process, so a preset's secondary policy must be exported too; an .onnx is refused with a clear message.
  • HOLOSOMA_WITH_EXECUTORCH=AUTO|ON|OFF, like TensorRT, using the CMake package in the pip wheel. Without it the build is unchanged. ExecuTorch's TensorRT delegate is not linked when the TensorRT backend is on, because two TensorRT libraries in one process clash.

ExecuTorch needs numpy 2 and torch 2.15, while holosoma pins numpy below 2, so the export runs in a separate environment (the README has the steps). CI installs neither, so the new tests do not run there.

Tested

Linux x86_64 only, NVIDIA H100 and A100, ExecuTorch 1.6, Torch-TensorRT 2.15:

  • ctest passes with ExecuTorch on and off. With the TensorRT backend also on, building a new engine needs a tar-installed TensorRT on LD_LIBRARY_PATH.
  • The shipped ppo_g1_29dof.onnx weights, exported both ways from PyTorch: over 200 random observations, actions match ONNX Runtime within 1.5e-5 (XNNPACK) and 2.1e-5 (TensorRT, same as the Run inference on TensorRT #277 backend).
  • Through Policy with g1-29dof-loco and that PPO model, for 200 ticks, joint targets match within 6.1e-7 rad. The preset's default FastSAC model was not exported.
  • A policy forced to split between TensorRT and the CUDA backend matches within 3e-7.
  • Median run(), busy shared host: ONNX Runtime 25 to 34 us, XNNPACK 18 to 23 us, TensorRT backend 42 to 51 us, ExecuTorch TensorRT 51 to 60 us.

Not tested: aarch64, Jetson, macOS, the robot runner.

Closes #307

Add export_policy_as_pte next to export_policy_as_onnx. It exports the
policy with torch.export and writes an ExecuTorch .pte file, either for
the CPU (XNNPACK) or for an NVIDIA GPU, where TensorRT runs what it
supports and ExecuTorch's CUDA backend runs the rest. When the CUDA
backend gets a part with weights, they go to a .ptd file next to the
.pte.

The .pte keeps what the ONNX export keeps: the actor_obs input and
action output names, and the metadata serialized the same way as
attach_onnx_metadata. A .pte cannot be edited after export, so the
metadata is passed in and returned by a constant "signature" method.

The ExecuTorch imports stay inside the function, so nothing changes for
users who do not call it, and mypy.ini treats them like the other
optional packages. The test skips when ExecuTorch is not installed.
Add an ExecuTorch backend behind the ModelBackend seam, selected with
runtime.backend: executorch. It runs a .pte file written by
export_policy_as_pte through ExecuTorch's Module API. Tensor shapes come
from the program, and tensor names and metadata come from its
"signature" method, so gains and joint names still reach the policy
without an .onnx beside it. A .ptd data file next to the .pte (written
for ExecuTorch's CUDA backend) is loaded with it. Inference runs on the
calling thread: ExecuTorch's thread pool spans every core, and on these
small policies it only adds wake-up delay.

runtime.backend applies to every model in the process, so a model that
is not a .pte (such as a preset's secondary policy) is refused with a
message that says so.

HOLOSOMA_WITH_EXECUTORCH=AUTO|ON|OFF works like the TensorRT option.
The C++ package ships in the executorch pip wheel and is found with
find_package(executorch); nothing is downloaded. The XNNPACK and CUDA
delegates are linked, and the TensorRT delegate from
torch_tensorrt_executorch_runtime too, unless the TensorRT backend is
also enabled: two TensorRT libraries in one process clash. A build
without ExecuTorch is unchanged, and naming the backend there throws.
The Python package build keeps ExecuTorch off unless asked for.

The tests export one small policy as .onnx and as .pte with the same
weights, then check that names, shapes, metadata and outputs match ONNX
Runtime, and that a .pte without a signature or an .onnx file is
refused with the right message.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add an ExecuTorch backend for OnnxModel

1 participant