Repository navigation
Run inference on ExecuTorch - #308
Open
shoumikhin wants to merge 2 commits into
Open
shoumikhin wants to merge 2 commits into
shoumikhin wants to merge 2 commits into
Conversation
Add export_policy_as_pte next to export_policy_as_onnx. It exports the policy with torch.export and writes an ExecuTorch .pte file, either for the CPU (XNNPACK) or for an NVIDIA GPU, where TensorRT runs what it supports and ExecuTorch's CUDA backend runs the rest. When the CUDA backend gets a part with weights, they go to a .ptd file next to the .pte. The .pte keeps what the ONNX export keeps: the actor_obs input and action output names, and the metadata serialized the same way as attach_onnx_metadata. A .pte cannot be edited after export, so the metadata is passed in and returned by a constant "signature" method. The ExecuTorch imports stay inside the function, so nothing changes for users who do not call it, and mypy.ini treats them like the other optional packages. The test skips when ExecuTorch is not installed.
Add an ExecuTorch backend behind the ModelBackend seam, selected with runtime.backend: executorch. It runs a .pte file written by export_policy_as_pte through ExecuTorch's Module API. Tensor shapes come from the program, and tensor names and metadata come from its "signature" method, so gains and joint names still reach the policy without an .onnx beside it. A .ptd data file next to the .pte (written for ExecuTorch's CUDA backend) is loaded with it. Inference runs on the calling thread: ExecuTorch's thread pool spans every core, and on these small policies it only adds wake-up delay. runtime.backend applies to every model in the process, so a model that is not a .pte (such as a preset's secondary policy) is refused with a message that says so. HOLOSOMA_WITH_EXECUTORCH=AUTO|ON|OFF works like the TensorRT option. The C++ package ships in the executorch pip wheel and is found with find_package(executorch); nothing is downloaded. The XNNPACK and CUDA delegates are linked, and the TensorRT delegate from torch_tensorrt_executorch_runtime too, unless the TensorRT backend is also enabled: two TensorRT libraries in one process clash. A build without ExecuTorch is unchanged, and naming the backend there throws. The Python package build keeps ExecuTorch off unless asked for. The tests export one small policy as .onnx and as .pte with the same weights, then check that names, shapes, metadata and outputs match ONNX Runtime, and that a .pte without a signature or an .onnx file is refused with the right message.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds ExecuTorch, PyTorch's solution for deploying models on devices, as one more backend behind the
ModelBackendseam that #277 added toOnnxModel.Why
.ptepath is built ontorch.export, with no ONNX conversion step..ptecarries the tensor names, gains and joint names. The TensorRT backend reads them from the.onnx.Changes
export_policy_as_pte(wrapper, pte_file_path, example_obs_dict, metadata, backend)next toexport_policy_as_onnx, with the sameactor_obsinput andactionoutput. Metadata goes into asignaturemethod of the program; CUDA backend weights go to a.ptdbeside it.runtime.backend: executorch, with ExecuTorch'sModule, on one thread. It applies to every model in the process, so a preset's secondary policy must be exported too; an.onnxis refused with a clear message.HOLOSOMA_WITH_EXECUTORCH=AUTO|ON|OFF, like TensorRT, using the CMake package in the pip wheel. Without it the build is unchanged. ExecuTorch's TensorRT delegate is not linked when the TensorRT backend is on, because two TensorRT libraries in one process clash.ExecuTorch needs numpy 2 and torch 2.15, while
holosomapins numpy below 2, so the export runs in a separate environment (the README has the steps). CI installs neither, so the new tests do not run there.Tested
Linux x86_64 only, NVIDIA H100 and A100, ExecuTorch 1.6, Torch-TensorRT 2.15:
LD_LIBRARY_PATH.ppo_g1_29dof.onnxweights, exported both ways from PyTorch: over 200 random observations, actions match ONNX Runtime within 1.5e-5 (XNNPACK) and 2.1e-5 (TensorRT, same as the Run inference on TensorRT #277 backend).Policywithg1-29dof-locoand that PPO model, for 200 ticks, joint targets match within 6.1e-7 rad. The preset's default FastSAC model was not exported.run(), busy shared host: ONNX Runtime 25 to 34 us, XNNPACK 18 to 23 us, TensorRT backend 42 to 51 us, ExecuTorch TensorRT 51 to 60 us.Not tested: aarch64, Jetson, macOS, the robot runner.
Closes #307