x

vLLM Hardware Plugin for Intel® Gaudi®

| Documentation | Intel® Gaudi® Documentation | Optimizing Training Platform Guide |

---

Latest News 🔥

• [2025/11] The 0.10.2 release introduces the production-ready version of the vLLM Hardware Plugin for Intel® Gaudi® v1.23.0. The plugin is an alternative to the vLLM fork, which reaches end of life with this release and will be deprecated in v1.24.0, remaining functional only for legacy use cases. We strongly encourage all fork users to begin planning their migration to the plugin. For more information about this release, see the Release Notes.
• [2025/06] We introduced an early developer preview of the vLLM Hardware Plugin for Intel® Gaudi®, which is not yet intended for general use.

---

About

The vLLM Hardware Plugin for Intel® Gaudi® integrates Intel® Gaudi® AI accelerators with vLLM to optimize large language model inference. It follows the [RFC]: Hardware pluggable and [RFC]: Enhancing vLLM Plugin Architecture principles, providing a modular interface for Intel® Gaudi® hardware. For more information, see the Plugin System document.

Getting Started

1. Set up your execution environment. Additionally, to achieve the best performance on HPU, follow the methods outlined in the Optimizing Training Platform Guide.

2. Get the last verified vLLM commit. While vLLM Hardware Plugin for Intel® Gaudi® follows the latest vLLM commits, upstream API updates may introduce compatibility issues. The saved commit has been thoroughly validated.

   git clone https://github.com/vllm-project/vllm-gaudi
   cd vllm-gaudi
   export VLLM_COMMIT_HASH=$(git show "origin/vllm/last-good-commit-for-vllm-gaudi:VLLM_STABLE_COMMIT" 2>/dev/null)
   cd ..

3. Install vLLM using pip or build it from source:

   # Build vLLM from source for empty platform, reusing existing torch installation
   git clone https://github.com/vllm-project/vllm
   cd vllm
   git checkout $VLLM_COMMIT_HASH
   pip install -r <(sed '/^torch/d' requirements/build.txt)
   VLLM_TARGET_DEVICE=empty pip install --no-build-isolation -e .
   cd ..

4. Install vLLM Hardware Plugin for Intel® Gaudi® from source:

   cd vllm-gaudi
   pip install -e .
   cd ..

   To see all the available installation methods, such as NIXL, see the Installation guide.

Contributing

We welcome and value any contributions and collaborations.

Contact Us

• For technical questions and feature requests, please use GitHub Issues.
• For discussing with fellow users, please use the vLLM Forum.
• For coordinating contributions and development, please use Slack.
• For security disclosures, please use GitHub's Security Advisories feature.