Repository navigation
[26.10] [DOC] FlashInfer: documentation updates for release #389
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
anisha-amd
wants to merge
25
commits into
release/0.6.18+amd.2
Choose a base branch
from
docs/26.10
base: release/0.6.18+amd.2
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
25 commits
Select commit
Hold shift + click to select a range
aa23526
[26.10] [DOC] FlashInfer: documentation updates for release
anisha-amd 2378b18
Update flashinfer-install.rst
anisha-amd dacbb65
Update flashinfer-install.rst
anisha-amd 996fa77
Potential fix for pull request finding 'Remove the incorrect prefill-…
anisha-amd 6312ef9
Potential fix for pull request finding 'Exempt the development image …
anisha-amd 7c30a53
Potential fix for pull request finding 'Use the supported AITER wheel…
anisha-amd 5ba4cbb
Potential fix for pull request finding 'Remove unsupported AITER vari…
anisha-amd 6295c39
Potential fix for pull request finding 'Remove the unsupported CK Til…
anisha-amd 2c8d10e
Potential fix for pull request finding 'Remove unsupported AITER asse…
anisha-amd fccd778
Potential fix for pull request finding 'Scope backend override instru…
anisha-amd 3640f4f
Potential fix for pull request finding 'Remove unsupported FP8 KV cac…
anisha-amd 7ac1c2a
Potential fix for pull request finding 'Qualify GELU AITER backend su…
anisha-amd edbe25f
Revert incorrect edits
anisha-amd 163baf0
Fix trailing whitespace and ruff format in amd-docs
demandal25 c051ec2
Merge pull request #396 from AMD-Ecosystem/release-0.6.18-amd.1-catch…
demandal25 71239bb
Merge branch 'release/0.6.18+amd.1' into docs/26.10
demandal25 f6173bb
Update amd-docs/what-is-flashinfer.rst
anisha-amd fa8d4a5
Address review on the AITER sections of the release docs
demandal25 bb27faa
Fix three factual errors the release docs carried over
demandal25 7ea42d7
Update flashinfer-install.rst
anisha-amd 901ee05
Merge branch 'release/0.6.18+amd.2' into docs/26.10
demandal25 293531a
Update flashinfer-install.rst
anisha-amd e4bda67
Update flashinfer-install.rst
anisha-amd c75f70d
review edits
anisha-amd e693793
Correct the amd-docs kernel support matrix against the README
demandal25 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| # Read the Docs configuration file | ||
| # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details | ||
|
|
||
| version: 2 | ||
|
|
||
| sphinx: | ||
| configuration: amd-docs/conf.py | ||
|
|
||
| # RTD by default builds html only | ||
| # Additional formats available for extra build time: htmlzip, pdf, epub | ||
| formats: [] | ||
|
|
||
| python: | ||
| install: | ||
| - requirements: amd-docs/sphinx/requirements.txt | ||
|
|
||
| # Defines build environment | ||
| build: | ||
| os: ubuntu-24.04 | ||
| tools: | ||
| python: "3.12" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| .. meta:: | ||
| :description: FlashInfer license | ||
| :keywords: FlashInfer, ROCm, developer, reference, deep learning | ||
|
|
||
| .. _license: | ||
|
|
||
| ****************************************** | ||
| License | ||
| ****************************************** | ||
|
|
||
| .. include:: ../../LICENSE | ||
| :literal: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| # Configuration file for the Sphinx documentation builder. | ||
| # | ||
| # This file only contains a selection of the most common options. For a full | ||
| # list see the documentation: | ||
| # https://www.sphinx-doc.org/en/master/usage/configuration.html | ||
|
|
||
| """ | ||
| html_theme is usually unchanged (rocm_docs_theme). | ||
| flavor defines the site header display, select the flavor for the corresponding portals | ||
| flavor options: rocm, rocm-docs-home, rocm-blogs, rocm-ds, instinct, ai-developer-hub, local, generic | ||
| """ | ||
|
|
||
| html_theme = "rocm_docs_theme" | ||
| html_theme_options = { | ||
| "flavor": "rocm-llmext", | ||
| "repository_url": "https://github.com/AMD-Ecosystem/flashinfer", | ||
| "repository_branch": "docs/26.10", | ||
| "path_to_docs": "amd-docs", | ||
| "use_repository_button": True, | ||
| "use_issues_button": True, | ||
| "use_source_button": True, | ||
| "use_download_button": True, | ||
| } | ||
| # Publish the llms.txt index at the docs site root and let | ||
| # rocm-docs-core generate llms-full.txt after each build (the llms.txt standard, | ||
| # https://llmstxt.org/). See the rocm-docs-core guide: | ||
| # https://rocm.docs.amd.com/projects/rocm-docs-core/en/latest/user_guide/llms.html | ||
| rocm_docs_generate_llms = True | ||
|
|
||
| """ | ||
| docs_header_version is used to manually configure the version in the header. If | ||
| there exists a non-null value mapped to docs_header_version, then the header in | ||
| the documentation page will contain the given version string. | ||
| """ | ||
| html_context = {"docs_header_version": "26.10"} | ||
|
|
||
|
|
||
| # This section turns on/off article info | ||
| setting_all_article_info = True | ||
| all_article_info_os = ["linux"] | ||
| all_article_info_author = "" | ||
|
|
||
| # Component version shown in the documentation title. | ||
| version_number = "0.6.18" | ||
|
|
||
| # for PDF output on Read the Docs | ||
| project = "FlashInfer" | ||
| author = "Advanced Micro Devices, Inc." | ||
| copyright = "Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved." | ||
| version = version_number | ||
| release = version_number | ||
|
|
||
| # Defines Table of Content structure definition path | ||
| external_toc_path = "./sphinx/_toc.yml" | ||
|
|
||
| # Add more addtional package accordingly | ||
| extensions = [ | ||
| "rocm_docs", | ||
| "sphinx.ext.autodoc", # for Python docstrings | ||
| ] | ||
|
|
||
| html_title = f"{project} {version_number} documentation" | ||
|
|
||
| external_projects_current_project = "FlashInfer" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| .. meta:: | ||
| :description: FlashInfer examples | ||
| :keywords: FlashInfer, programming, ROCm, example, sample, tutorial | ||
|
|
||
| .. _run-a-flashinfer-example: | ||
|
|
||
| ******************************************************************** | ||
| Run a FlashInfer example | ||
| ******************************************************************** | ||
|
|
||
| AITER supports single and batch prefill, along with other operations listed in the `kernel support matrix <https://rocm.docs.amd.com/projects/flashinfer/en/docs-26.10/what-is-flashinfer.html#kernel-support-matrix>`__. | ||
| The ``auto`` backend calls the most optimized kernels from AITER or the in-tree HIP kernels, based on the benchmarking results. | ||
| To select AITER explicitly for an operation that exposes a ``backend`` argument, pass ``backend="aiter"``. | ||
|
|
||
| 1. Save the following code snippet to a Python script named ``flashinfer_example.py``. | ||
|
|
||
| .. code-block:: python | ||
|
|
||
| import torch | ||
| import flashinfer | ||
|
|
||
| # Configuration | ||
| seq_len = 1024 # Prompt length | ||
| num_qo_heads = 32 # Number of query/output heads | ||
| num_kv_heads = 8 # Number of KV heads (GQA with 4:1 ratio) | ||
| head_dim = 128 | ||
|
|
||
| # Create Q, K, V tensors (NHD layout: sequence, heads, dimension) | ||
| q = torch.randn(seq_len, num_qo_heads, head_dim, dtype=torch.float16, device="cuda") | ||
| k = torch.randn(seq_len, num_kv_heads, head_dim, dtype=torch.float16, device="cuda") | ||
| v = torch.randn(seq_len, num_kv_heads, head_dim, dtype=torch.float16, device="cuda") | ||
|
|
||
| # Run single prefill attention with causal masking | ||
| output = flashinfer.single_prefill_with_kv_cache(q, k, v, causal=True, backend="aiter") | ||
|
|
||
| 2. Run the script to use FlashInfer. | ||
|
|
||
| .. code-block:: bash | ||
|
|
||
| python flashinfer_example.py |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| .. meta:: | ||
| :description: FlashInfer documentation | ||
| :keywords: FlashInfer, ROCm, documentation, deep learning, framework, GPU | ||
|
|
||
| .. _flashinfer-documentation-index: | ||
|
|
||
| ******************************************************************** | ||
| FlashInfer on ROCm documentation | ||
| ******************************************************************** | ||
|
|
||
| Accelerate LLM attention and decoding kernels with FlashInfer on ROCm for | ||
| AMD Instinct GPUs, enabling high-throughput batch and streaming generation | ||
| for real-time applications like multilingual chat and code completion. | ||
|
|
||
| `FlashInfer <https://docs.flashinfer.ai/index.html>`__ is a library and kernel generator | ||
| for Large Language Models (LLMs) that provides a high-performance implementation of | ||
| kernels for graphic processing units (GPUs). FlashInfer focuses on LLM serving and inference, | ||
| as well as advanced performance across diverse scenarios. | ||
|
|
||
| FlashInfer on ROCm includes capabilities such as load balancing, | ||
| sparse and dense attention optimizations, and single and batch decode, alongside | ||
| prefill for high‑performance execution on AMD Instinct MI300X, MI325X, MI350X, and MI355X GPUs. | ||
| FlashInfer features highly efficient attention kernels, load-balanced scheduling, | ||
| and memory-optimized techniques, while supporting customized attention variants. | ||
| It’s compatible with ``torch.compile``, and offers high-performance LLM-specific | ||
| operators, with easy integration through PyTorch, and C++ APIs. | ||
|
|
||
| FlashInfer is part of the `AMD LLM Extension toolkit | ||
| <https://rocm.docs.amd.com/projects/llmextension/en/docs-26.10/>`__. | ||
|
|
||
| The FlashInfer public repository is located at `https://github.com/AMD-Ecosystem/flashinfer <https://github.com/AMD-Ecosystem/flashinfer>`__. | ||
|
|
||
| .. grid:: 2 | ||
| :gutter: 3 | ||
|
|
||
| .. grid-item-card:: Install | ||
|
|
||
| * :doc:`Install FlashInfer <install/flashinfer-install>` | ||
|
|
||
| .. grid-item-card:: Examples | ||
|
|
||
| * :doc:`Run a FlashInfer example <examples/flashinfer-examples>` | ||
|
|
||
| .. grid-item-card:: Reference | ||
|
|
||
| * :doc:`Environment variables <reference/environment-variables>` | ||
| * `API reference (upstream) <https://docs.flashinfer.ai/>`__ | ||
|
|
||
| To contribute to the documentation, refer to | ||
| `Contributing to FlashInfer <https://github.com/AMD-Ecosystem/flashinfer/blob/amd-integration/CONTRIBUTING.md>`__. | ||
|
|
||
| You can find licensing information on the :doc:`Licensing <about/license>` page. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| .. meta:: | ||
| :description: installing FlashInfer for ROCm | ||
| :keywords: installation instructions, Docker, AMD, ROCm, FlashInfer | ||
|
|
||
| .. _flashinfer-on-rocm-installation: | ||
|
|
||
| ******************************************************************** | ||
| FlashInfer on ROCm installation | ||
| ******************************************************************** | ||
|
|
||
| System requirements | ||
| ==================================================================== | ||
|
|
||
| To use FlashInfer `0.6.18 <https://github.com/AMD-Ecosystem/flashinfer/tree/release/0.6.18%2Bamd.2>`__, you need the following prerequisites: | ||
|
|
||
| - **ROCm version:** `10.1.0 <https://rocm.docs.amd.com/en/docs-10.1.0/>`__ | ||
| - **Operating system:** Ubuntu 24.04 | ||
| - **GPU platform:** AMD Instinct™ MI300X, MI325X, MI350X, MI355X | ||
| - **PyTorch:** `2.12.0 <https://github.com/ROCm/pytorch/tree/release/2.12>`__ | ||
| - **Python:** `3.12 <https://www.python.org/downloads/release/python-31215/>`__ | ||
| - **AITER:** `0.1.21.post2 <https://github.com/ROCm/aiter/releases/tag/v0.1.21.post2>`__ (source build) | ||
|
|
||
| Install FlashInfer | ||
| ==================================================================== | ||
|
|
||
| FlashInfer on ROCm can be run directly by setting up a Docker container from scratch. | ||
|
|
||
| Build FlashInfer from source | ||
| -------------------------------------------------------------------- | ||
|
|
||
| 1. Clone the `https://github.com/AMD-Ecosystem/flashinfer <https://github.com/AMD-Ecosystem/flashinfer>`__ repository. | ||
|
|
||
| .. code-block:: bash | ||
|
|
||
| git clone https://github.com/AMD-Ecosystem/flashinfer.git | ||
|
|
||
| 2. Enter the directory and build the Dockerfile (`https://github.com/AMD-Ecosystem/flashinfer/blob/amd-integration/docker/Dockerfile.rocm <https://github.com/AMD-Ecosystem/flashinfer/blob/amd-integration/docker/Dockerfile.rocm>`__) to create a Docker image. | ||
|
|
||
| .. code-block:: bash | ||
|
|
||
| cd flashinfer | ||
| docker build \ | ||
| -t rocm-flashinfer-dev \ | ||
| -f docker/Dockerfile.rocm . \ | ||
| --build-arg USERNAME=$USER \ | ||
| --build-arg USER_UID=$(id -u) \ | ||
| --build-arg USER_GID=$(id -g) | ||
|
|
||
| 3. Start a Docker container using the image. | ||
|
|
||
| .. code-block:: bash | ||
|
|
||
| docker run -it --rm \ | ||
| --privileged --network=host --device=/dev/kfd \ | ||
| --device=/dev/dri --group-add video \ | ||
| --group-add "$(getent group render | cut -d: -f3)" \ | ||
| --cap-add=SYS_PTRACE \ | ||
| --security-opt seccomp=unconfined \ | ||
| --shm-size 16G \ | ||
| -v $PWD:/workspace \ | ||
| rocm-flashinfer-dev | ||
|
|
||
| Pass ``render`` as the host's numeric GID, as shown. The image runs as a | ||
| non-root user, so ``--privileged`` on its own does not grant access, and | ||
| passing the group by name picks up the image's own ``render`` group, whose | ||
| GID does not match the device. | ||
|
|
||
| 4. Once you are inside the container, you can now install FlashInfer inside it. | ||
|
|
||
| .. code-block:: bash | ||
|
|
||
| cd /workspace | ||
| python -m pip install --no-build-isolation -ve . | ||
|
|
||
| Test the FlashInfer installation | ||
| ====================================================================================== | ||
|
|
||
| Verify that FlashInfer is installed correctly. | ||
|
|
||
| .. code-block:: bash | ||
|
|
||
| python -c "import flashinfer; print(flashinfer.__version__)" | ||
|
|
||
| Expected output: | ||
|
|
||
| .. code-block:: text | ||
|
|
||
| 0.6.18+amd.2 | ||
|
|
||
| If you see the version string above, FlashInfer ``0.6.18`` has been installed successfully. You can now use FlashInfer in your projects. | ||
|
|
||
| AITER Support | ||
| ====================================================================================== | ||
|
|
||
| FlashInfer on ROCm supports `https://github.com/ROCm/aiter <https://github.com/ROCm/aiter>`__ as a backend. AITER is a library for efficient attention operations. | ||
|
|
||
| The development image built above already includes the validated AITER build; no separate AITER installation is needed. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.