Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
aa23526
[26.10] [DOC] FlashInfer: documentation updates for release
anisha-amd Oct 20, 2026
2378b18
Update flashinfer-install.rst
anisha-amd Oct 20, 2026
dacbb65
Update flashinfer-install.rst
anisha-amd Oct 20, 2026
996fa77
Potential fix for pull request finding 'Remove the incorrect prefill-…
anisha-amd Oct 20, 2026
6312ef9
Potential fix for pull request finding 'Exempt the development image …
anisha-amd Oct 20, 2026
7c30a53
Potential fix for pull request finding 'Use the supported AITER wheel…
anisha-amd Oct 20, 2026
5ba4cbb
Potential fix for pull request finding 'Remove unsupported AITER vari…
anisha-amd Oct 20, 2026
6295c39
Potential fix for pull request finding 'Remove the unsupported CK Til…
anisha-amd Oct 20, 2026
2c8d10e
Potential fix for pull request finding 'Remove unsupported AITER asse…
anisha-amd Oct 20, 2026
fccd778
Potential fix for pull request finding 'Scope backend override instru…
anisha-amd Oct 20, 2026
3640f4f
Potential fix for pull request finding 'Remove unsupported FP8 KV cac…
anisha-amd Oct 20, 2026
7ac1c2a
Potential fix for pull request finding 'Qualify GELU AITER backend su…
anisha-amd Oct 20, 2026
edbe25f
Revert incorrect edits
anisha-amd Oct 20, 2026
163baf0
Fix trailing whitespace and ruff format in amd-docs
demandal25 Oct 20, 2026
c051ec2
Merge pull request #396 from AMD-Ecosystem/release-0.6.18-amd.1-catch…
demandal25 Oct 21, 2026
71239bb
Merge branch 'release/0.6.18+amd.1' into docs/26.10
demandal25 Oct 21, 2026
f6173bb
Update amd-docs/what-is-flashinfer.rst
anisha-amd Oct 21, 2026
fa8d4a5
Address review on the AITER sections of the release docs
demandal25 Oct 21, 2026
bb27faa
Fix three factual errors the release docs carried over
demandal25 Oct 21, 2026
7ea42d7
Update flashinfer-install.rst
anisha-amd Oct 21, 2026
901ee05
Merge branch 'release/0.6.18+amd.2' into docs/26.10
demandal25 Oct 26, 2026
293531a
Update flashinfer-install.rst
anisha-amd Oct 27, 2026
e4bda67
Update flashinfer-install.rst
anisha-amd Oct 27, 2026
c75f70d
review edits
anisha-amd Oct 29, 2026
e693793
Correct the amd-docs kernel support matrix against the README
demandal25 Oct 31, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions amd-docs/.readthedocs.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details

version: 2

sphinx:
configuration: amd-docs/conf.py

# RTD by default builds html only
# Additional formats available for extra build time: htmlzip, pdf, epub
formats: []

python:
install:
- requirements: amd-docs/sphinx/requirements.txt

# Defines build environment
build:
os: ubuntu-24.04
tools:
python: "3.12"
12 changes: 12 additions & 0 deletions amd-docs/about/license.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
.. meta::
:description: FlashInfer license
:keywords: FlashInfer, ROCm, developer, reference, deep learning

.. _license:

******************************************
License
******************************************

.. include:: ../../LICENSE
:literal:
64 changes: 64 additions & 0 deletions amd-docs/conf.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html

"""
html_theme is usually unchanged (rocm_docs_theme).
flavor defines the site header display, select the flavor for the corresponding portals
flavor options: rocm, rocm-docs-home, rocm-blogs, rocm-ds, instinct, ai-developer-hub, local, generic
"""

html_theme = "rocm_docs_theme"
html_theme_options = {
"flavor": "rocm-llmext",
"repository_url": "https://github.com/AMD-Ecosystem/flashinfer",
"repository_branch": "docs/26.10",
"path_to_docs": "amd-docs",
"use_repository_button": True,
"use_issues_button": True,
"use_source_button": True,
"use_download_button": True,
}
# Publish the llms.txt index at the docs site root and let
# rocm-docs-core generate llms-full.txt after each build (the llms.txt standard,
# https://llmstxt.org/). See the rocm-docs-core guide:
# https://rocm.docs.amd.com/projects/rocm-docs-core/en/latest/user_guide/llms.html
rocm_docs_generate_llms = True

"""
docs_header_version is used to manually configure the version in the header. If
there exists a non-null value mapped to docs_header_version, then the header in
the documentation page will contain the given version string.
"""
html_context = {"docs_header_version": "26.10"}


# This section turns on/off article info
setting_all_article_info = True
all_article_info_os = ["linux"]
all_article_info_author = ""

# Component version shown in the documentation title.
version_number = "0.6.18"

# for PDF output on Read the Docs
project = "FlashInfer"
author = "Advanced Micro Devices, Inc."
copyright = "Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved."
version = version_number
release = version_number

# Defines Table of Content structure definition path
external_toc_path = "./sphinx/_toc.yml"

# Add more addtional package accordingly
extensions = [
"rocm_docs",
"sphinx.ext.autodoc", # for Python docstrings
]

html_title = f"{project} {version_number} documentation"

external_projects_current_project = "FlashInfer"
40 changes: 40 additions & 0 deletions amd-docs/examples/flashinfer-examples.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
.. meta::
:description: FlashInfer examples
:keywords: FlashInfer, programming, ROCm, example, sample, tutorial

.. _run-a-flashinfer-example:

********************************************************************
Run a FlashInfer example
********************************************************************

AITER supports single and batch prefill, along with other operations listed in the `kernel support matrix <https://rocm.docs.amd.com/projects/flashinfer/en/docs-26.10/what-is-flashinfer.html#kernel-support-matrix>`__.
The ``auto`` backend calls the most optimized kernels from AITER or the in-tree HIP kernels, based on the benchmarking results.
To select AITER explicitly for an operation that exposes a ``backend`` argument, pass ``backend="aiter"``.

1. Save the following code snippet to a Python script named ``flashinfer_example.py``.

.. code-block:: python

import torch
import flashinfer

# Configuration
seq_len = 1024 # Prompt length
num_qo_heads = 32 # Number of query/output heads
num_kv_heads = 8 # Number of KV heads (GQA with 4:1 ratio)
head_dim = 128

# Create Q, K, V tensors (NHD layout: sequence, heads, dimension)
q = torch.randn(seq_len, num_qo_heads, head_dim, dtype=torch.float16, device="cuda")
k = torch.randn(seq_len, num_kv_heads, head_dim, dtype=torch.float16, device="cuda")
v = torch.randn(seq_len, num_kv_heads, head_dim, dtype=torch.float16, device="cuda")

# Run single prefill attention with causal masking
output = flashinfer.single_prefill_with_kv_cache(q, k, v, causal=True, backend="aiter")

2. Run the script to use FlashInfer.

.. code-block:: bash

python flashinfer_example.py
52 changes: 52 additions & 0 deletions amd-docs/index.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
.. meta::
:description: FlashInfer documentation
:keywords: FlashInfer, ROCm, documentation, deep learning, framework, GPU

.. _flashinfer-documentation-index:

********************************************************************
FlashInfer on ROCm documentation
********************************************************************

Accelerate LLM attention and decoding kernels with FlashInfer on ROCm for
AMD Instinct GPUs, enabling high-throughput batch and streaming generation
for real-time applications like multilingual chat and code completion.

`FlashInfer <https://docs.flashinfer.ai/index.html>`__ is a library and kernel generator
for Large Language Models (LLMs) that provides a high-performance implementation of
kernels for graphic processing units (GPUs). FlashInfer focuses on LLM serving and inference,
as well as advanced performance across diverse scenarios.

FlashInfer on ROCm includes capabilities such as load balancing,
sparse and dense attention optimizations, and single and batch decode, alongside
prefill for high‑performance execution on AMD Instinct MI300X, MI325X, MI350X, and MI355X GPUs.
FlashInfer features highly efficient attention kernels, load-balanced scheduling,
and memory-optimized techniques, while supporting customized attention variants.
It’s compatible with ``torch.compile``, and offers high-performance LLM-specific
operators, with easy integration through PyTorch, and C++ APIs.

FlashInfer is part of the `AMD LLM Extension toolkit
<https://rocm.docs.amd.com/projects/llmextension/en/docs-26.10/>`__.

The FlashInfer public repository is located at `https://github.com/AMD-Ecosystem/flashinfer <https://github.com/AMD-Ecosystem/flashinfer>`__.

.. grid:: 2
:gutter: 3

.. grid-item-card:: Install

* :doc:`Install FlashInfer <install/flashinfer-install>`

.. grid-item-card:: Examples

* :doc:`Run a FlashInfer example <examples/flashinfer-examples>`

.. grid-item-card:: Reference

* :doc:`Environment variables <reference/environment-variables>`
* `API reference (upstream) <https://docs.flashinfer.ai/>`__

To contribute to the documentation, refer to
`Contributing to FlashInfer <https://github.com/AMD-Ecosystem/flashinfer/blob/amd-integration/CONTRIBUTING.md>`__.

You can find licensing information on the :doc:`Licensing <about/license>` page.
97 changes: 97 additions & 0 deletions amd-docs/install/flashinfer-install.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
.. meta::
:description: installing FlashInfer for ROCm
:keywords: installation instructions, Docker, AMD, ROCm, FlashInfer

.. _flashinfer-on-rocm-installation:

********************************************************************
FlashInfer on ROCm installation
********************************************************************

System requirements
====================================================================

To use FlashInfer `0.6.18 <https://github.com/AMD-Ecosystem/flashinfer/tree/release/0.6.18%2Bamd.2>`__, you need the following prerequisites:

- **ROCm version:** `10.1.0 <https://rocm.docs.amd.com/en/docs-10.1.0/>`__
- **Operating system:** Ubuntu 24.04
- **GPU platform:** AMD Instinct™ MI300X, MI325X, MI350X, MI355X
- **PyTorch:** `2.12.0 <https://github.com/ROCm/pytorch/tree/release/2.12>`__
- **Python:** `3.12 <https://www.python.org/downloads/release/python-31215/>`__
- **AITER:** `0.1.21.post2 <https://github.com/ROCm/aiter/releases/tag/v0.1.21.post2>`__ (source build)
Comment thread
anisha-amd marked this conversation as resolved.
Resolved

Install FlashInfer
====================================================================

FlashInfer on ROCm can be run directly by setting up a Docker container from scratch.

Build FlashInfer from source
--------------------------------------------------------------------

1. Clone the `https://github.com/AMD-Ecosystem/flashinfer <https://github.com/AMD-Ecosystem/flashinfer>`__ repository.

.. code-block:: bash

git clone https://github.com/AMD-Ecosystem/flashinfer.git

2. Enter the directory and build the Dockerfile (`https://github.com/AMD-Ecosystem/flashinfer/blob/amd-integration/docker/Dockerfile.rocm <https://github.com/AMD-Ecosystem/flashinfer/blob/amd-integration/docker/Dockerfile.rocm>`__) to create a Docker image.

.. code-block:: bash

cd flashinfer
docker build \
-t rocm-flashinfer-dev \
-f docker/Dockerfile.rocm . \
--build-arg USERNAME=$USER \
--build-arg USER_UID=$(id -u) \
--build-arg USER_GID=$(id -g)

3. Start a Docker container using the image.

.. code-block:: bash

docker run -it --rm \
--privileged --network=host --device=/dev/kfd \
--device=/dev/dri --group-add video \
--group-add "$(getent group render | cut -d: -f3)" \
--cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
--shm-size 16G \
-v $PWD:/workspace \
rocm-flashinfer-dev

Pass ``render`` as the host's numeric GID, as shown. The image runs as a
non-root user, so ``--privileged`` on its own does not grant access, and
passing the group by name picks up the image's own ``render`` group, whose
GID does not match the device.

4. Once you are inside the container, you can now install FlashInfer inside it.

.. code-block:: bash

cd /workspace
python -m pip install --no-build-isolation -ve .

Test the FlashInfer installation
======================================================================================

Verify that FlashInfer is installed correctly.

.. code-block:: bash

python -c "import flashinfer; print(flashinfer.__version__)"

Expected output:

.. code-block:: text

0.6.18+amd.2

If you see the version string above, FlashInfer ``0.6.18`` has been installed successfully. You can now use FlashInfer in your projects.

AITER Support
======================================================================================

FlashInfer on ROCm supports `https://github.com/ROCm/aiter <https://github.com/ROCm/aiter>`__ as a backend. AITER is a library for efficient attention operations.

The development image built above already includes the validated AITER build; no separate AITER installation is needed.
Loading
Loading