Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .claude/skills/pr-workflow/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -266,7 +266,8 @@ Run this gate on the branch's full diff before `gh pr create`, in order:

| Where | Covers |
| :--- | :--- |
| `README.md` | Support matrix, env vars, quick start, supported versions |
| `README.md` | Support matrix, quick start, the four common env vars, supported versions |
| `docs/rocm/environment.md`, `docs/rocm/torch-compile.md` | Runtime env vars; `torch.compile` support and its known issue |
| `docs/rocm/backends.md` | Routing rules, per-op constraints, unavailable modules, AITER install |
| `CONTRIBUTING.md` | Build, tests, coverage, code structure, upstream-sync procedure |
| `CLAUDE.md`, `.claude/skills/` | Commands, gotchas and workflows an agent will act on |
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -441,7 +441,7 @@ with concrete examples.
# Build / JIT Gotchas

**Build-time environment variables.** These affect compilation only; the
runtime ones are tabulated in [README.md](README.md#environment-variables).
runtime ones are tabulated in [docs/rocm/environment.md](docs/rocm/environment.md).

| Variable | Purpose |
| :--- | :--- |
Expand Down
224 changes: 75 additions & 149 deletions README.md

Large diffs are not rendered by default.

51 changes: 51 additions & 0 deletions docs/rocm/environment.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Environment variables and runtime helpers

## Runtime variables

Read at runtime or import time:

| Variable | Default | Purpose |
| :--- | :--- | :--- |
| `FLASHINFER_USE_TORCH_CUSTOM_OPS` | `0` | Wrap kernels for `torch.compile`; set before importing `flashinfer`. See [`torch-compile.md`](torch-compile.md). |
| `FLASHINFER_AITER_STRICT` | `0` | `1` raises instead of degrading: when AITER cannot serve a page size natively, and when an AITER kernel bootstrap fails under `auto`. Set in CI to catch coverage regressions rather than absorb them as a slowdown. |
| `FLASHINFER_AITER_ASM_PREFILL` | `1` | Exactly `0` keeps single prefill on AITER's CK Tile kernel; any other value leaves the asm arm on. Read once per process. AITER aborts the process on several asm failure modes rather than returning an error, so this is the supported way out. |
| `FLASHINFER_AITER_ASM_VERBOSE` | `0` | Any value except `0` logs to stderr which prefill arm ran. The two arms agree numerically, so this is the only way to tell asm from CK Tile. |
| `FLASHINFER_ARCH_ALLOW_KNOWN_BAD` | `0` | `1` runs an (op, backend, arch) combination the capability table marks known-broken on your toolchain. No row carries such a window in this release, and it does not lift the gfx950 soft-cap gate. |
| `FLASHINFER_HIP_FUSED_CASCADE` | `0` | In `MultiLevelCascadeAttentionWrapper` only, pass each level's partial state into the next prefill call instead of merging afterwards; AITER levels and the shared-prefix wrappers ignore it. Both paths tested. Read once at import, so set it first. |
| `FLASHINFER_WORKSPACE_BASE` | `$HOME` | Parent of the JIT cache (`.cache/flashinfer/`); point at fast local disk when `$HOME` is on NFS. Prefer an absolute path: the value is wrapped in `pathlib.Path` unvalidated, so a relative one resolves against the working directory and `~` becomes a literal `./~` directory. |
| `FLASHINFER_DISABLE_JIT` | unset | **Any non-empty value** — including `0` — loads only AOT or jit-cache kernels: it refuses to compile, and refuses modules already in `~/.cache/flashinfer` too. The AITER shims are never prebuilt, so an `auto` call that picks AITER raises `MissingJITCacheError` rather than falling back. |
| `FLASHINFER_DISABLE_VERSION_CHECK` | unset | Any non-empty value skips the JIT-cache package version check. |
| `FLASHINFER_LOGGING_LEVEL` | `INFO` | Logger verbosity (`DEBUG`, `INFO`, `WARNING`, …). Affects AITER fallback warnings and JIT build messages. |
| `FLASHINFER_LOGLEVEL` | `0` | API call logging, distinct from `FLASHINFER_LOGGING_LEVEL`: `1` logs function names, `3` adds inputs and outputs, `5` adds tensor statistics, `10` also dumps tensors to disk. |
| `FLASHINFER_LOGDEST` | `stdout` | Where `FLASHINFER_LOGLEVEL` output goes: `stdout`, `stderr`, or a file path (`%i` becomes the process ID). |
| `FLASHINFER_TRACE_APPLY` | unset | `1` enables upstream's opt-in kernel substitution at import, from the mapping `FLASHINFER_TRACE_APPLY_PATH` names; any other value except `0` warns and leaves it off. A failure to enable it warns rather than breaking the import. |
| `FLASHINFER_DISABLE_AOT_ARCH_CHECK` | unset | Any non-empty value — including `0` — uses jit-cache kernels even when their architecture does not match the running GPU. By default a mismatch discards them, with a warning, and everything JIT-compiles. |
| `ROCM_PATH` / `ROCM_HOME` | `/opt/rocm` | Where `flashinfer.rocm.hip_utils` looks for ROCm. Override only for non-standard layouts. |
| `AITER_JIT_DIR` | AITER's own | Where the C++ shim `dlopen`s AITER's built `.so` files, overriding the path compiled in at build time. |
| `FLASHINFER_AITER_VARIANT_DIR` | FlashInfer's stores | `os.pathsep`-separated prebuilt AITER variant stores to load from instead of the cache and jit-cache stores; `AITER_JIT_DIR` is still tried first. |
| `AITER_ASM_DIR` | set by `import aiter` | Where AITER reads its asm code objects. Unset or empty, single prefill stays on CK Tile rather than let AITER abort. |
| `GPU_ARCHS` | autodetected | AITER's own JIT architecture. A shim build overrides it from `FLASHINFER_ROCM_ARCH_LIST` and restores your value afterwards, leaving the derived one only if you had not set it. |

Build-time variables — `FLASHINFER_ROCM_ARCH_LIST`, `PYTORCH_ROCM_ARCH`,
`FLASHINFER_JIT_VERBOSE`, `FLASHINFER_EXTRA_{LDFLAGS,CFLAGS,CUDAFLAGS}`,
`FLASHINFER_OWN_HEADERS_NON_SYSTEM`, `MAX_JOBS` — live in
[CONTRIBUTING.md](../../CONTRIBUTING.md#build--jit-gotchas),
as does `FLASHINFER_JIT_DEBUG`, the debug-build switch.

## Runtime helpers

```python
import torch

from flashinfer.rocm.aiter_utils import is_aiter_supported
from flashinfer.rocm.hip_utils import check_torch_rocm_compatibility

# True on gfx942/gfx950 with a ROCm torch build. Does *not* verify the
# `aiter` package imports — wrap in try/except ImportError if you need that.
if is_aiter_supported(torch.device("cuda")):
...

# Warns when torch's ROCm version differs from the system's. A CPU-only torch
# never gets here: importing flashinfer already raises.
check_torch_rocm_compatibility()
```
41 changes: 41 additions & 0 deletions docs/rocm/torch-compile.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# `torch.compile` on ROCm

Set `FLASHINFER_USE_TORCH_CUSTOM_OPS=1` **before** importing `flashinfer` to
register the kernels as `torch.library` custom ops so TorchDynamo can trace
them (PyTorch ≥ 2.4). Each eager call pays a host-side dispatch cost, measured
at 21 µs and 11 µs per single prefill on our MI300X and MI355X hosts. Measured
in bf16 on gfx942 and gfx950 (identical) on the fa2 and AITER backends, each
compiled result checked against eager:

| Op | Flag off | Flag on | Flag on, `fullgraph=True` |
|---|---|---|---|
| Paged and ragged prefill, batch decode `run()` | fails | compiles | compiles¹ |
| Single prefill, `aiter_fused_moe` | fails | compiles | fails |
| MLA | compiles | compiles | fails |
| `append_paged_kv_cache`, `append_paged_mla_kv_cache` | fails | compiles | compiles² |

¹ Except AITER decode with `return_lse=True`, which breaks the graph to build
its fa2 LSE plan eagerly; use `backend="fa2"` there.
² Except `append_paged_kv_cache(backend="aiter")`, which breaks the graph.

Ops outside the table were not measured beyond the known issue below; sampling
ops that take a `generator`, POD, `jit_args` custom variants, the
fp8-quantizing norms and AITER's opt-in norm, RoPE and activation backends are
not registered. Call `plan()` outside the compiled region: its ops are not
registered either. With the flag off the error is often torch's `RuntimeError
when making fake tensor call` rather than a message naming the flag.

**Known issue:** under every `dynamic` setting, PyTorch 2.12 can miscompile an
op that writes into a view of a larger tensor, silently or with a GPU fault.
That covers an `out=` slice, an in-place norm on part of a buffer, K and V
caches carved from one allocation (a layer of a multi-layer pool, the halves of
a `[2, pages, ...]` tensor), and the single `[pages, 2, ...]` cache
`append_paged_kv_cache` accepts, which it splits into K and V views itself.
Allocate every tensor an op writes as its own buffer up front, rather than
slicing it from a larger one, and pass append two separately allocated caches;
a `.clone()` per call is not a fix, since the write never reaches the original.
In-place RoPE on `qkv.split` views and MLA's `torch.split(buf, [512, 64],
dim=-1)` cache layout measured exact, so keep those. Under
`mode="reduce-overhead"`, a KV cache passed as an argument is copied into the
CUDA graph on every replay (measured with AITER decode); register it as a
module buffer (`persistent=False`) instead.
Loading
Loading