[Issue]: AITER all reduce kernel segmentation fault in MI300x

[graceleeis]

Problem Description

I am trying to run DeepSeek-V3.2 on an MI300X machine via SGLang, using the NSA + AITER (tilelang / flashMLA) pipeline on ROCm. That pipeline depends on AITER’s ROCm kernels, including its custom all-reduce, layernorm, and the FlashMLA module (flash_mla). On this machine those pieces are either missing or broken.

Here is what I am seeing.

AITER custom all-reduce segfaults

As soon as workers start running the NSA + AITER path, the process segfaults in aiter/aiter/jit/core.py.

This happens even when SGLang is started with --disable-custom-all-reduce, so it looks like AITER’s own ROCm all-reduce kernel is still being used somewhere in the pipeline.

The Python side never sees a PyTorch RuntimeError. It is a hard segmentation fault.

Operating System

Ubuntu 22.04.5 LTS

CPU

Intel(R) Xeon(R) Platinum 8480C

GPU

8* AMD MI300X

ROCm Version

6.10.5

ROCm Component

No response

Steps to Reproduce

docker image: lmsysorg/sglang:dsv32-rocm
command to run: python -m sglang.launch_server --model-path /models/DeepSeek-V3.2/ --tp 8 --disable-cuda-graph --disable-custom-all-reduce

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

No response

[graceleeis]

Add sglang related discussion for reference sgl-project/sglang#11060

[JCasahuga]

Same issue

Hardware:

• 8x AMD Instinct MI300X (gfx942)
• ROCm 6.16.6

Setup:

• Docker image: rocm/sgl-dev:v0.5.6.post2-rocm700-mi30x-20251214
• Model: DeepSeek-V3.2-Speciale
• Command: python -m sglang.launch_server --model-path /work/DeepSeek-V3.2-Speciale --tp 8 --disable-cuda-graph --disable-radix-cache

Observed behavior:

Without tilelang installed:

• Model loads successfully (163 safetensors shards)
• Server starts and becomes healthy on port 8001
• AITER modules load: module_rmsnorm, module_quant, module_rope_pos_fwd
• Segfault on first inference request:
  ModuleNotFoundError: No module named 'tilelang'
  [TP3] Scheduler hit an exception: ... segmentation fault in AITER all-reduce kernel
  SIGQUIT received. signum=None, frame=None. It usually means one child failed.

After pip install tilelang:

• Installation appears successful initially
• Runtime failure when tilelang tries to initialize:
  RuntimeError: ninja exited with status 1
  fatal error: cuda_runtime_api.h: No such file or directory
  3 | #include <cuda_runtime_api.h>
  compilation terminated.
• Then crashes with:
  OSError: libnvrtc.so.12: cannot open shared object file: No such file or directory
• Root cause: tilelang requires CUDA (NVIDIA) libraries which don't exist on ROCm systems
• Still results in same SIGQUIT/segfault

Notes:

• Issue occurs even with --disable-custom-all-reduce flag
• tilelang is CUDA-only with no ROCm support

Would appreciate any updates on ROCm support for AITER with SGLang. Happy to provide additional logs or test fixes.