[Issue]: debug symbol crashes hipcc

[wkong-amd]

Problem Description

While compiling my code with the following argument (pre-proc code attached later):
hipcc -g -O3 -std=gnu++20 -Wno-unused-value -Wno-undefined-var-template -fno-offload-uniform-block -mllvm --lsr-drop-solution=1 -mllvm -enable-post-misched=0 -mllvm -amdgpu-coerce-illegal-types=1 -mllvm -amdgpu-early-inline-all=true -mllvm -amdgpu-function-calls=false -mllvm -greedy-reverse-local-assignment=1 winograd.cpp -o winograd.out -lutility

clang reported the error below, but the code compiles & runs with '-g' removed:

PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: /opt/rocm-7.0.0/lib/llvm/bin/clang-20 -cc1 -triple amdgcn-amd-amdhsa -aux-triple x86_64-unknown-linux-gnu -Werror=atomic-alignment -emit-obj -dumpdir winograd.out- -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name winograd.cpp -mrelocation-model pic -pic-level 2 -fhalf-no-semantic-interposition -mframe-pointer=none -fno-rounding-math -mconstructor-aliases -aux-target-cpu x86-64 -fcuda-is-device -fno-threadsafe-statics -mllvm -amdgpu-internalize-symbols -fcuda-allow-variadic-functions -fvisibility=hidden -fapply-global-visibility-to-externs -mlink-builtin-bitcode /opt/rocm-7.0.0/lib/llvm/lib/clang/20/lib/amdgcn/bitcode/hip.bc -mlink-builtin-bitcode /opt/rocm-7.0.0/lib/llvm/lib/clang/20/lib/amdgcn/bitcode/ocml.bc -mlink-builtin-bitcode /opt/rocm-7.0.0/lib/llvm/lib/clang/20/lib/amdgcn/bitcode/ockl.bc -mlink-builtin-bitcode /opt/rocm-7.0.0/lib/llvm/lib/clang/20/lib/amdgcn/bitcode/oclc_daz_opt_off.bc -mlink-builtin-bitcode /opt/rocm-7.0.0/lib/llvm/lib/clang/20/lib/amdgcn/bitcode/oclc_unsafe_math_off.bc -mlink-builtin-bitcode /opt/rocm-7.0.0/lib/llvm/lib/clang/20/lib/amdgcn/bitcode/oclc_finite_only_off.bc -mlink-builtin-bitcode /opt/rocm-7.0.0/lib/llvm/lib/clang/20/lib/amdgcn/bitcode/oclc_correctly_rounded_sqrt_on.bc -mlink-builtin-bitcode /opt/rocm-7.0.0/lib/llvm/lib/clang/20/lib/amdgcn/bitcode/oclc_wavefrontsize64_on.bc -mlink-builtin-bitcode /opt/rocm-7.0.0/lib/llvm/lib/clang/20/lib/amdgcn/bitcode/oclc_isa_version_942.bc -mlink-builtin-bitcode /opt/rocm-7.0.0/lib/llvm/lib/clang/20/lib/amdgcn/bitcode/oclc_abi_version_600.bc -target-cpu gfx942 -debug-info-kind=constructor -dwarf-version=5 -debugger-tuning=gdb -mllvm -amdgpu-spill-cfi-saved-regs -gheterogeneous-dwarf=diexpression -fdebug-compilation-dir=/workspace/repo/ck-oot-kernels/kernels/winograd -resource-dir /opt/rocm-7.0.0/lib/llvm/lib/clang/20 -internal-isystem /opt/rocm-7.0.0/lib/llvm/lib/clang/20/include/cuda_wrappers -idirafter /opt/rocm-7.0.0/include -include __clang_hip_runtime_wrapper.h -D __HIP_HCC_COMPAT_MODE__=1 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/x86_64-linux-gnu/c++/12 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/backward -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/x86_64-linux-gnu/c++/12 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/backward -internal-isystem /opt/rocm-7.0.0/lib/llvm/lib/clang/20/include -internal-isystem /usr/local/include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/12/../../../../x86_64-linux-gnu/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /opt/rocm-7.0.0/lib/llvm/lib/clang/20/include -internal-isystem /usr/local/include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/12/../../../../x86_64-linux-gnu/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -O3 -Wno-unused-value -Wno-undefined-var-template -Wno-format-nonliteral -std=gnu++20 -fdeprecated-macro -fno-autolink -ferror-limit 19 -fhip-new-launch-api -fgnuc-version=4.2.1 -fno-implicit-modules -fskip-odr-check-in-gmf -fcxx-exceptions -fexceptions -fcolor-diagnostics -vectorize-loops -vectorize-slp -fallow-half-arguments-and-returns -mllvm --lsr-drop-solution=1 -mllvm -enable-post-misched=0 -mllvm -amdgpu-coerce-illegal-types=1 -mllvm -amdgpu-early-inline-all=true -mllvm -amdgpu-function-calls=false -mllvm -greedy-reverse-local-assignment=1 -cuid=642a880f86b5edcd -fcuda-allow-variadic-functions -fno-offload-uniform-block -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /tmp/winograd-gfx942-5df728.o -x hip winograd.cpp
1.      <eof> parser at end of file
2.      Code generation
3.      Running pass 'CallGraph Pass Manager' on module 'winograd.cpp'.
4.      Running pass 'Machine Instruction Scheduler' on function '@_Z7k_entryI12WinogradConvEvNT_4ArgsE'
clang++: error: unable to execute command: Segmentation fault (core dumped)
clang++: error: clang frontend command failed due to signal (use -v to see invocation)
AMD clang version 20.0.0git (https://github.com/RadeonOpenCompute/llvm-project roc-7.0.0 25304 82aed4e69d70bef3c89c38a2ee85c8c41294dfc9)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/rocm-7.0.0/lib/llvm/bin
Configuration file: /opt/rocm-7.0.0/lib/llvm/bin/clang++.cfg
clang++: note: diagnostic msg: Error generating preprocessed source(s).
failed to execute:/opt/rocm-7.0.0/lib/llvm/bin/clang++  -Xclang -fallow-half-arguments-and-returns -D__HIP_HCC_COMPAT_MODE__=1 --offload-arch=gfx942 --offload-arch=gfx942 --offload-arch=gfx942 --offload-arch=gfx942 --offload-arch=gfx942 --offload-arch=gfx942 --offload-arch=gfx942 --offload-arch=gfx942 --driver-mode=g++ -Llib --hip-link  -g -O3 -std=gnu++20 -Wno-unused-value -Wno-undefined-var-template -fno-offload-uniform-block -mllvm --lsr-drop-solution=1 -mllvm -enable-post-misched=0 -mllvm -amdgpu-coerce-illegal-types=1 -mllvm -amdgpu-early-inline-all=true -mllvm -amdgpu-function-calls=false -mllvm -greedy-reverse-local-assignment=1 -x hip winograd.cpp -o "winograd.out" -lutility -O3 -Wno-format-nonliteral -parallel-jobs=4 -O3 -parallel-jobs=4
make: *** [Makefile:38: winograd] Error 1

To access the code that reproduces this issue, please find it below:

https://amdcloud-my.sharepoint.com/:f:/g/personal/wkong_amd_com/EhY7bKt2GQxHi4okwq_BPxUBqtj6e0wUF0SHIYhzMOI8kw?e=XVHmPz

I've observed similar errors while compiling ck after switching to rocm 7.0 containers, but never encountered them in rocm 6.4. Maybe this provide certain clue.

Operating System

Ubuntu 22.04.5 LTS (Jammy Jellyfish)

CPU

Intel(R) Xeon(R) Platinum 8480C

GPU

AMD Instinct MI308X

ROCm Version

Rocm 7.0.0

ROCm Component

No response

Steps to Reproduce

The command above should be able to reproduce the issue. My code only requires a few ck headers which are typically within the include path of hipcc (normally official containers have them already).

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

No response

[doru1004]

@wkong-amd I've requested access to the code. :)

Does this failure only occur on MI308?

[wkong-amd]

Thank you for checking :-). The error is also reproducible on MI355 machines.

I've approved the access. Please verify now.

[doru1004]

It worked thank you! :)

I'll try to get access to a 355 again. Is rocm 7.0 also a strict requirement?

[wkong-amd]

I start observing errors like this since rocm 7.0, sometimes they cannot be reproduced in rocm 6.4. Not sure if they are fixed in newer versions.

[doru1004]

Can you share which docker image you're using?

[doru1004]

After including the missing ck header, on gfx942 with rocm 7.0.0 I am getting this:
ld.lld: error: undefined symbol: main
Do you expect this? Indeed the file does not contain a main.

[wkong-amd]

I could see that the main function is in line 40 of winograd.cpp

On MI355, I'm using one of the official pytorch release from docker hub: rocm/7.0:rocm7.0_ubuntu22.04_py3.10_pytorch_release_2.8.0_rc1

[doru1004]

I could see that the main function is in line 40 of winograd.cpp

On MI355, I'm using one of the official pytorch release from docker hub: rocm/7.0:rocm7.0_ubuntu22.04_py3.10_pytorch_release_2.8.0_rc1

You are right. I somehow picked up the wrong file.

But I am actually able to reproduce the issue on gfx942 as well (locally, no docker).

[doru1004]

So it looks like "older" releases of rocm-7.0.0 which do not include this commit:

69734ee

will suffer from the error which you reported.

If you are working on a machine with rocm-7.0.0 I would suggest upgrading to a newer rocm release like 7.0.1 or 7.0.2 and then you will be good.

[wkong-amd]

Indeed the issue is gone after switching to newer releases. Thanks a lot for your support! Closing the issue.