Runtime exception when using OpenVinoExecution provider on FP16 onnx model

[hlevring]

Describe the issue

Python 3.11
openvino==2024.4.0
onnxruntime-openvino==1.20.0

I am getting an error like this on Openvino GPU device (issue only in FP16, FP32 works)

onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running SkipSimplifiedLayerNormalization node. Name:'/model/layers.0/post_attention_layernorm/SkipLayerNorm' Status Message: C:\Users\Administrator\Desktop\windows_perf\release_validation_1.20.0\final_build\onnxruntime\include\onnxruntime\core/framework/op_kernel_context.h:42 onnxruntime::OpKernelContext::Input Missing Input: model.layers.0.input_layernorm.weight

I just found this ticket with the exactly same error
microsoft#22704

Supposedly the issue should have been fixed in Microsoft repository with 1.20.1, but no new release from Intel.

To reproduce

I created a short script to test the models below, and find the same results and error
TestFP16.py.zip

(BREAKS) https://huggingface.co/onnx-community/tiny-random-LlamaForCausalLM-ONNX/resolve/main/onnx/model_fp16.onnx

(WORKS) https://huggingface.co/onnx-community/tiny-random-LlamaForCausalLM-ONNX/resolve/main/onnx/model.onnx

Urgency

It seems like a pretty urgent and critical issue.

Platform

Windows

OS Version

Windows 11 24H2

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

Execution Provider for ONNXRuntime 5.5 c4fb724

ONNX Runtime API

Python

Architecture

X64

Execution Provider

OpenVINO

Execution Provider Library Version

No response

[sfatimar]

Hello I do not think 1.20.1 will fix OpenVINO Issue. As SkipLayerNorm is getting added only in 1.20.1 for NPU. I will check the model in GPU for 1.20.1 with OpenVINO and see if it works.

[hlevring]

Thanks it would be great if I can get this working for both OpenVINO GPU and NPU. The example model is not the model I am actually targeting, but I got identical error so it was an easy example.

[hlevring]

@sfatimar Just wanted to follow up on this issue.

[hlevring]

I just noticed that Microsoft released version 1.21 a few days ago. So I guess it's just good to know if Intel will also have version 1.21 out in the next few days and if that fixes the issue. Otherwise, I need to get our own team to start looking closer into this, but would rather avoid that since we are super busy with other stuff.

[n1harika]

Hi, the models that you have provided contain certain unsupported operators:
1.SimplifiedLayerNormalisation
2. RotaryEmbedding
3. MultiHeadAttention
These won't be enabled even with the 1.21 release.

The Intel OVEP release 1.21 will enable support for the 'SkipSimplifiedLayerNormalization' operator that you have mentioned, on both GPU and NPU (the Python packages have not yet been released but you can build from source from the GitHub repo as per your requirements).

Furthermore, I ran your script, there seems to be no issue- I am able to get the logits output. Please upgrade the OV version to 2025.0, and use the latest OVEP.