unable to quantize model based on Qwen/Qwen2.5-3B-Instruct

[maiiabocharova]

Can you please help: I see in the Qwen repo that they have a quantized model, but when I try to do it myself - I get an error. Please help!

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_SAVE_PATH = "Qwen/Qwen2.5-3B-Instruct"

quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }


# Load model
model = AutoAWQForCausalLM.from_pretrained(MODEL_SAVE_PATH)
tokenizer = AutoTokenizer.from_pretrained(MODEL_SAVE_PATH)

model.quantize(tokenizer, quant_config=quant_config)

Got an error: AttributeError: 'Qwen2SdpaAttention' object has no attribute 'apply_qkv'