Is it in scope of the llama.cpp project to add a new "endianness changer" feature to llama-quantize cli tooling?

[aarmn]

Hello everyone!
I was considering adding an endian-ness changer which is currently a dangling python file to the llama.cpp project, llama-quantize module, do you think there is a scope problem with that? as I'm new to source-code, and not quite a top notch c/cpp developer, so I'd love to have a heads-up before investing some time on implementing that.

I think personally it's super useful to have that or even, auto convert with a prompt to the user, but I think a flag would be enough to improve workflow

Is there a similar feature implemented? and should I?

Thanks for your time!

[taronaeo]

I would be speaking the context for Big-Endian systems. Llama.cpp currently provides the following scripts that already implement byteswapping:

1. convert_hf_to_gguf.py (via --bigendian flag)
2. gguf_convert_endian.py

Both of these will produce Big-Endian GGUF files. However, gguf_convert_endian.py has limited support for byteswapping the GGML types, so more implementation is required.

Next, quantization. On s390x, we usually take the Big-Endian GGUF model created from above and run it through llama-quantize to create a quantized model. This ensures that the model is still in Big-Endian byteorder.

If you could somehow merge these steps where we can take a BF16 Little-Endian GGUF model, and quantize it to lets say Q4_K_M while byteswapping it to Big-Endian, that would be great :)

lmk your thoughts

[aarmn]

Hello again!

My initial plan was to get gguf_convert_endian.py and port it to C/++ as a utility of llama_quantize or llama_gguf

Also, unlike convert_hf_to_gguf.py, in default builds and installations (e.g.: archlinux) gguf_convert_endian.py doesn't even get shipped, I'm not sure if other distros are different. Also, the idea of having python files with the .py extension in path feels really eerie for some reason! And tools naming feels a bit arbitrary, as in, a user doesn't know what belongs to llama.cpp without explicit lookup using pkg manager.

My goal was to collect these python scripts and port them as subcommands of other llama.cpp utils so that there be less clutter, and suggest user on changing endianness (while I was trying to run a model with wrong endianness, I got a warning that it might be endianness, but I had to Google, download the py script, and run it manually, while I think it makes a lot more sense for it to be a simple flag or subcommands of one the existing llama.cpp utils)

Or at least unify them as a python script, under a more unified CLI, e.g.: llama_format?

Based on current project structure, is that a good idea? Or am I unaware of some critical details in my assumptions?

Thanks alot for your feedback! @taronaeo

[taronaeo]

My initial plan was to get gguf_convert_endian.py and port it to C/++ as a utility of llama_quantize or llama_gguf

Just a thought about code maintainability. gguf_convert_endian.py includes numpy to structure the GGUF file and safetensors to read the original model tensors. I'm not sure how maintainable this port would be in the long run, given that both numpy and safetensors are well-maintained. (Thinking of why reinvent the wheel?)

Also, unlike convert_hf_to_gguf.py, in default builds and installations (e.g.: archlinux) gguf_convert_endian.py doesn't even get shipped, I'm not sure if other distros are different.

Most users will never touch gguf_convert_endian.py tbh. It's a very niche case for Big-Endian systems because most, if not all models available online are by default Little-Endian. If by "default builds" you mean the GitHub releases we have, then yes, we intend to keep the releases as minimal as possible.

Btw, what hardware are you using? AIX / s390x?

Also, the idea of having python files with the .py extension in path feels really eerie for some reason!

A lot of major projects available online use multiple programming languages, so this shouldn't come as a surprise :)

Or at least unify them as a python script, under a more unified CLI, e.g.: llama_format?

I think this is a better direction than to port the existing convert_hf_to_gguf.py and gguf_convert_endian.py to C/C++, where its really difficult to maintain. Pretty sure this would be a welcome change.

Ping @CISC for opinion as well :)

[aarmn]

Just a thought about code maintainability. gguf_convert_endian.py includes numpy to structure the GGUF file and safetensors to read the original model tensors. I'm not sure how maintainable this port would be in the long run, given that both numpy and safetensors are well-maintained. (Thinking of why reinvent the wheel?)

Most users will never touch gguf_convert_endian.py tbh. It's a very niche case for Big-Endian systems because most, if not all models available online are by default Little-Endian. If by "default builds" you mean the GitHub releases we have, then yes, we intend to keep the releases as minimal as possible.

Great points! my concern is mostly DX, so I'm more than fine with python, in fact, I'm more comfortable with it!

Btw, what hardware are you using? AIX / s390x?

Nope, I'm using Linux + Intel 13650HX (x86_64) + RTX 4050, quite a conventional home setup. The model I downloaded from huggingface had a different endianness, so I had to flip endianness for it to work. The process felt a bit odd and unnecessary, as there is quite a descriptive error already in place, guessing exactly what should be the next course of action, yet the tooling is not shipped. and I had to install deps (make a venv) + download script off gitlab to get it to work, not a quagmire, but didn't feel right either.

A lot of major projects available online use multiple programming languages, so this shouldn't come as a surprise :)

Agreed, but it's way less common to not simply use a shebang instead of .py extension :)

I think this is a better direction than to port the existing convert_hf_to_gguf.py and gguf_convert_endian.py to C/C++, where its really difficult to maintain. Pretty sure this would be a welcome change.

Ping @CISC for opinion as well :)

Cool! Hello @CISC ! what's your take on it?

[CISC]

Apparently this is more a documentation issue than a tooling issue, all you had to do was pip install gguf and run gguf-convert-endian, no .py extension nor script from gitlab in sight. :)

[aarmn]

Oh, so it's not part of the llama.cpp, but rather gguf python package?
I dunno how these two projects are related (or not) to eachother, but I guess that means things are already pretty good, thanks @CISC