Cannot run a model container - macOS, error loading shared library

[planetf1]

macOS silicon, gpu enabled podman configuration:

➜  / ramalama run hf://ibm-granite/granite-3.1-8b-instruct
Fetching 15 files:   0%|                                                                              | 0/15 [00:00<?, ?it/s]Downloading 'merges.txt' to '/Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/.cache/huggingface/download/merges.txt.f8479fb696fe07332c55300a6accf8cc191acc6a.incomplete'
Downloading '.gitattributes' to '/Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/.cache/huggingface/download/.gitattributes.a6344aac8c09253b3b630fb776ae94478aa0275b.incomplete'
Downloading 'model-00002-of-00004.safetensors' to '/Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/.cache/huggingface/download/model-00002-of-00004.safetensors.c7c38b0d5a436775b09d764465ed6e6eb7a8c4e302d05e301e151c96e3076f22.incomplete'
Downloading 'added_tokens.json' to '/Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/.cache/huggingface/download/added_tokens.json.183eb810668f005a1ed0e0c8be060e5f47c23f2f.incomplete'
.gitattributes: 100%|███████████████████████████████████████████████████████████████████| 1.52k/1.52k [00:00<00:00, 18.1MB/s]
Download complete. Moving file to /Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/.gitattributes
Fetching 15 files:   7%|████▋                                                                 | 1/15 [00:00<00:02,  4.75it/s]Downloading 'generation_config.json' to '/Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/.cache/huggingface/download/generation_config.json.0e7ca8f1a4cf587858c703b962bf0730b90ce699.incomplete'
                                                                                                                            Downloading 'model-00001-of-00004.safetensors' to '/Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/.cache/huggingface/download/model-00001-of-00004.safetensors.191c4e9c6263d9cf591104f2d16ab2c39dcc43c1ad0680cc5a34d5c86d61ee41.incomplete'
Downloading 'README.md' to '/Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/.cache/huggingface/download/README.md.163f9f4701789ca407a0990df209d409395f1c9f.incomplete'
Downloading 'config.json' to '/Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/.cache/huggingface/download/config.json.9e06dd77173ab8d7abcb7c5d8104c9979a09158f.incomplete'
added_tokens.json: 100%|██████████████████████████████████████████████████████████████████| 87.0/87.0 [00:00<00:00, 1.55MB/s]
Download complete. Moving file to /Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/added_tokens.jsonn:   0%|                                                                          | 0.00/87.0 [00:00<?, ?B/s]
Downloading 'model-00003-of-00004.safetensors' to '/Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/.cache/huggingface/download/model-00003-of-00004.safetensors.f02784b72391fa04e9b986313c1a1720ce88f0eb40f7ae81fa0daadc93049457.incomplete'
generation_config.json: 100%|███████████████████████████████████████████████████████████████| 132/132 [00:00<00:00, 2.03MB/s]
Download complete. Moving file to /Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/generation_config.jsonn:   0%|                                                                      | 0.00/132 [00:00<?, ?B/s]
README.md: 100%|████████████████████████████████████████████████████████████████████████| 6.78k/6.78k [00:00<00:00, 50.3MB/s]
Download complete. Moving file to /Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/README.mdd:   0%|                                                                                 | 0.00/6.78k [00:00<?, ?B/s]
config.json: 100%|██████████████████████████████████████████████████████████████████████████| 790/790 [00:00<00:00, 11.0MB/s]
Download complete. Moving file to /Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/config.jsonn:   0%|                                                                                 | 0.00/790 [00:00<?, ?B/s]
                                                                                                                            Downloading 'model-00004-of-00004.safetensors' to '/Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/.cache/huggingface/download/model-00004-of-00004.safetensors.9d86d201ff8e73d8a46e92b543c9dd44f133e60b35ccada4a76439af62f22212.incomplete'
merges.txt: 100%|█████████████████████████████████████████████████████████████████████████| 442k/442k [00:00<00:00, 2.45MB/s]
Download complete. Moving file to /Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/merges.txt
Downloading 'special_tokens_map.json' to '/Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/.cache/huggingface/download/special_tokens_map.json.386500a5040da66c6db3d8b9c44ccd1ee202c744.incomplete'
Downloading 'tokenizer.json' to '/Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/.cache/huggingface/download/tokenizer.json.11f3f2c66235d30bc9bf04ad7e1b4a19d80e84da.incomplete'
special_tokens_map.json: 100%|██████████████████████████████████████████████████████████████| 701/701 [00:00<00:00, 5.35MB/s]
Download complete. Moving file to /Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/special_tokens_map.json
                                                                                                                            Downloading 'model.safetensors.index.json' to '/Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/.cache/huggingface/download/model.safetensors.index.json.6e06d9ba85709934f6a8cad738d36225c8044890.incomplete'
                                                                                                                            Downloading 'vocab.json' to '/Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/.cache/huggingface/download/vocab.json.0a11f2016e660fd490f7bf168e6d1f9c86a8f744.incomplete'    | 10.5M/4.99G [00:00<02:11, 37.8MB/s]
                                                                                                                            Downloading 'tokenizer_config.json' to '/Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/.cache/huggingface/download/tokenizer_config.json.3e1e4c1b08817b77f3010e7aab1b87ad645e6a42.incomplete'97G [00:00<?, ?B/s]
model.safetensors.index.json: 100%|█████████████████████████████████████████████████████| 29.8k/29.8k [00:00<00:00, 21.8MB/s]
Download complete. Moving file to /Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/model.safetensors.index.json
tokenizer_config.json: 100%|████████████████████████████████████████████████████████████| 8.07k/8.07k [00:00<00:00, 52.5MB/s]
Download complete. Moving file to /Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/tokenizer_config.json
vocab.json: 100%|█████████████████████████████████████████████████████████████████████████| 777k/777k [00:00<00:00, 2.08MB/s]
Download complete. Moving file to /Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/vocab.json02-of-00004.safetensors:   1%|▎                                                | 31.5M/4.99G [00:00<02:04, 39.9MB/s]
tokenizer.json: 100%|███████████████████████████████████████████████████████████████████| 3.48M/3.48M [00:00<00:00, 4.44MB/s]
Download complete. Moving file to /Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/tokenizer.json
model-00004-of-00004.safetensors: 100%|█████████████████████████████████████████████████| 1.41G/1.41G [01:10<00:00, 19.9MB/s]
Download complete. Moving file to /Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/model-00004-of-00004.safetensorss:  58%|████████████████████████████▌                    | 2.92G/4.99G [01:10<00:48, 42.4MB/s]
model-00002-of-00004.safetensors: 100%|█████████████████████████████████████████████████| 4.99G/4.99G [01:59<00:00, 41.6MB/s]
Download complete. Moving file to /Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/model-00002-of-00004.safetensorss: 100%|████████████████████████████████████████████████▉| 4.99G/4.99G [01:59<00:00, 43.3MB/s]
model-00003-of-00004.safetensors: 100%|█████████████████████████████████████████████████| 4.97G/4.97G [02:28<00:00, 33.4MB/s]
Download complete. Moving file to /Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/model-00003-of-00004.safetensorss:  75%|████████████████████████████████████▉            | 3.74G/4.97G [01:59<00:31, 38.9MB/s]
model-00001-of-00004.safetensors: 100%|█████████████████████████████████████████████████| 4.97G/4.97G [02:55<00:00, 28.3MB/s]
Download complete. Moving file to /Users/jonesn/.local/share/ramalama/repos/huggingface/ibm-granite/granite-3.1-8b-instruct/model-00001-of-00004.safetensorss: 100%|████████████████████████████████████████████████▉| 4.97G/4.97G [02:28<00:00, 42.3MB/s]
Fetching 15 files: 100%|█████████████████████████████████████████████████████████████████████| 15/15 [02:56<00:00, 11.75s/it]
Trying to pull quay.io/ramalama/ramalama:latest...
Getting image source signatures
Copying blob sha256:9f7f00a94e2fe3a87926c552ad095252d86788384fbec313b56cfeff044f9745
Copying blob sha256:d1d82f01ce6444423dc8ed8648a4857b3ec3b02a6ff08d1fcc224527a2007a06
Copying blob sha256:4a2f52b996b9594f76ab5bf3be9fa41cf4aa51dce9f0d0efc4310c1fd0d8a3f6
Copying config sha256:479ffa0d5cd6e8dda1cb10df11fea11848256e33826b8a8bf4b40b42d0e482fa
Writing manifest to image destination
Error: Command '['podman', 'run', '--rm', '-i', '--label', 'RAMALAMA', '--security-opt=label=disable', '--name', 'ramalama_kNj3O36riP', '-t', '--device', '/dev/dri', '--mount=type=bind,src=/Users/jonesn/.local/share/ramalama/models/huggingface/ibm-granite/granite-3.1-8b-instruct,destination=/mnt/models/model.file,rw=false', 'quay.io/ramalama/ramalama:latest', '/bin/sh', '-c', "llama-run -c 2048 --temp 0.8 /mnt/models/model.file '' 2> /dev/null"]' returned non-zero exit status 127.

Running manually (note - stderr is redirected above, which masks any error - problem?) :

➜  / podman run --rm -i --label RAMALAMA --security-opt=label=disable --name ramalama_kNj3036riP -t --device /dev/dri --mount=type=bind,src=/Users/jonesn/.local/share/ramalama/models/huggingface/ibm-granite/granite-3.1-8b-instruct,destination=/mnt/models/model.file,rw=false quay.io/ramalama/ramalama:latest /bin/sh -c llama-run -c 2048 --temp 0.8 /mnt/models/model.file
llama-run: error while loading shared libraries: libllama.so: cannot open shared object file: No such file or directory
➜  /

[ericcurtin]

This could be a bug, @rhatdan this looks like it's coming from the freshly build container:

https://quay.io/repository/ramalama/ramalama

[planetf1]

To debug - see below. These libraries are present, but not being found by default:

➜  / podman run -it quay.io/ramalama/ramalama:latest /bin/sh
sh-5.1# llama-run
llama-run: error while loading shared libraries: libllama.so: cannot open shared object file: No such file or directory
sh-5.1# find / -name 'libllama.so'
/usr/local/lib64/libllama.so
sh-5.1# uname -a
Linux 15908f4c8b0f 6.11.3-200.fc40.aarch64 #1 SMP PREEMPT_DYNAMIC Thu Oct 10 22:53:48 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
sh-5.1# export LD_LIBRARY_PATH=/usr/local/lib64:${LD_LIBRARY_PATH}
sh-5.1# llama-run
llama-run: error while loading shared libraries: libggml-cpu.so: cannot open shared object file: No such file or directory
sh-5.1# find / -name 'libggml-cpu.so'
/usr/local/lib/libggml-cpu.so
sh-5.1# export LD_LIBRARY_PATH=/usr/local/lib:${LD_LIBRARY_PATH}
sh-5.1# llama-run
Error: No arguments provided.
Description:
  Runs a llm

Usage:
  llama-run [options] model [prompt]

The executable has this for the loader:

sh-5.1# type llama-run
llama-run is hashed (/usr/local/bin/llama-run)
sh-5.1# ldd /usr/local/bin/llama-run
	linux-vdso.so.1 (0x0000ffff95bb5000)
	libllama.so => /usr/local/lib64/libllama.so (0x0000ffff95a1d000)
	libggml.so => /usr/local/lib64/libggml.so (0x0000ffff959fc000)
	libggml-cpu.so => /usr/local/lib/libggml-cpu.so (0x0000ffff95947000)
	libggml-base.so => /usr/local/lib64/libggml-base.so (0x0000ffff95886000)
	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x0000ffff95600000)
	libm.so.6 => /lib64/libm.so.6 (0x0000ffff9555f000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000ffff95855000)
	libc.so.6 => /lib64/libc.so.6 (0x0000ffff953b1000)
	libgomp.so.1 => /lib64/libgomp.so.1 (0x0000ffff95360000)
	/lib/ld-linux-aarch64.so.1 (0x0000ffff95b70000)

[planetf1]

Should we convert to issue @ericcurtin ?

[ericcurtin]

I don't mind

[ericcurtin]

This might be a quick temporary fix:

#547

won't be able to look at this today heading offline shortly

[planetf1]

Moved to issue: #548

[rhatdan]

Should be fixed by #549 but we will need new images.