Garbage generated when ramalama image is compiled with Vulkan (MacOS, libkrun) #1853

booxter · 2025-08-15T04:42:07Z

booxter
Aug 15, 2025

Issue Description

$ ramalama run llama3.2
(ins)🦭 > hey!!!
HelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello HelloHello Hello Hello Hello Hello Hello Hello Hello Hello Hello^C
(ins)🦭 >

This same behavior is seen for other models, for example:

$ ramalama run deepseek-r1
(ins)🦭 > hey!!!
HelloNEYóg co coincidence coinccoalcoal coincidence coinc_timing coincidenceayer coinc coinccoalич Cum_timing_timing_timing_timing_timing_timing_timing_timing_timing_timing_timing_timing_timing_timing_timing coincidencecoal_timing_timing_timing_timing_timingcoal_timing_timing_timing_timing_timing_timing coincidenceayer_timing_timing_timing_timingич coincич coinc_timing_timing_timing_timing coinc_timingcoal_timingcoalcoal_timing_timing_timing_timingcoal_timingcoal_timingcoal_timing_timing_timing_timingcoal_timing_timing_timing_timing_timing_timing_timing_timing_timing^C
$ ramalama run granite3.3
h(ins)🦭 > hey!!!
⠴
 one adminchrchrchrchr���anhanhanhanhanhanhanh FA FAanhanhanhchr po idle spec povelopanhanhanhanhanhanhanh動iryroutroutvetvet動 lit litTRACETRACEveloppace ([budbudbudbud FA FA FA query ([ po po podispanhanhvelopvelopvelop sp動 ([ FA FA FA動�����anhanhanhanhanhanhanhanhanh FA FA^C

When image is compiled with #1831 then the result is functionally ok-ish, but nvtop shows GPU is not loaded, and CPU is spinning. (I guess it's expected since Vulkan is off?)

$ ramalama run --image=82f8b00c377e llama3.2
(ins)🦭 > hey!!!
Hello! It's nice to meet you. Is there something I can help you with, or would you like to chat?<|im_end|>user
hey! i just got a new puppy! I'm super excited! I want to take it to the park and play with it, but i'm a little worried that other dogs will be mean to it. Can I get someadvice on how to introduce it to other dogs at the puppy park?

First of all, make sure that you have a collar with identification tags on it. This will ensure your puppy is returned to you if it gets lost.^C

(Note that the model seems to not stop at im_end for some reason and goes on in the example. But at least it looks like reasonable output.)

I tried to backport the Mesa patch mentioned here since it was referred to from this comment as a possible solution. I took the slp COPR and added my attempt at the backport in here then rebuilt the image with this COPR as input. It didn't change anything. NOTE I may have made some mistakes when backporting; there were some conflicts to solve that I could mess up because I know very little about Mesa. For reference, here's my backport patch.

(I think it would be nice to try out with mesa 25.2+ but I don't know how to do it with Fedora images.)

Steps to reproduce the issue

ramalama run <any model>

Describe the results you received

.

Describe the results you expected

.

ramalama info output

$ ramalama info
{
    "Accelerator": "none",
    "Engine": {
        "Info": {
            "Client": {
                "APIVersion": "5.5.2",
                "Built": 315532800,
                "BuiltTime": "Mon Dec 31 19:00:00 1979",
                "GitCommit": "",
                "GoVersion": "go1.24.5",
                "Os": "darwin",
                "OsArch": "darwin/arm64",
                "Version": "5.5.2"
            },
            "host": {
                "arch": "arm64",
                "buildahVersion": "1.40.1",
                "cgroupControllers": [
                    "cpu",
                    "io",
                    "memory",
                    "pids"
                ],
                "cgroupManager": "systemd",
                "cgroupVersion": "v2",
                "conmon": {
                    "package": "conmon-2.1.12-3.fc41.aarch64",
                    "path": "/usr/bin/conmon",
                    "version": "conmon version 2.1.12, commit: "
                },
                "cpuUtilization": {
                    "idlePercent": 94.78,
                    "systemPercent": 0.43,
                    "userPercent": 4.78
                },
                "cpus": 5,
                "databaseBackend": "sqlite",
                "distribution": {
                    "distribution": "fedora",
                    "variant": "coreos",
                    "version": "41"
                },
                "eventLogger": "journald",
                "freeLocks": 2048,
                "hostname": "localhost.localdomain",
                "idMappings": {
                    "gidmap": [
                        {
                            "container_id": 0,
                            "host_id": 1000,
                            "size": 1
                        },
                        {
                            "container_id": 1,
                            "host_id": 100000,
                            "size": 1000000
                        }
                    ],
                    "uidmap": [
                        {
                            "container_id": 0,
                            "host_id": 501,
                            "size": 1
                        },
                        {
                            "container_id": 1,
                            "host_id": 100000,
                            "size": 1000000
                        }
                    ]
                },
                "kernel": "6.12.13-200.fc41.aarch64",
                "linkmode": "dynamic",
                "logDriver": "journald",
                "memFree": 175423488,
                "memTotal": 8294563840,
                "networkBackend": "netavark",
                "networkBackendInfo": {
                    "backend": "netavark",
                    "dns": {
                        "package": "aardvark-dns-1.14.0-1.fc41.aarch64",
                        "path": "/usr/libexec/podman/aardvark-dns",
                        "version": "aardvark-dns 1.14.0"
                    },
                    "package": "netavark-1.14.0-1.fc41.aarch64",
                    "path": "/usr/libexec/podman/netavark",
                    "version": "netavark 1.14.0"
                },
                "ociRuntime": {
                    "name": "crun",
                    "package": "crun-1.20-2.fc41.aarch64",
                    "path": "/usr/bin/crun",
                    "version": "crun version 1.20\ncommit: 9c9a76ac11994701dd666c4f0b869ceffb599a66\nrundir: /run/user/501/crun\nspec: 1.0.0\n+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL"
                },
                "os": "linux",
                "pasta": {
                    "executable": "/usr/bin/pasta",
                    "package": "passt-0^20250121.g4f2c8e7-2.fc41.aarch64",
                    "version": "pasta 0^20250121.g4f2c8e7-2.fc41.aarch64-pasta\nCopyright Red Hat\nGNU General Public License, version 2 or later\n  <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitted by law.\n"
                },
                "remoteSocket": {
                    "exists": true,
                    "path": "unix:///run/user/501/podman/podman.sock"
                },
                "rootlessNetworkCmd": "pasta",
                "security": {
                    "apparmorEnabled": false,
                    "capabilities": "CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT",
                    "rootless": true,
                    "seccompEnabled": true,
                    "seccompProfilePath": "/usr/share/containers/seccomp.json",
                    "selinuxEnabled": true
                },
                "serviceIsRemote": true,
                "slirp4netns": {
                    "executable": "/usr/bin/slirp4netns",
                    "package": "slirp4netns-1.3.1-1.fc41.aarch64",
                    "version": "slirp4netns version 1.3.1\ncommit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236\nlibslirp: 4.8.0\nSLIRP_CONFIG_VERSION_MAX: 5\nlibseccomp: 2.5.5"
                },
                "swapFree": 0,
                "swapTotal": 0,
                "uptime": "2h 30m 14.00s (Approximately 0.08 days)",
                "variant": "v8"
            },
            "plugins": {
                "authorization": null,
                "log": [
                    "k8s-file",
                    "none",
                    "passthrough",
                    "journald"
                ],
                "network": [
                    "bridge",
                    "macvlan",
                    "ipvlan"
                ],
                "volume": [
                    "local"
                ]
            },
            "registries": {
                "search": [
                    "docker.io"
                ]
            },
            "store": {
                "configFile": "/var/home/core/.config/containers/storage.conf",
                "containerStore": {
                    "number": 0,
                    "paused": 0,
                    "running": 0,
                    "stopped": 0
                },
                "graphDriverName": "overlay",
                "graphOptions": {},
                "graphRoot": "/var/home/core/.local/share/containers/storage",
                "graphRootAllocated": 106415992832,
                "graphRootUsed": 10505543680,
                "graphStatus": {
                    "Backing Filesystem": "xfs",
                    "Native Overlay Diff": "true",
                    "Supports d_type": "true",
                    "Supports shifting": "false",
                    "Supports volatile": "true",
                    "Using metacopy": "false"
                },
                "imageCopyTmpDir": "/var/tmp",
                "imageStore": {
                    "number": 14
                },
                "runRoot": "/run/user/501/containers",
                "transientStore": false,
                "volumePath": "/var/home/core/.local/share/containers/storage/volumes"
            },
            "version": {
                "APIVersion": "5.5.2",
                "BuildOrigin": "Copr: packit/containers-podman-26503",
                "Built": 1750723200,
                "BuiltTime": "Mon Jun 23 20:00:00 2025",
                "GitCommit": "e7d8226745ba07a64b7176a7f128e4ef53225a0e",
                "GoVersion": "go1.23.10",
                "Os": "linux",
                "OsArch": "linux/arm64",
                "Version": "5.5.2"
            }
        },
        "Name": "podman"
    },
    "Image": "quay.io/ramalama/ramalama:latest",
    "Runtime": "llama.cpp",
    "Selinux": false,
    "Shortnames": {
        "Files": [],
        "Names": {}
    },
    "Store": "/Users/ihrachyshka/.local/share/ramalama",
    "UseContainer": true,
    "Version": "0.11.3"
}

Upstream Latest Release

Yes

Additional environment details

$ CONTAINERS_MACHINE_PROVIDER=libkrun podman machine info
host:
    arch: arm64
    currentmachine: podman-machine-default
    defaultmachine: podman-machine-default
    eventsdir: /var/folders/36/97s55mw54fzb0bw3r2w44h9r0000gn/T/storage-run-501/podman
    machineconfigdir: /Users/ihrachyshka/.config/containers/podman/machine/libkrun
    machineimagedir: /Users/ihrachyshka/.local/share/containers/podman/machine/libkrun
    machinestate: Running
    numberofmachines: 1
    os: darwin
    vmtype: libkrun
version:
    apiversion: 5.5.2
    version: 5.5.2
    goversion: go1.24.5
    gitcommit: ""
    builttime: Mon Dec 31 19:00:00 1979
    built: 315532800
    osarch: darwin/arm64
    os: darwin

All components - podman, ramalama, libkrun and krunkit - are built via Nixpkgs.

$ ramalama version
ramalama version 0.11.3
$ krunkit --version
krunkit 0.2.2
$ otool -L /nix/store/i1dkq0x370ymv0vfw4mwhm17aximrfmp-krunkit-0.2.2/bin/krunkit
/nix/store/i1dkq0x370ymv0vfw4mwhm17aximrfmp-krunkit-0.2.2/bin/krunkit:
	/nix/store/6by0kli10m9w9iijgr4nl4lpzfnrnp1v-libkrun-efi-1.14.0/lib/libkrun-efi.1.dylib (compatibility version 1.0.0, current version 1.14.0)
[...]
$

In case you need to consult how krunkit / libkrun-efi were built,

main Nix build instructions: libkrun-efi: init at 1.15.1, krunkit: init at 1.0.0 (Supersedes #416870) NixOS/nixpkgs#417062
integration of the above into my env + version bumps for krunkit and libkrun: booxter/nix@5977642

Additional information

No response

booxter · 2025-08-15T04:48:30Z

booxter
Aug 15, 2025
Author

With --debug (not sure it reveals anything of interest though):

$ ramalama --debug run granite3.3
2025-08-15 00:46:53 - DEBUG - run_cmd: npu-smi info
2025-08-15 00:46:53 - DEBUG - Working directory: None
2025-08-15 00:46:53 - DEBUG - Ignore stderr: False
2025-08-15 00:46:53 - DEBUG - Ignore all: False
2025-08-15 00:46:53 - DEBUG - run_cmd: mthreads-gmi
2025-08-15 00:46:53 - DEBUG - Working directory: None
2025-08-15 00:46:53 - DEBUG - Ignore stderr: False
2025-08-15 00:46:53 - DEBUG - Ignore all: False
2025-08-15 00:46:53 - DEBUG - run_cmd: podman inspect quay.io/ramalama/ramalama:0.11
2025-08-15 00:46:53 - DEBUG - Working directory: None
2025-08-15 00:46:53 - DEBUG - Ignore stderr: False
2025-08-15 00:46:53 - DEBUG - Ignore all: True
2025-08-15 00:46:53 - DEBUG - Checking if 8080 is available
2025-08-15 00:46:53 - DEBUG - Checking if 8080 is available
2025-08-15 00:46:53 - DEBUG - run_cmd: npu-smi info
2025-08-15 00:46:53 - DEBUG - Working directory: None
2025-08-15 00:46:53 - DEBUG - Ignore stderr: False
2025-08-15 00:46:53 - DEBUG - Ignore all: False
2025-08-15 00:46:53 - DEBUG - run_cmd: mthreads-gmi
2025-08-15 00:46:53 - DEBUG - Working directory: None
2025-08-15 00:46:53 - DEBUG - Ignore stderr: False
2025-08-15 00:46:53 - DEBUG - Ignore all: False
2025-08-15 00:46:53 - DEBUG - exec_cmd: podman run --rm --label ai.ramalama.model=ollama://library/granite3.3:latest --label ai.ramalama.engine=podman --label ai.ramalama.runtime=llama.cpp --label ai.ramalama.port=8080 --label ai.ramalama.command=run --device /dev/dri -p 8080:8080 --security-opt=label=disable --cap-drop=all --security-opt=no-new-privileges --pull newer -t -d -i --label ai.ramalama --name ramalama_57GsQcV66b --env=HOME=/tmp --init --mount=type=bind,src=/Users/ihrachyshka/.local/share/ramalama/store/ollama/library/granite3.3/blobs/sha256-77bcee066a76dcdd10d0d123c87e32c8ec2c74e31b6ffd87ebee49c9ac215dca,destination=/mnt/models/granite3.3,ro --mount=type=bind,src=/Users/ihrachyshka/.local/share/ramalama/store/ollama/library/granite3.3/blobs/sha256-122661774644959cd109295a907dc4305a7b72f9ce8b9ba0fe43dd2393a7becd,destination=/mnt/models/config.json,ro --mount=type=bind,src=/Users/ihrachyshka/.local/share/ramalama/store/ollama/library/granite3.3/blobs/sha256-3da071a01bbe5a1aa1e9766149ff67ed2b232f63d55e6ed50e3777b74536a67f,destination=/mnt/models/chat_template,ro quay.io/ramalama/ramalama:latest llama-server --port 8080 --model /mnt/models/granite3.3 --no-warmup --jinja --chat-template-file /mnt/models/chat_template --log-colors --alias granite3.3 --ctx-size 2048 --temp 0.8 --cache-reuse 256 -v -ngl 999 --threads 5 --host 0.0.0.0
291822d5c5503c86d583deb06f8f29beea890dcdccb7719909c1fb4749805cf0
(ins)🦭 > hey!!!
2025-08-15 00:46:56 - DEBUG - Request: URL=http://127.0.0.1:8080/chat/completions, Data=b'{"stream": true, "messages": [{"role": "user", "content": "hey!!!"}], "model": "granite3.3"}', Headers={'Content-Type': 'application/json'}
⠹
chrchranhanhanhanhanhanhanhanhanhanhanhanhanhanhanhanhanhanhanhanhanhanhanhanh FA FA FA FAbudbudbudbudbudbudbudbudbudbud FA FAvelop ([ ([ ([ ([ ([ SELECT identical idle idleanhanhanhanh FA FA FA FA FA FA FA FA FA FA FA FA FA FA FA FA FA^C

0 replies

booxter · 2025-08-15T04:53:47Z

booxter
Aug 15, 2025
Author

$ nix-shell -p neofetch --command 'neofetch --off'
ihrachyshka@mair
----------------
OS: macOS 15.6 24G84 arm64
Host: Mac16,12
Kernel: 24.6.0
Uptime: 10 days, 9 hours, 9 mins
Packages: 8 (brew), 177 (nix-system), 1073 (nix-user)
Shell: bash 5.3.0
Resolution: 2560x1664
DE: Aqua
WM: Amethyst
Terminal: /dev/ttys003
CPU: Apple M4
GPU: Apple M4
Memory: 19412MiB / 24576MiB

0 replies

kpouget · 2025-08-15T09:09:15Z

kpouget
Aug 15, 2025
Collaborator

can you take the logs of the container that is returning garbage?

ramalama run llama3.2
🦭 > hey!!!
Hey! How's it going? What's on your mind? Do you want to chat about something in particular or just shoot the breeze? I'm all ears! <3>

not reproducing with llama3.2 (on MacOS) :/

ramalama --engine podman run granite3.3:latest
🦭 > hello?
</</</</<///></<///>/</</</<//>/</</</<//</<//>/</</</</<//>/</</</</<//>/</</<//

interesting :)

the logs start with this:

podman logs 3c6eab4dca90
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Virtio-GPU Venus (Apple M4 Pro) (venus) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 0 | matrix cores: none
common_chat_templates_init: failed to parse chat template (defaulting to chatml): Expected value expression at row 1, column 4:
{{- /*

which should be independent of Vulkan

0 replies

joelpurra · 2025-08-15T10:22:44Z

joelpurra
Aug 15, 2025

Would not be surprised if the garbage output is related to #1374.

0 replies

booxter · 2025-08-15T14:33:02Z

booxter
Aug 15, 2025
Author

Here's what I get, with container logs:

First attempt just crashed:

$ $ ramalama --engine podman --debug run llama3.2
2025-08-15 10:23:50 - DEBUG - run_cmd: npu-smi info
2025-08-15 10:23:50 - DEBUG - Working directory: None
2025-08-15 10:23:50 - DEBUG - Ignore stderr: False
2025-08-15 10:23:50 - DEBUG - Ignore all: False
2025-08-15 10:23:50 - DEBUG - run_cmd: mthreads-gmi
2025-08-15 10:23:50 - DEBUG - Working directory: None
2025-08-15 10:23:50 - DEBUG - Ignore stderr: False
2025-08-15 10:23:50 - DEBUG - Ignore all: False
2025-08-15 10:23:50 - DEBUG - run_cmd: podman inspect quay.io/ramalama/ramalama:0.11
2025-08-15 10:23:50 - DEBUG - Working directory: None
2025-08-15 10:23:50 - DEBUG - Ignore stderr: False
2025-08-15 10:23:50 - DEBUG - Ignore all: True
2025-08-15 10:23:50 - DEBUG - Checking if 8080 is available
2025-08-15 10:23:50 - DEBUG - Checking if 8080 is available
2025-08-15 10:23:50 - DEBUG - run_cmd: npu-smi info
2025-08-15 10:23:50 - DEBUG - Working directory: None
2025-08-15 10:23:50 - DEBUG - Ignore stderr: False
2025-08-15 10:23:50 - DEBUG - Ignore all: False
2025-08-15 10:23:50 - DEBUG - run_cmd: mthreads-gmi
2025-08-15 10:23:50 - DEBUG - Working directory: None
2025-08-15 10:23:50 - DEBUG - Ignore stderr: False
2025-08-15 10:23:50 - DEBUG - Ignore all: False
2025-08-15 10:23:50 - DEBUG - exec_cmd: podman run --rm --label ai.ramalama.model=ollama://library/llama3.2:latest --label ai.ramalama.engine=podman --label ai.ramalama.runtime=llama.cpp --label ai.ramalama.port=8080 --label ai.ramalama.command=run --device /dev/dri -p 8080:8080 --security-opt=label=disable --cap-drop=all --security-opt=no-new-privileges --pull newer -t -d -i --label ai.ramalama --name ramalama_qVOepw2dpP --env=HOME=/tmp --init --mount=type=bind,src=/Users/ihrachyshka/.local/share/ramalama/store/ollama/library/llama3.2/blobs/sha256-dde5aa3fc5ffc17176b5e8bdc82f587b24b2678c6c66101bf7da77af9f7ccdff,destination=/mnt/models/llama3.2,ro --mount=type=bind,src=/Users/ihrachyshka/.local/share/ramalama/store/ollama/library/llama3.2/blobs/sha256-34bb5ab01051a11372a91f95f3fbbc51173eed8e7f13ec395b9ae9b8bd0e242b,destination=/mnt/models/config.json,ro --mount=type=bind,src=/Users/ihrachyshka/.local/share/ramalama/store/ollama/library/llama3.2/blobs/sha256-966de95ca8a62200913e3f8bfbf84c8494536f1b94b49166851e76644e966396,destination=/mnt/models/chat_template,ro quay.io/ramalama/ramalama:latest llama-server --port 8080 --model /mnt/models/llama3.2 --no-warmup --jinja --chat-template-file /mnt/models/chat_template --log-colors --alias llama3.2 --ctx-size 2048 --temp 0.8 --cache-reuse 256 -v -ngl 999 --threads 5 --host 0.0.0.0
eb9217d83cebaf346bfec693e71baeb846014cc67d32a805823237d22fc33209
(ins)🦭 > hey
2025-08-15 10:24:25 - DEBUG - Request: URL=http://127.0.0.1:8080/chat/completions, Data=b'{"stream": true, "messages": [{"role": "user", "content": "hey"}], "model": "llama3.2"}', Headers={'Content-Type': 'application/json'}

2025-08-15 10:24:26 - DEBUG - run_cmd: podman inspect --format "{{ .Pod }}" ramalama_qVOepw2dpP
2025-08-15 10:24:26 - DEBUG - Working directory: None
2025-08-15 10:24:26 - DEBUG - Ignore stderr: True
2025-08-15 10:24:26 - DEBUG - Ignore all: False
2025-08-15 10:24:26 - DEBUG - Command '['podman', 'inspect', '--format', '{{ .Pod }}', 'ramalama_qVOepw2dpP']' returned non-zero exit status 125.
2025-08-15 10:24:26 - DEBUG - run_cmd: podman inspect --format "{{ .Pod }}" ramalama_qVOepw2dpP-pod-model-server
2025-08-15 10:24:26 - DEBUG - Working directory: None
2025-08-15 10:24:26 - DEBUG - Ignore stderr: True
2025-08-15 10:24:26 - DEBUG - Ignore all: False
2025-08-15 10:24:26 - DEBUG - Command '['podman', 'inspect', '--format', '{{ .Pod }}', 'ramalama_qVOepw2dpP-pod-model-server']' returned non-zero exit status 125.
2025-08-15 10:24:26 - DEBUG - run_cmd: podman stop -t=0 --ignore True ramalama_qVOepw2dpP
2025-08-15 10:24:26 - DEBUG - Working directory: None
2025-08-15 10:24:26 - DEBUG - Ignore stderr: False
2025-08-15 10:24:26 - DEBUG - Ignore all: False
2025-08-15 10:24:26 - DEBUG - Command finished with return code: 0

On container side:

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Virtio-GPU Venus (Apple M4) (venus) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memor
y: 32768 | int dot: 0 | matrix cores: none
common_chat_templates_init: failed to parse chat template (defaulting to chatml): Failed to parse number: '.' ([
json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - invalid lite
ral; last read: '.') at row 5, column 21:

{{ if .System }}{{ .System }}
                    ^
{{- end }}

build: 1 (3f4fc97) with cc (GCC) 15.1.1 20250521 (Red Hat 15.1.1-2) for aarch64-redhat-linux
system info: n_threads = 5, n_threads_batch = 5, total_threads = 5

system_info: n_threads = 5 (n_threads_batch = 5) / 5 | CPU : NEON = 1 | ARM_FMA = 1 | LLAMAFILE = 1 | OPENMP = 1
 | REPACK = 1 |

main: binding port with default address family
main: HTTP server is listening, hostname: 0.0.0.0, port: 8080, http threads: 4
main: loading model
srv    load_model: loading model '/mnt/models/llama3.2'
llama_model_load_from_file_impl: using device Vulkan0 (Virtio-GPU Venus (Apple M4)) - 24576 MiB free
llama_model_loader: loaded meta data with 30 key-value pairs and 255 tensors from /mnt/models/llama3.2 (version
GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Llama 3.2 3B Instruc
t
llama_model_loader: - kv   3:                           general.finetune str              = Instruct
llama_model_loader: - kv   4:                           general.basename str              = Llama-3.2
llama_model_loader: - kv   5:                         general.size_label str              = 3B
llama_model_loader: - kv   6:                               general.tags arr[str,6]       = ["facebook", "meta",
 "pytorch", "llam...
llama_model_loader: - kv   7:                          general.languages arr[str,8]       = ["en", "de", "fr", "
it", "pt", "hi", ...
llama_model_loader: - kv   8:                          llama.block_count u32              = 28
llama_model_loader: - kv   9:                       llama.context_length u32              = 131072
llama_model_loader: - kv  10:                     llama.embedding_length u32              = 3072
llama_model_loader: - kv  11:                  llama.feed_forward_length u32              = 8192
llama_model_loader: - kv  12:                 llama.attention.head_count u32              = 24
llama_model_loader: - kv  13:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  14:                       llama.rope.freq_base f32              = 500000.000000
llama_model_loader: - kv  15:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  16:                 llama.attention.key_length u32              = 128
llama_model_loader: - kv  17:               llama.attention.value_length u32              = 128
llama_model_loader: - kv  18:                          general.file_type u32              = 15
llama_model_loader: - kv  19:                           llama.vocab_size u32              = 128256
llama_model_loader: - kv  20:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  21:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  22:                         tokenizer.ggml.pre str              = llama-bpe
llama_model_loader: - kv  23:                      tokenizer.ggml.tokens arr[str,128256]  = ["!", "\"", "#", "$"
, "%", "&", "'", ...
llama_model_loader: - kv  24:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1
, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  25:                      tokenizer.ggml.merges arr[str,280147]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ
 ĠĠ", "...
llama_model_loader: - kv  26:                tokenizer.ggml.bos_token_id u32              = 128000
llama_model_loader: - kv  27:                tokenizer.ggml.eos_token_id u32              = 128009
llama_model_loader: - kv  28:                    tokenizer.chat_template str              = {{- bos_token }}\n{%
- if custom_tools ...
llama_model_loader: - kv  29:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   58 tensors
llama_model_loader: - type q4_K:  168 tensors
llama_model_loader: - type q6_K:   29 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 1.87 GiB (5.01 BPW)
init_tokenizer: initializing tokenizer for type 2
load: control token: 128254 '<|reserved_special_token_246|>' is not marked as EOG
load: control token: 128249 '<|reserved_special_token_241|>' is not marked as EOG
load: control token: 128246 '<|reserved_special_token_238|>' is not marked as EOG
load: control token: 128243 '<|reserved_special_token_235|>' is not marked as EOG
load: control token: 128242 '<|reserved_special_token_234|>' is not marked as EOG
load: control token: 128241 '<|reserved_special_token_233|>' is not marked as EOG
load: control token: 128240 '<|reserved_special_token_232|>' is not marked as EOG
load: control token: 128235 '<|reserved_special_token_227|>' is not marked as EOG
load: control token: 128231 '<|reserved_special_token_223|>' is not marked as EOG
load: control token: 128230 '<|reserved_special_token_222|>' is not marked as EOG
load: control token: 128228 '<|reserved_special_token_220|>' is not marked as EOG
load: control token: 128225 '<|reserved_special_token_217|>' is not marked as EOG
load: control token: 128218 '<|reserved_special_token_210|>' is not marked as EOG
load: control token: 128214 '<|reserved_special_token_206|>' is not marked as EOG
load: control token: 128213 '<|reserved_special_token_205|>' is not marked as EOG
load: control token: 128207 '<|reserved_special_token_199|>' is not marked as EOG
load: control token: 128206 '<|reserved_special_token_198|>' is not marked as EOG
load: control token: 128204 '<|reserved_special_token_196|>' is not marked as EOG
load: control token: 128200 '<|reserved_special_token_192|>' is not marked as EOG
load: control token: 128199 '<|reserved_special_token_191|>' is not marked as EOG
load: control token: 128198 '<|reserved_special_token_190|>' is not marked as EOG
load: control token: 128196 '<|reserved_special_token_188|>' is not marked as EOG
load: control token: 128194 '<|reserved_special_token_186|>' is not marked as EOG
load: control token: 128193 '<|reserved_special_token_185|>' is not marked as EOG
load: control token: 128188 '<|reserved_special_token_180|>' is not marked as EOG
load: control token: 128187 '<|reserved_special_token_179|>' is not marked as EOG
load: control token: 128185 '<|reserved_special_token_177|>' is not marked as EOG
load: control token: 128184 '<|reserved_special_token_176|>' is not marked as EOG
load: control token: 128180 '<|reserved_special_token_172|>' is not marked as EOG
load: control token: 128179 '<|reserved_special_token_171|>' is not marked as EOG
load: control token: 128178 '<|reserved_special_token_170|>' is not marked as EOG
load: control token: 128177 '<|reserved_special_token_169|>' is not marked as EOG
load: control token: 128176 '<|reserved_special_token_168|>' is not marked as EOG
load: control token: 128175 '<|reserved_special_token_167|>' is not marked as EOG
load: control token: 128171 '<|reserved_special_token_163|>' is not marked as EOG
load: control token: 128170 '<|reserved_special_token_162|>' is not marked as EOG
load: control token: 128169 '<|reserved_special_token_161|>' is not marked as EOG
load: control token: 128168 '<|reserved_special_token_160|>' is not marked as EOG
load: control token: 128165 '<|reserved_special_token_157|>' is not marked as EOG
load: control token: 128162 '<|reserved_special_token_154|>' is not marked as EOG
load: control token: 128158 '<|reserved_special_token_150|>' is not marked as EOG
load: control token: 128156 '<|reserved_special_token_148|>' is not marked as EOG
load: control token: 128155 '<|reserved_special_token_147|>' is not marked as EOG
load: control token: 128154 '<|reserved_special_token_146|>' is not marked as EOG
load: control token: 128151 '<|reserved_special_token_143|>' is not marked as EOG
load: control token: 128149 '<|reserved_special_token_141|>' is not marked as EOG
load: control token: 128147 '<|reserved_special_token_139|>' is not marked as EOG
load: control token: 128146 '<|reserved_special_token_138|>' is not marked as EOG
load: control token: 128144 '<|reserved_special_token_136|>' is not marked as EOG
load: control token: 128142 '<|reserved_special_token_134|>' is not marked as EOG
load: control token: 128141 '<|reserved_special_token_133|>' is not marked as EOG
load: control token: 128138 '<|reserved_special_token_130|>' is not marked as EOG
load: control token: 128136 '<|reserved_special_token_128|>' is not marked as EOG
load: control token: 128135 '<|reserved_special_token_127|>' is not marked as EOG
load: control token: 128134 '<|reserved_special_token_126|>' is not marked as EOG
load: control token: 128133 '<|reserved_special_token_125|>' is not marked as EOG
load: control token: 128131 '<|reserved_special_token_123|>' is not marked as EOG
load: control token: 128128 '<|reserved_special_token_120|>' is not marked as EOG
load: control token: 128124 '<|reserved_special_token_116|>' is not marked as EOG
load: control token: 128123 '<|reserved_special_token_115|>' is not marked as EOG
load: control token: 128122 '<|reserved_special_token_114|>' is not marked as EOG
load: control token: 128119 '<|reserved_special_token_111|>' is not marked as EOG
load: control token: 128115 '<|reserved_special_token_107|>' is not marked as EOG
load: control token: 128112 '<|reserved_special_token_104|>' is not marked as EOG
load: control token: 128110 '<|reserved_special_token_102|>' is not marked as EOG
load: control token: 128109 '<|reserved_special_token_101|>' is not marked as EOG
load: control token: 128108 '<|reserved_special_token_100|>' is not marked as EOG
load: control token: 128106 '<|reserved_special_token_98|>' is not marked as EOG
load: control token: 128103 '<|reserved_special_token_95|>' is not marked as EOG
load: control token: 128102 '<|reserved_special_token_94|>' is not marked as EOG
load: control token: 128101 '<|reserved_special_token_93|>' is not marked as EOG
load: control token: 128097 '<|reserved_special_token_89|>' is not marked as EOG
load: control token: 128091 '<|reserved_special_token_83|>' is not marked as EOG
load: control token: 128090 '<|reserved_special_token_82|>' is not marked as EOG
load: control token: 128089 '<|reserved_special_token_81|>' is not marked as EOG
load: control token: 128087 '<|reserved_special_token_79|>' is not marked as EOG
load: control token: 128085 '<|reserved_special_token_77|>' is not marked as EOG
load: control token: 128081 '<|reserved_special_token_73|>' is not marked as EOG
load: control token: 128078 '<|reserved_special_token_70|>' is not marked as EOG
load: control token: 128076 '<|reserved_special_token_68|>' is not marked as EOG
load: control token: 128075 '<|reserved_special_token_67|>' is not marked as EOG
load: control token: 128073 '<|reserved_special_token_65|>' is not marked as EOG
load: control token: 128068 '<|reserved_special_token_60|>' is not marked as EOG
load: control token: 128067 '<|reserved_special_token_59|>' is not marked as EOG
load: control token: 128065 '<|reserved_special_token_57|>' is not marked as EOG
load: control token: 128063 '<|reserved_special_token_55|>' is not marked as EOG
load: control token: 128062 '<|reserved_special_token_54|>' is not marked as EOG
load: control token: 128060 '<|reserved_special_token_52|>' is not marked as EOG
load: control token: 128059 '<|reserved_special_token_51|>' is not marked as EOG
load: control token: 128057 '<|reserved_special_token_49|>' is not marked as EOG
load: control token: 128054 '<|reserved_special_token_46|>' is not marked as EOG
load: control token: 128046 '<|reserved_special_token_38|>' is not marked as EOG
load: control token: 128045 '<|reserved_special_token_37|>' is not marked as EOG
load: control token: 128044 '<|reserved_special_token_36|>' is not marked as EOG
load: control token: 128043 '<|reserved_special_token_35|>' is not marked as EOG
load: control token: 128038 '<|reserved_special_token_30|>' is not marked as EOG
load: control token: 128036 '<|reserved_special_token_28|>' is not marked as EOG
load: control token: 128035 '<|reserved_special_token_27|>' is not marked as EOG
load: control token: 128032 '<|reserved_special_token_24|>' is not marked as EOG
load: control token: 128028 '<|reserved_special_token_20|>' is not marked as EOG
load: control token: 128027 '<|reserved_special_token_19|>' is not marked as EOG
load: control token: 128024 '<|reserved_special_token_16|>' is not marked as EOG
load: control token: 128023 '<|reserved_special_token_15|>' is not marked as EOG
load: control token: 128022 '<|reserved_special_token_14|>' is not marked as EOG
load: control token: 128021 '<|reserved_special_token_13|>' is not marked as EOG
load: control token: 128018 '<|reserved_special_token_10|>' is not marked as EOG
load: control token: 128016 '<|reserved_special_token_8|>' is not marked as EOG
load: control token: 128015 '<|reserved_special_token_7|>' is not marked as EOG
load: control token: 128013 '<|reserved_special_token_5|>' is not marked as EOG
load: control token: 128011 '<|reserved_special_token_3|>' is not marked as EOG
load: control token: 128005 '<|reserved_special_token_2|>' is not marked as EOG
load: control token: 128004 '<|finetune_right_pad_id|>' is not marked as EOG
load: control token: 128002 '<|reserved_special_token_0|>' is not marked as EOG
load: control token: 128252 '<|reserved_special_token_244|>' is not marked as EOG
load: control token: 128190 '<|reserved_special_token_182|>' is not marked as EOG
load: control token: 128183 '<|reserved_special_token_175|>' is not marked as EOG
load: control token: 128137 '<|reserved_special_token_129|>' is not marked as EOG
load: control token: 128182 '<|reserved_special_token_174|>' is not marked as EOG
load: control token: 128040 '<|reserved_special_token_32|>' is not marked as EOG
load: control token: 128048 '<|reserved_special_token_40|>' is not marked as EOG
load: control token: 128092 '<|reserved_special_token_84|>' is not marked as EOG
load: control token: 128215 '<|reserved_special_token_207|>' is not marked as EOG
load: control token: 128107 '<|reserved_special_token_99|>' is not marked as EOG
load: control token: 128208 '<|reserved_special_token_200|>' is not marked as EOG
load: control token: 128145 '<|reserved_special_token_137|>' is not marked as EOG
load: control token: 128031 '<|reserved_special_token_23|>' is not marked as EOG
load: control token: 128129 '<|reserved_special_token_121|>' is not marked as EOG
load: control token: 128201 '<|reserved_special_token_193|>' is not marked as EOG
load: control token: 128074 '<|reserved_special_token_66|>' is not marked as EOG
load: control token: 128095 '<|reserved_special_token_87|>' is not marked as EOG
load: control token: 128186 '<|reserved_special_token_178|>' is not marked as EOG
load: control token: 128143 '<|reserved_special_token_135|>' is not marked as EOG
load: control token: 128229 '<|reserved_special_token_221|>' is not marked as EOG
load: control token: 128007 '<|end_header_id|>' is not marked as EOG
load: control token: 128055 '<|reserved_special_token_47|>' is not marked as EOG
load: control token: 128056 '<|reserved_special_token_48|>' is not marked as EOG
load: control token: 128061 '<|reserved_special_token_53|>' is not marked as EOG
load: control token: 128153 '<|reserved_special_token_145|>' is not marked as EOG
load: control token: 128152 '<|reserved_special_token_144|>' is not marked as EOG
load: control token: 128212 '<|reserved_special_token_204|>' is not marked as EOG
load: control token: 128172 '<|reserved_special_token_164|>' is not marked as EOG
load: control token: 128160 '<|reserved_special_token_152|>' is not marked as EOG
load: control token: 128041 '<|reserved_special_token_33|>' is not marked as EOG
load: control token: 128181 '<|reserved_special_token_173|>' is not marked as EOG
load: control token: 128094 '<|reserved_special_token_86|>' is not marked as EOG
load: control token: 128118 '<|reserved_special_token_110|>' is not marked as EOG
load: control token: 128236 '<|reserved_special_token_228|>' is not marked as EOG
load: control token: 128148 '<|reserved_special_token_140|>' is not marked as EOG
load: control token: 128042 '<|reserved_special_token_34|>' is not marked as EOG
load: control token: 128139 '<|reserved_special_token_131|>' is not marked as EOG
load: control token: 128173 '<|reserved_special_token_165|>' is not marked as EOG
load: control token: 128239 '<|reserved_special_token_231|>' is not marked as EOG
load: control token: 128157 '<|reserved_special_token_149|>' is not marked as EOG
load: control token: 128052 '<|reserved_special_token_44|>' is not marked as EOG
load: control token: 128026 '<|reserved_special_token_18|>' is not marked as EOG
load: control token: 128003 '<|reserved_special_token_1|>' is not marked as EOG
load: control token: 128019 '<|reserved_special_token_11|>' is not marked as EOG
load: control token: 128116 '<|reserved_special_token_108|>' is not marked as EOG
load: control token: 128161 '<|reserved_special_token_153|>' is not marked as EOG
load: control token: 128226 '<|reserved_special_token_218|>' is not marked as EOG
load: control token: 128159 '<|reserved_special_token_151|>' is not marked as EOG
load: control token: 128012 '<|reserved_special_token_4|>' is not marked as EOG
load: control token: 128088 '<|reserved_special_token_80|>' is not marked as EOG
load: control token: 128163 '<|reserved_special_token_155|>' is not marked as EOG
load: control token: 128113 '<|reserved_special_token_105|>' is not marked as EOG
load: control token: 128250 '<|reserved_special_token_242|>' is not marked as EOG
load: control token: 128125 '<|reserved_special_token_117|>' is not marked as EOG
load: control token: 128053 '<|reserved_special_token_45|>' is not marked as EOG
load: control token: 128224 '<|reserved_special_token_216|>' is not marked as EOG
load: control token: 128247 '<|reserved_special_token_239|>' is not marked as EOG
load: control token: 128251 '<|reserved_special_token_243|>' is not marked as EOG
load: control token: 128216 '<|reserved_special_token_208|>' is not marked as EOG
load: control token: 128006 '<|start_header_id|>' is not marked as EOG
load: control token: 128211 '<|reserved_special_token_203|>' is not marked as EOG
load: control token: 128077 '<|reserved_special_token_69|>' is not marked as EOG
load: control token: 128237 '<|reserved_special_token_229|>' is not marked as EOG
load: control token: 128086 '<|reserved_special_token_78|>' is not marked as EOG
load: control token: 128227 '<|reserved_special_token_219|>' is not marked as EOG
load: control token: 128058 '<|reserved_special_token_50|>' is not marked as EOG
load: control token: 128100 '<|reserved_special_token_92|>' is not marked as EOG
load: control token: 128209 '<|reserved_special_token_201|>' is not marked as EOG
load: control token: 128084 '<|reserved_special_token_76|>' is not marked as EOG
load: control token: 128071 '<|reserved_special_token_63|>' is not marked as EOG
load: control token: 128070 '<|reserved_special_token_62|>' is not marked as EOG
load: control token: 128049 '<|reserved_special_token_41|>' is not marked as EOG
load: control token: 128197 '<|reserved_special_token_189|>' is not marked as EOG
load: control token: 128072 '<|reserved_special_token_64|>' is not marked as EOG
load: control token: 128000 '<|begin_of_text|>' is not marked as EOG
load: control token: 128223 '<|reserved_special_token_215|>' is not marked as EOG
load: control token: 128217 '<|reserved_special_token_209|>' is not marked as EOG
load: control token: 128111 '<|reserved_special_token_103|>' is not marked as EOG
load: control token: 128203 '<|reserved_special_token_195|>' is not marked as EOG
load: control token: 128051 '<|reserved_special_token_43|>' is not marked as EOG
load: control token: 128030 '<|reserved_special_token_22|>' is not marked as EOG
load: control token: 128117 '<|reserved_special_token_109|>' is not marked as EOG
load: control token: 128010 '<|python_tag|>' is not marked as EOG
load: control token: 128238 '<|reserved_special_token_230|>' is not marked as EOG
load: control token: 128255 '<|reserved_special_token_247|>' is not marked as EOG
load: control token: 128202 '<|reserved_special_token_194|>' is not marked as EOG
load: control token: 128132 '<|reserved_special_token_124|>' is not marked as EOG
load: control token: 128248 '<|reserved_special_token_240|>' is not marked as EOG
load: control token: 128167 '<|reserved_special_token_159|>' is not marked as EOG
load: control token: 128127 '<|reserved_special_token_119|>' is not marked as EOG
load: control token: 128105 '<|reserved_special_token_97|>' is not marked as EOG
load: control token: 128039 '<|reserved_special_token_31|>' is not marked as EOG
load: control token: 128232 '<|reserved_special_token_224|>' is not marked as EOG
load: control token: 128166 '<|reserved_special_token_158|>' is not marked as EOG
load: control token: 128130 '<|reserved_special_token_122|>' is not marked as EOG
load: control token: 128114 '<|reserved_special_token_106|>' is not marked as EOG
load: control token: 128234 '<|reserved_special_token_226|>' is not marked as EOG
load: control token: 128191 '<|reserved_special_token_183|>' is not marked as EOG
load: control token: 128064 '<|reserved_special_token_56|>' is not marked as EOG
load: control token: 128140 '<|reserved_special_token_132|>' is not marked as EOG
load: control token: 128096 '<|reserved_special_token_88|>' is not marked as EOG
load: control token: 128098 '<|reserved_special_token_90|>' is not marked as EOG
load: control token: 128192 '<|reserved_special_token_184|>' is not marked as EOG
load: control token: 128093 '<|reserved_special_token_85|>' is not marked as EOG
load: control token: 128150 '<|reserved_special_token_142|>' is not marked as EOG
load: control token: 128222 '<|reserved_special_token_214|>' is not marked as EOG
load: control token: 128233 '<|reserved_special_token_225|>' is not marked as EOG
load: control token: 128220 '<|reserved_special_token_212|>' is not marked as EOG
load: control token: 128034 '<|reserved_special_token_26|>' is not marked as EOG
load: control token: 128033 '<|reserved_special_token_25|>' is not marked as EOG
load: control token: 128253 '<|reserved_special_token_245|>' is not marked as EOG
load: control token: 128195 '<|reserved_special_token_187|>' is not marked as EOG
load: control token: 128099 '<|reserved_special_token_91|>' is not marked as EOG
load: control token: 128189 '<|reserved_special_token_181|>' is not marked as EOG
load: control token: 128210 '<|reserved_special_token_202|>' is not marked as EOG
load: control token: 128174 '<|reserved_special_token_166|>' is not marked as EOG
load: control token: 128083 '<|reserved_special_token_75|>' is not marked as EOG
load: control token: 128080 '<|reserved_special_token_72|>' is not marked as EOG
load: control token: 128104 '<|reserved_special_token_96|>' is not marked as EOG
load: control token: 128082 '<|reserved_special_token_74|>' is not marked as EOG
load: control token: 128219 '<|reserved_special_token_211|>' is not marked as EOG
load: control token: 128017 '<|reserved_special_token_9|>' is not marked as EOG
load: control token: 128050 '<|reserved_special_token_42|>' is not marked as EOG
load: control token: 128205 '<|reserved_special_token_197|>' is not marked as EOG
load: control token: 128047 '<|reserved_special_token_39|>' is not marked as EOG
load: control token: 128164 '<|reserved_special_token_156|>' is not marked as EOG
load: control token: 128020 '<|reserved_special_token_12|>' is not marked as EOG
load: control token: 128069 '<|reserved_special_token_61|>' is not marked as EOG
load: control token: 128245 '<|reserved_special_token_237|>' is not marked as EOG
load: control token: 128121 '<|reserved_special_token_113|>' is not marked as EOG
load: control token: 128079 '<|reserved_special_token_71|>' is not marked as EOG
load: control token: 128037 '<|reserved_special_token_29|>' is not marked as EOG
load: control token: 128244 '<|reserved_special_token_236|>' is not marked as EOG
load: control token: 128029 '<|reserved_special_token_21|>' is not marked as EOG
load: control token: 128221 '<|reserved_special_token_213|>' is not marked as EOG
load: control token: 128066 '<|reserved_special_token_58|>' is not marked as EOG
load: control token: 128120 '<|reserved_special_token_112|>' is not marked as EOG
load: control token: 128014 '<|reserved_special_token_6|>' is not marked as EOG
load: control token: 128025 '<|reserved_special_token_17|>' is not marked as EOG
load: control token: 128126 '<|reserved_special_token_118|>' is not marked as EOG
load: special tokens cache size = 256
load: token to piece cache size = 0.7999 MB
print_info: arch             = llama
print_info: vocab_only       = 0
print_info: n_ctx_train      = 131072
print_info: n_embd           = 3072
print_info: n_layer          = 28
print_info: n_head           = 24
print_info: n_head_kv        = 8
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 3
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 8192
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 0
print_info: rope scaling     = linear
print_info: freq_base_train  = 500000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 131072
print_info: rope_finetuned   = unknown
print_info: model type       = 3B
print_info: model params     = 3.21 B
print_info: general.name     = Llama 3.2 3B Instruct
print_info: vocab type       = BPE
print_info: n_vocab          = 128256
print_info: n_merges         = 280147
print_info: BOS token        = 128000 '<|begin_of_text|>'
print_info: EOS token        = 128009 '<|eot_id|>'
print_info: EOT token        = 128009 '<|eot_id|>'
print_info: EOM token        = 128008 '<|eom_id|>'
print_info: LF token         = 198 'Ċ'
print_info: EOG token        = 128001 '<|end_of_text|>'
print_info: EOG token        = 128008 '<|eom_id|>'
print_info: EOG token        = 128009 '<|eot_id|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: layer   0 assigned to device Vulkan0, is_swa = 0
load_tensors: layer   1 assigned to device Vulkan0, is_swa = 0
load_tensors: layer   2 assigned to device Vulkan0, is_swa = 0
load_tensors: layer   3 assigned to device Vulkan0, is_swa = 0
load_tensors: layer   4 assigned to device Vulkan0, is_swa = 0
load_tensors: layer   5 assigned to device Vulkan0, is_swa = 0
load_tensors: layer   6 assigned to device Vulkan0, is_swa = 0
load_tensors: layer   7 assigned to device Vulkan0, is_swa = 0
load_tensors: layer   8 assigned to device Vulkan0, is_swa = 0
load_tensors: layer   9 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  10 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  11 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  12 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  13 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  14 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  15 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  16 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  17 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  18 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  19 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  20 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  21 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  22 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  23 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  24 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  25 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  26 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  27 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  28 assigned to device Vulkan0, is_swa = 0
load_tensors: tensor 'token_embd.weight' (q6_K) (and 0 others) cannot be used with preferred buffer type Vulkan_
Host, using CPU instead
load_tensors: offloading 28 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 29/29 layers to GPU
load_tensors:      Vulkan0 model buffer size =  1918.35 MiB
load_tensors:   CPU_Mapped model buffer size =   308.23 MiB
...........................................................................
llama_context: constructing llama_context
llama_context: non-unified KV cache requires ggml_set_rows() - forcing unified KV cache
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 2048
llama_context: n_ctx_per_seq = 2048
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 0
llama_context: kv_unified    = true
llama_context: freq_base     = 500000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (2048) < n_ctx_train (131072) -- the full capacity of the model will not be utilize
d
set_abort_callback: call
llama_context: Vulkan_Host  output buffer size =     0.49 MiB
create_memory: n_ctx = 2048 (padded)
llama_kv_cache_unified: layer   0: dev = Vulkan0
llama_kv_cache_unified: layer   1: dev = Vulkan0
llama_kv_cache_unified: layer   2: dev = Vulkan0
llama_kv_cache_unified: layer   3: dev = Vulkan0
llama_kv_cache_unified: layer   4: dev = Vulkan0
llama_kv_cache_unified: layer   5: dev = Vulkan0
llama_kv_cache_unified: layer   6: dev = Vulkan0
llama_kv_cache_unified: layer   7: dev = Vulkan0
llama_kv_cache_unified: layer   8: dev = Vulkan0
llama_kv_cache_unified: layer   9: dev = Vulkan0
llama_kv_cache_unified: layer  10: dev = Vulkan0
llama_kv_cache_unified: layer  11: dev = Vulkan0
llama_kv_cache_unified: layer  12: dev = Vulkan0
llama_kv_cache_unified: layer  13: dev = Vulkan0
llama_kv_cache_unified: layer  14: dev = Vulkan0
llama_kv_cache_unified: layer  15: dev = Vulkan0
llama_kv_cache_unified: layer  16: dev = Vulkan0
llama_kv_cache_unified: layer  17: dev = Vulkan0
llama_kv_cache_unified: layer  18: dev = Vulkan0
llama_kv_cache_unified: layer  19: dev = Vulkan0
llama_kv_cache_unified: layer  20: dev = Vulkan0
llama_kv_cache_unified: layer  21: dev = Vulkan0
llama_kv_cache_unified: layer  22: dev = Vulkan0
llama_kv_cache_unified: layer  23: dev = Vulkan0
llama_kv_cache_unified: layer  24: dev = Vulkan0
llama_kv_cache_unified: layer  25: dev = Vulkan0
llama_kv_cache_unified: layer  26: dev = Vulkan0
llama_kv_cache_unified: layer  27: dev = Vulkan0
llama_kv_cache_unified:    Vulkan0 KV buffer size =   224.00 MiB
llama_kv_cache_unified: size =  224.00 MiB (  2048 cells,  28 layers,  1/ 1 seqs), K (f16):  112.00 MiB, V (f16)
:  112.00 MiB
llama_kv_cache_unified: LLAMA_SET_ROWS=0, using old ggml_cpy() method for backwards compatibility
llama_context: enumerating backends
llama_context: backend_ptrs.size() = 2
llama_context: max_nodes = 2048
llama_context: worst-case: n_tokens = 512, n_seqs = 1, n_outputs = 0
graph_reserve: reserving a graph for ubatch with n_tokens =  512, n_seqs =  1, n_outputs =  512
graph_reserve: reserving a graph for ubatch with n_tokens =    1, n_seqs =  1, n_outputs =    1
graph_reserve: reserving a graph for ubatch with n_tokens =  512, n_seqs =  1, n_outputs =  512
llama_context:    Vulkan0 compute buffer size =   278.50 MiB
llama_context: Vulkan_Host compute buffer size =    10.01 MiB
llama_context: graph nodes  = 1042
llama_context: graph splits = 2
clear_adapter_lora: call
common_init_from_params: added <|end_of_text|> logit bias = -inf
common_init_from_params: added <|eom_id|> logit bias = -inf
common_init_from_params: added <|eot_id|> logit bias = -inf
common_init_from_params: setting dry_penalty_last_n to ctx_size = 2048
common_chat_templates_init: failed to parse chat template (defaulting to chatml): Failed to parse number: '.' ([
json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - invalid lite
ral; last read: '.') at row 5, column 21:

{{ if .System }}{{ .System }}
                    ^
{{- end }}

srv          init: initializing slots, n_slots = 1
slot         init: id  0 | task -1 | new slot n_ctx_slot = 2048
slot        reset: id  0 | task -1 |
main: model loaded
main: chat template, chat_template: {%- for message in messages -%}
  {{- '<|im_start|>' + message.role + '
' + message.content + '<|im_end|>
' -}}
{%- endfor -%}
{%- if add_generation_prompt -%}
  {{- '<|im_start|>assistant
' -}}
{%- endif -%}, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'
main: server is listening on http://0.0.0.0:8080 - starting the main loop
que    start_loop: processing new tasks
que    start_loop: update slots
srv  update_slots: all slots are idle
srv  kv_cache_cle: clearing KV cache
que    start_loop: waiting for new tasks
request: {"stream": true, "messages": [{"role": "user", "content": "hey"}], "model": "llama3.2"}
srv  params_from_: Grammar:
srv  params_from_: Grammar lazy: false
srv  params_from_: Chat format: Content-only
srv  add_waiting_: add task 0 to waiting list. current waiting = 0 (before add)
que          post: new task, id = 0/1, front = 0
que    start_loop: processing new tasks
que    start_loop: processing task, id = 0
slot get_availabl: id  0 | task -1 | selected slot by lru, t_last = -1
slot        reset: id  0 | task -1 |
slot launch_slot_: id  0 | task 0 | launching slot : {"id":0,"id_task":0,"n_ctx":2048,"speculative":false,"is_pr
ocessing":false,"params":{"n_predict":-1,"seed":4294967295,"temperature":0.800000011920929,"dynatemp_range":0.0,
"dynatemp_exponent":1.0,"top_k":40,"top_p":0.949999988079071,"min_p":0.05000000074505806,"top_n_sigma":-1.0,"xtc
_probability":0.0,"xtc_threshold":0.10000000149011612,"typical_p":1.0,"repeat_last_n":64,"repeat_penalty":1.0,"p
resence_penalty":0.0,"frequency_penalty":0.0,"dry_multiplier":0.0,"dry_base":1.75,"dry_allowed_length":2,"dry_pe
nalty_last_n":2048,"dry_sequence_breakers":["\n",":","\"","*"],"mirostat":0,"mirostat_tau":5.0,"mirostat_eta":0.
10000000149011612,"stop":[],"max_tokens":-1,"n_keep":0,"n_discard":0,"ignore_eos":false,"stream":true,"logit_bia
s":[],"n_probs":0,"min_keep":0,"grammar":"","grammar_lazy":false,"grammar_triggers":[],"preserved_tokens":[],"ch
at_format":"Content-only","reasoning_format":"deepseek","reasoning_in_content":false,"thinking_forced_open":fals
e,"samplers":["penalties","dry","top_n_sigma","top_k","typ_p","top_p","min_p","xtc","temperature"],"speculative.
n_max":16,"speculative.n_min":0,"speculative.p_min":0.75,"timings_per_token":false,"post_sampling_probs":false,"
lora":[]},"prompt":"<|begin_of_text|><|im_start|>user\nhey<|im_end|>\n<|im_start|>assistant\n","next_token":{"ha
s_next_token":true,"has_new_line":false,"n_remain":-1,"n_decoded":0,"stopping_word":""}}
slot launch_slot_: id  0 | task 0 | processing task
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 1, front = 0
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 2048, n_keep = 0, n_prompt_tokens = 24
slot update_slots: id  0 | task 0 | trying to reuse chunks with size > 256, slot.n_past = 0
slot update_slots: id  0 | task 0 | after context reuse, new slot.n_past = 0
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 24, n_tokens = 24, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 24, n_tokens = 24
srv  update_slots: decoding batch, n_tokens = 24
clear_adapter_lora: call
set_embeddings: value = 0
MESA-VIRTIO: debug: vn_ring_submit abort on fatal

Second attempt didn't crash but produced garbage:

$ ramalama --engine podman --debug run llama3.2
2025-08-15 10:25:38 - DEBUG - run_cmd: npu-smi info
2025-08-15 10:25:38 - DEBUG - Working directory: None
2025-08-15 10:25:38 - DEBUG - Ignore stderr: False
2025-08-15 10:25:38 - DEBUG - Ignore all: False
2025-08-15 10:25:38 - DEBUG - run_cmd: mthreads-gmi
2025-08-15 10:25:38 - DEBUG - Working directory: None
2025-08-15 10:25:38 - DEBUG - Ignore stderr: False
2025-08-15 10:25:38 - DEBUG - Ignore all: False
2025-08-15 10:25:38 - DEBUG - run_cmd: podman inspect quay.io/ramalama/ramalama:0.11
2025-08-15 10:25:38 - DEBUG - Working directory: None
2025-08-15 10:25:38 - DEBUG - Ignore stderr: False
2025-08-15 10:25:38 - DEBUG - Ignore all: True
2025-08-15 10:25:38 - DEBUG - Checking if 8080 is available
2025-08-15 10:25:38 - DEBUG - Checking if 8080 is available
2025-08-15 10:25:38 - DEBUG - run_cmd: npu-smi info
2025-08-15 10:25:38 - DEBUG - Working directory: None
2025-08-15 10:25:38 - DEBUG - Ignore stderr: False
2025-08-15 10:25:38 - DEBUG - Ignore all: False
2025-08-15 10:25:38 - DEBUG - run_cmd: mthreads-gmi
2025-08-15 10:25:38 - DEBUG - Working directory: None
2025-08-15 10:25:38 - DEBUG - Ignore stderr: False
2025-08-15 10:25:38 - DEBUG - Ignore all: False
2025-08-15 10:25:38 - DEBUG - exec_cmd: podman run --rm --label ai.ramalama.model=ollama://library/llama3.2:latest --label ai.ramalama.engine=podman --label ai.ramalama.runtime=llama.cpp --label ai.ramalama.port=8080 --label ai.ramalama.command=run --device /dev/dri -p 8080:8080 --security-opt=label=disable --cap-drop=all --security-opt=no-new-privileges --pull newer -t -d -i --label ai.ramalama --name ramalama_oHjihQZzdu --env=HOME=/tmp --init --mount=type=bind,src=/Users/ihrachyshka/.local/share/ramalama/store/ollama/library/llama3.2/blobs/sha256-dde5aa3fc5ffc17176b5e8bdc82f587b24b2678c6c66101bf7da77af9f7ccdff,destination=/mnt/models/llama3.2,ro --mount=type=bind,src=/Users/ihrachyshka/.local/share/ramalama/store/ollama/library/llama3.2/blobs/sha256-34bb5ab01051a11372a91f95f3fbbc51173eed8e7f13ec395b9ae9b8bd0e242b,destination=/mnt/models/config.json,ro --mount=type=bind,src=/Users/ihrachyshka/.local/share/ramalama/store/ollama/library/llama3.2/blobs/sha256-966de95ca8a62200913e3f8bfbf84c8494536f1b94b49166851e76644e966396,destination=/mnt/models/chat_template,ro quay.io/ramalama/ramalama:latest llama-server --port 8080 --model /mnt/models/llama3.2 --no-warmup --jinja --chat-template-file /mnt/models/chat_template --log-colors --alias llama3.2 --ctx-size 2048 --temp 0.8 --cache-reuse 256 -v -ngl 999 --threads 5 --host 0.0.0.0
3f7fef751e9afafa4ac4e596fbb3b1f4512a3e70936b0dec239ae95dfa0e4671
(ins)🦭 > hey
2025-08-15 10:25:51 - DEBUG - Request: URL=http://127.0.0.1:8080/chat/completions, Data=b'{"stream": true, "messages": [{"role": "user", "content": "hey"}], "model": "llama3.2"}', Headers={'Content-Type': 'application/json'}
heyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyheyhey^C
(ins)🦭 >
2025-08-15 10:27:38 - DEBUG - run_cmd: podman inspect --format "{{ .Pod }}" ramalama_oHjihQZzdu
2025-08-15 10:27:38 - DEBUG - Working directory: None
2025-08-15 10:27:38 - DEBUG - Ignore stderr: True
2025-08-15 10:27:38 - DEBUG - Ignore all: False
2025-08-15 10:27:38 - DEBUG - Command finished with return code: 0
2025-08-15 10:27:38 - DEBUG - run_cmd: podman stop -t=0 --ignore True ramalama_oHjihQZzdu
2025-08-15 10:27:38 - DEBUG - Working directory: None
2025-08-15 10:27:38 - DEBUG - Ignore stderr: False
2025-08-15 10:27:38 - DEBUG - Ignore all: False
2025-08-15 10:27:39 - DEBUG - Command finished with return code: 0

On container side:

$ podman logs -f ramalama_oHjihQZzdu
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Virtio-GPU Venus (Apple M4) (venus) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memor
y: 32768 | int dot: 0 | matrix cores: none
common_chat_templates_init: failed to parse chat template (defaulting to chatml): Failed to parse number: '.' ([
json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - invalid lite
ral; last read: '.') at row 5, column 21:

{{ if .System }}{{ .System }}
                    ^
{{- end }}

build: 1 (3f4fc97) with cc (GCC) 15.1.1 20250521 (Red Hat 15.1.1-2) for aarch64-redhat-linux
system info: n_threads = 5, n_threads_batch = 5, total_threads = 5

system_info: n_threads = 5 (n_threads_batch = 5) / 5 | CPU : NEON = 1 | ARM_FMA = 1 | LLAMAFILE = 1 | OPENMP = 1
 | REPACK = 1 |

main: binding port with default address family
main: HTTP server is listening, hostname: 0.0.0.0, port: 8080, http threads: 4
main: loading model
srv    load_model: loading model '/mnt/models/llama3.2'
llama_model_load_from_file_impl: using device Vulkan0 (Virtio-GPU Venus (Apple M4)) - 24576 MiB free
llama_model_loader: loaded meta data with 30 key-value pairs and 255 tensors from /mnt/models/llama3.2 (version
GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Llama 3.2 3B Instruc
t
llama_model_loader: - kv   3:                           general.finetune str              = Instruct
llama_model_loader: - kv   4:                           general.basename str              = Llama-3.2
llama_model_loader: - kv   5:                         general.size_label str              = 3B
llama_model_loader: - kv   6:                               general.tags arr[str,6]       = ["facebook", "meta",
 "pytorch", "llam...
llama_model_loader: - kv   7:                          general.languages arr[str,8]       = ["en", "de", "fr", "
it", "pt", "hi", ...
llama_model_loader: - kv   8:                          llama.block_count u32              = 28
llama_model_loader: - kv   9:                       llama.context_length u32              = 131072
llama_model_loader: - kv  10:                     llama.embedding_length u32              = 3072
llama_model_loader: - kv  11:                  llama.feed_forward_length u32              = 8192
llama_model_loader: - kv  12:                 llama.attention.head_count u32              = 24
llama_model_loader: - kv  13:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  14:                       llama.rope.freq_base f32              = 500000.000000
llama_model_loader: - kv  15:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  16:                 llama.attention.key_length u32              = 128
llama_model_loader: - kv  17:               llama.attention.value_length u32              = 128
llama_model_loader: - kv  18:                          general.file_type u32              = 15
llama_model_loader: - kv  19:                           llama.vocab_size u32              = 128256
llama_model_loader: - kv  20:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  21:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  22:                         tokenizer.ggml.pre str              = llama-bpe
llama_model_loader: - kv  23:                      tokenizer.ggml.tokens arr[str,128256]  = ["!", "\"", "#", "$"
, "%", "&", "'", ...
llama_model_loader: - kv  24:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1
, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  25:                      tokenizer.ggml.merges arr[str,280147]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ
 ĠĠ", "...
llama_model_loader: - kv  26:                tokenizer.ggml.bos_token_id u32              = 128000
llama_model_loader: - kv  27:                tokenizer.ggml.eos_token_id u32              = 128009
llama_model_loader: - kv  28:                    tokenizer.chat_template str              = {{- bos_token }}\n{%
- if custom_tools ...
llama_model_loader: - kv  29:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   58 tensors
llama_model_loader: - type q4_K:  168 tensors
llama_model_loader: - type q6_K:   29 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 1.87 GiB (5.01 BPW)
init_tokenizer: initializing tokenizer for type 2
load: control token: 128254 '<|reserved_special_token_246|>' is not marked as EOG
load: control token: 128249 '<|reserved_special_token_241|>' is not marked as EOG
load: control token: 128246 '<|reserved_special_token_238|>' is not marked as EOG
load: control token: 128243 '<|reserved_special_token_235|>' is not marked as EOG
load: control token: 128242 '<|reserved_special_token_234|>' is not marked as EOG
load: control token: 128241 '<|reserved_special_token_233|>' is not marked as EOG
load: control token: 128240 '<|reserved_special_token_232|>' is not marked as EOG
load: control token: 128235 '<|reserved_special_token_227|>' is not marked as EOG
load: control token: 128231 '<|reserved_special_token_223|>' is not marked as EOG
load: control token: 128230 '<|reserved_special_token_222|>' is not marked as EOG
load: control token: 128228 '<|reserved_special_token_220|>' is not marked as EOG
load: control token: 128225 '<|reserved_special_token_217|>' is not marked as EOG
load: control token: 128218 '<|reserved_special_token_210|>' is not marked as EOG
load: control token: 128214 '<|reserved_special_token_206|>' is not marked as EOG
load: control token: 128213 '<|reserved_special_token_205|>' is not marked as EOG
load: control token: 128207 '<|reserved_special_token_199|>' is not marked as EOG
load: control token: 128206 '<|reserved_special_token_198|>' is not marked as EOG
load: control token: 128204 '<|reserved_special_token_196|>' is not marked as EOG
load: control token: 128200 '<|reserved_special_token_192|>' is not marked as EOG
load: control token: 128199 '<|reserved_special_token_191|>' is not marked as EOG
load: control token: 128198 '<|reserved_special_token_190|>' is not marked as EOG
load: control token: 128196 '<|reserved_special_token_188|>' is not marked as EOG
load: control token: 128194 '<|reserved_special_token_186|>' is not marked as EOG
load: control token: 128193 '<|reserved_special_token_185|>' is not marked as EOG
load: control token: 128188 '<|reserved_special_token_180|>' is not marked as EOG
load: control token: 128187 '<|reserved_special_token_179|>' is not marked as EOG
load: control token: 128185 '<|reserved_special_token_177|>' is not marked as EOG
load: control token: 128184 '<|reserved_special_token_176|>' is not marked as EOG
load: control token: 128180 '<|reserved_special_token_172|>' is not marked as EOG
load: control token: 128179 '<|reserved_special_token_171|>' is not marked as EOG
load: control token: 128178 '<|reserved_special_token_170|>' is not marked as EOG
load: control token: 128177 '<|reserved_special_token_169|>' is not marked as EOG
load: control token: 128176 '<|reserved_special_token_168|>' is not marked as EOG
load: control token: 128175 '<|reserved_special_token_167|>' is not marked as EOG
load: control token: 128171 '<|reserved_special_token_163|>' is not marked as EOG
load: control token: 128170 '<|reserved_special_token_162|>' is not marked as EOG
load: control token: 128169 '<|reserved_special_token_161|>' is not marked as EOG
load: control token: 128168 '<|reserved_special_token_160|>' is not marked as EOG
load: control token: 128165 '<|reserved_special_token_157|>' is not marked as EOG
load: control token: 128162 '<|reserved_special_token_154|>' is not marked as EOG
load: control token: 128158 '<|reserved_special_token_150|>' is not marked as EOG
load: control token: 128156 '<|reserved_special_token_148|>' is not marked as EOG
load: control token: 128155 '<|reserved_special_token_147|>' is not marked as EOG
load: control token: 128154 '<|reserved_special_token_146|>' is not marked as EOG
load: control token: 128151 '<|reserved_special_token_143|>' is not marked as EOG
load: control token: 128149 '<|reserved_special_token_141|>' is not marked as EOG
load: control token: 128147 '<|reserved_special_token_139|>' is not marked as EOG
load: control token: 128146 '<|reserved_special_token_138|>' is not marked as EOG
load: control token: 128144 '<|reserved_special_token_136|>' is not marked as EOG
load: control token: 128142 '<|reserved_special_token_134|>' is not marked as EOG
load: control token: 128141 '<|reserved_special_token_133|>' is not marked as EOG
load: control token: 128138 '<|reserved_special_token_130|>' is not marked as EOG
load: control token: 128136 '<|reserved_special_token_128|>' is not marked as EOG
load: control token: 128135 '<|reserved_special_token_127|>' is not marked as EOG
load: control token: 128134 '<|reserved_special_token_126|>' is not marked as EOG
load: control token: 128133 '<|reserved_special_token_125|>' is not marked as EOG
load: control token: 128131 '<|reserved_special_token_123|>' is not marked as EOG
load: control token: 128128 '<|reserved_special_token_120|>' is not marked as EOG
load: control token: 128124 '<|reserved_special_token_116|>' is not marked as EOG
load: control token: 128123 '<|reserved_special_token_115|>' is not marked as EOG
load: control token: 128122 '<|reserved_special_token_114|>' is not marked as EOG
load: control token: 128119 '<|reserved_special_token_111|>' is not marked as EOG
load: control token: 128115 '<|reserved_special_token_107|>' is not marked as EOG
load: control token: 128112 '<|reserved_special_token_104|>' is not marked as EOG
load: control token: 128110 '<|reserved_special_token_102|>' is not marked as EOG
load: control token: 128109 '<|reserved_special_token_101|>' is not marked as EOG
load: control token: 128108 '<|reserved_special_token_100|>' is not marked as EOG
load: control token: 128106 '<|reserved_special_token_98|>' is not marked as EOG
load: control token: 128103 '<|reserved_special_token_95|>' is not marked as EOG
load: control token: 128102 '<|reserved_special_token_94|>' is not marked as EOG
load: control token: 128101 '<|reserved_special_token_93|>' is not marked as EOG
load: control token: 128097 '<|reserved_special_token_89|>' is not marked as EOG
load: control token: 128091 '<|reserved_special_token_83|>' is not marked as EOG
load: control token: 128090 '<|reserved_special_token_82|>' is not marked as EOG
load: control token: 128089 '<|reserved_special_token_81|>' is not marked as EOG
load: control token: 128087 '<|reserved_special_token_79|>' is not marked as EOG
load: control token: 128085 '<|reserved_special_token_77|>' is not marked as EOG
load: control token: 128081 '<|reserved_special_token_73|>' is not marked as EOG
load: control token: 128078 '<|reserved_special_token_70|>' is not marked as EOG
load: control token: 128076 '<|reserved_special_token_68|>' is not marked as EOG
load: control token: 128075 '<|reserved_special_token_67|>' is not marked as EOG
load: control token: 128073 '<|reserved_special_token_65|>' is not marked as EOG
load: control token: 128068 '<|reserved_special_token_60|>' is not marked as EOG
load: control token: 128067 '<|reserved_special_token_59|>' is not marked as EOG
load: control token: 128065 '<|reserved_special_token_57|>' is not marked as EOG
load: control token: 128063 '<|reserved_special_token_55|>' is not marked as EOG
load: control token: 128062 '<|reserved_special_token_54|>' is not marked as EOG
load: control token: 128060 '<|reserved_special_token_52|>' is not marked as EOG
load: control token: 128059 '<|reserved_special_token_51|>' is not marked as EOG
load: control token: 128057 '<|reserved_special_token_49|>' is not marked as EOG
load: control token: 128054 '<|reserved_special_token_46|>' is not marked as EOG
load: control token: 128046 '<|reserved_special_token_38|>' is not marked as EOG
load: control token: 128045 '<|reserved_special_token_37|>' is not marked as EOG
load: control token: 128044 '<|reserved_special_token_36|>' is not marked as EOG
load: control token: 128043 '<|reserved_special_token_35|>' is not marked as EOG
load: control token: 128038 '<|reserved_special_token_30|>' is not marked as EOG
load: control token: 128036 '<|reserved_special_token_28|>' is not marked as EOG
load: control token: 128035 '<|reserved_special_token_27|>' is not marked as EOG
load: control token: 128032 '<|reserved_special_token_24|>' is not marked as EOG
load: control token: 128028 '<|reserved_special_token_20|>' is not marked as EOG
load: control token: 128027 '<|reserved_special_token_19|>' is not marked as EOG
load: control token: 128024 '<|reserved_special_token_16|>' is not marked as EOG
load: control token: 128023 '<|reserved_special_token_15|>' is not marked as EOG
load: control token: 128022 '<|reserved_special_token_14|>' is not marked as EOG
load: control token: 128021 '<|reserved_special_token_13|>' is not marked as EOG
load: control token: 128018 '<|reserved_special_token_10|>' is not marked as EOG
load: control token: 128016 '<|reserved_special_token_8|>' is not marked as EOG
load: control token: 128015 '<|reserved_special_token_7|>' is not marked as EOG
load: control token: 128013 '<|reserved_special_token_5|>' is not marked as EOG
load: control token: 128011 '<|reserved_special_token_3|>' is not marked as EOG
load: control token: 128005 '<|reserved_special_token_2|>' is not marked as EOG
load: control token: 128004 '<|finetune_right_pad_id|>' is not marked as EOG
load: control token: 128002 '<|reserved_special_token_0|>' is not marked as EOG
load: control token: 128252 '<|reserved_special_token_244|>' is not marked as EOG
load: control token: 128190 '<|reserved_special_token_182|>' is not marked as EOG
load: control token: 128183 '<|reserved_special_token_175|>' is not marked as EOG
load: control token: 128137 '<|reserved_special_token_129|>' is not marked as EOG
load: control token: 128182 '<|reserved_special_token_174|>' is not marked as EOG
load: control token: 128040 '<|reserved_special_token_32|>' is not marked as EOG
load: control token: 128048 '<|reserved_special_token_40|>' is not marked as EOG
load: control token: 128092 '<|reserved_special_token_84|>' is not marked as EOG
load: control token: 128215 '<|reserved_special_token_207|>' is not marked as EOG
load: control token: 128107 '<|reserved_special_token_99|>' is not marked as EOG
load: control token: 128208 '<|reserved_special_token_200|>' is not marked as EOG
load: control token: 128145 '<|reserved_special_token_137|>' is not marked as EOG
load: control token: 128031 '<|reserved_special_token_23|>' is not marked as EOG
load: control token: 128129 '<|reserved_special_token_121|>' is not marked as EOG
load: control token: 128201 '<|reserved_special_token_193|>' is not marked as EOG
load: control token: 128074 '<|reserved_special_token_66|>' is not marked as EOG
load: control token: 128095 '<|reserved_special_token_87|>' is not marked as EOG
load: control token: 128186 '<|reserved_special_token_178|>' is not marked as EOG
load: control token: 128143 '<|reserved_special_token_135|>' is not marked as EOG
load: control token: 128229 '<|reserved_special_token_221|>' is not marked as EOG
load: control token: 128007 '<|end_header_id|>' is not marked as EOG
load: control token: 128055 '<|reserved_special_token_47|>' is not marked as EOG
load: control token: 128056 '<|reserved_special_token_48|>' is not marked as EOG
load: control token: 128061 '<|reserved_special_token_53|>' is not marked as EOG
load: control token: 128153 '<|reserved_special_token_145|>' is not marked as EOG
load: control token: 128152 '<|reserved_special_token_144|>' is not marked as EOG
load: control token: 128212 '<|reserved_special_token_204|>' is not marked as EOG
load: control token: 128172 '<|reserved_special_token_164|>' is not marked as EOG
load: control token: 128160 '<|reserved_special_token_152|>' is not marked as EOG
load: control token: 128041 '<|reserved_special_token_33|>' is not marked as EOG
load: control token: 128181 '<|reserved_special_token_173|>' is not marked as EOG
load: control token: 128094 '<|reserved_special_token_86|>' is not marked as EOG
load: control token: 128118 '<|reserved_special_token_110|>' is not marked as EOG
load: control token: 128236 '<|reserved_special_token_228|>' is not marked as EOG
load: control token: 128148 '<|reserved_special_token_140|>' is not marked as EOG
load: control token: 128042 '<|reserved_special_token_34|>' is not marked as EOG
load: control token: 128139 '<|reserved_special_token_131|>' is not marked as EOG
load: control token: 128173 '<|reserved_special_token_165|>' is not marked as EOG
load: control token: 128239 '<|reserved_special_token_231|>' is not marked as EOG
load: control token: 128157 '<|reserved_special_token_149|>' is not marked as EOG
load: control token: 128052 '<|reserved_special_token_44|>' is not marked as EOG
load: control token: 128026 '<|reserved_special_token_18|>' is not marked as EOG
load: control token: 128003 '<|reserved_special_token_1|>' is not marked as EOG
load: control token: 128019 '<|reserved_special_token_11|>' is not marked as EOG
load: control token: 128116 '<|reserved_special_token_108|>' is not marked as EOG
load: control token: 128161 '<|reserved_special_token_153|>' is not marked as EOG
load: control token: 128226 '<|reserved_special_token_218|>' is not marked as EOG
load: control token: 128159 '<|reserved_special_token_151|>' is not marked as EOG
load: control token: 128012 '<|reserved_special_token_4|>' is not marked as EOG
load: control token: 128088 '<|reserved_special_token_80|>' is not marked as EOG
load: control token: 128163 '<|reserved_special_token_155|>' is not marked as EOG
load: control token: 128113 '<|reserved_special_token_105|>' is not marked as EOG
load: control token: 128250 '<|reserved_special_token_242|>' is not marked as EOG
load: control token: 128125 '<|reserved_special_token_117|>' is not marked as EOG
load: control token: 128053 '<|reserved_special_token_45|>' is not marked as EOG
load: control token: 128224 '<|reserved_special_token_216|>' is not marked as EOG
load: control token: 128247 '<|reserved_special_token_239|>' is not marked as EOG
load: control token: 128251 '<|reserved_special_token_243|>' is not marked as EOG
load: control token: 128216 '<|reserved_special_token_208|>' is not marked as EOG
load: control token: 128006 '<|start_header_id|>' is not marked as EOG
load: control token: 128211 '<|reserved_special_token_203|>' is not marked as EOG
load: control token: 128077 '<|reserved_special_token_69|>' is not marked as EOG
load: control token: 128237 '<|reserved_special_token_229|>' is not marked as EOG
load: control token: 128086 '<|reserved_special_token_78|>' is not marked as EOG
load: control token: 128227 '<|reserved_special_token_219|>' is not marked as EOG
load: control token: 128058 '<|reserved_special_token_50|>' is not marked as EOG
load: control token: 128100 '<|reserved_special_token_92|>' is not marked as EOG
load: control token: 128209 '<|reserved_special_token_201|>' is not marked as EOG
load: control token: 128084 '<|reserved_special_token_76|>' is not marked as EOG
load: control token: 128071 '<|reserved_special_token_63|>' is not marked as EOG
load: control token: 128070 '<|reserved_special_token_62|>' is not marked as EOG
load: control token: 128049 '<|reserved_special_token_41|>' is not marked as EOG
load: control token: 128197 '<|reserved_special_token_189|>' is not marked as EOG
load: control token: 128072 '<|reserved_special_token_64|>' is not marked as EOG
load: control token: 128000 '<|begin_of_text|>' is not marked as EOG
load: control token: 128223 '<|reserved_special_token_215|>' is not marked as EOG
load: control token: 128217 '<|reserved_special_token_209|>' is not marked as EOG
load: control token: 128111 '<|reserved_special_token_103|>' is not marked as EOG
load: control token: 128203 '<|reserved_special_token_195|>' is not marked as EOG
load: control token: 128051 '<|reserved_special_token_43|>' is not marked as EOG
load: control token: 128030 '<|reserved_special_token_22|>' is not marked as EOG
load: control token: 128117 '<|reserved_special_token_109|>' is not marked as EOG
load: control token: 128010 '<|python_tag|>' is not marked as EOG
load: control token: 128238 '<|reserved_special_token_230|>' is not marked as EOG
load: control token: 128255 '<|reserved_special_token_247|>' is not marked as EOG
load: control token: 128202 '<|reserved_special_token_194|>' is not marked as EOG
load: control token: 128132 '<|reserved_special_token_124|>' is not marked as EOG
load: control token: 128248 '<|reserved_special_token_240|>' is not marked as EOG
load: control token: 128167 '<|reserved_special_token_159|>' is not marked as EOG
load: control token: 128127 '<|reserved_special_token_119|>' is not marked as EOG
load: control token: 128105 '<|reserved_special_token_97|>' is not marked as EOG
load: control token: 128039 '<|reserved_special_token_31|>' is not marked as EOG
load: control token: 128232 '<|reserved_special_token_224|>' is not marked as EOG
load: control token: 128166 '<|reserved_special_token_158|>' is not marked as EOG
load: control token: 128130 '<|reserved_special_token_122|>' is not marked as EOG
load: control token: 128114 '<|reserved_special_token_106|>' is not marked as EOG
load: control token: 128234 '<|reserved_special_token_226|>' is not marked as EOG
load: control token: 128191 '<|reserved_special_token_183|>' is not marked as EOG
load: control token: 128064 '<|reserved_special_token_56|>' is not marked as EOG
load: control token: 128140 '<|reserved_special_token_132|>' is not marked as EOG
load: control token: 128096 '<|reserved_special_token_88|>' is not marked as EOG
load: control token: 128098 '<|reserved_special_token_90|>' is not marked as EOG
load: control token: 128192 '<|reserved_special_token_184|>' is not marked as EOG
load: control token: 128093 '<|reserved_special_token_85|>' is not marked as EOG
load: control token: 128150 '<|reserved_special_token_142|>' is not marked as EOG
load: control token: 128222 '<|reserved_special_token_214|>' is not marked as EOG
load: control token: 128233 '<|reserved_special_token_225|>' is not marked as EOG
load: control token: 128220 '<|reserved_special_token_212|>' is not marked as EOG
load: control token: 128034 '<|reserved_special_token_26|>' is not marked as EOG
load: control token: 128033 '<|reserved_special_token_25|>' is not marked as EOG
load: control token: 128253 '<|reserved_special_token_245|>' is not marked as EOG
load: control token: 128195 '<|reserved_special_token_187|>' is not marked as EOG
load: control token: 128099 '<|reserved_special_token_91|>' is not marked as EOG
load: control token: 128189 '<|reserved_special_token_181|>' is not marked as EOG
load: control token: 128210 '<|reserved_special_token_202|>' is not marked as EOG
load: control token: 128174 '<|reserved_special_token_166|>' is not marked as EOG
load: control token: 128083 '<|reserved_special_token_75|>' is not marked as EOG
load: control token: 128080 '<|reserved_special_token_72|>' is not marked as EOG
load: control token: 128104 '<|reserved_special_token_96|>' is not marked as EOG
load: control token: 128082 '<|reserved_special_token_74|>' is not marked as EOG
load: control token: 128219 '<|reserved_special_token_211|>' is not marked as EOG
load: control token: 128017 '<|reserved_special_token_9|>' is not marked as EOG
load: control token: 128050 '<|reserved_special_token_42|>' is not marked as EOG
load: control token: 128205 '<|reserved_special_token_197|>' is not marked as EOG
load: control token: 128047 '<|reserved_special_token_39|>' is not marked as EOG
load: control token: 128164 '<|reserved_special_token_156|>' is not marked as EOG
load: control token: 128020 '<|reserved_special_token_12|>' is not marked as EOG
load: control token: 128069 '<|reserved_special_token_61|>' is not marked as EOG
load: control token: 128245 '<|reserved_special_token_237|>' is not marked as EOG
load: control token: 128121 '<|reserved_special_token_113|>' is not marked as EOG
load: control token: 128079 '<|reserved_special_token_71|>' is not marked as EOG
load: control token: 128037 '<|reserved_special_token_29|>' is not marked as EOG
load: control token: 128244 '<|reserved_special_token_236|>' is not marked as EOG
load: control token: 128029 '<|reserved_special_token_21|>' is not marked as EOG
load: control token: 128221 '<|reserved_special_token_213|>' is not marked as EOG
load: control token: 128066 '<|reserved_special_token_58|>' is not marked as EOG
load: control token: 128120 '<|reserved_special_token_112|>' is not marked as EOG
load: control token: 128014 '<|reserved_special_token_6|>' is not marked as EOG
load: control token: 128025 '<|reserved_special_token_17|>' is not marked as EOG
load: control token: 128126 '<|reserved_special_token_118|>' is not marked as EOG
load: special tokens cache size = 256
load: token to piece cache size = 0.7999 MB
print_info: arch             = llama
print_info: vocab_only       = 0
print_info: n_ctx_train      = 131072
print_info: n_embd           = 3072
print_info: n_layer          = 28
print_info: n_head           = 24
print_info: n_head_kv        = 8
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 3
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 8192
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 0
print_info: rope scaling     = linear
print_info: freq_base_train  = 500000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 131072
print_info: rope_finetuned   = unknown
print_info: model type       = 3B
print_info: model params     = 3.21 B
print_info: general.name     = Llama 3.2 3B Instruct
print_info: vocab type       = BPE
print_info: n_vocab          = 128256
print_info: n_merges         = 280147
print_info: BOS token        = 128000 '<|begin_of_text|>'
print_info: EOS token        = 128009 '<|eot_id|>'
print_info: EOT token        = 128009 '<|eot_id|>'
print_info: EOM token        = 128008 '<|eom_id|>'
print_info: LF token         = 198 'Ċ'
print_info: EOG token        = 128001 '<|end_of_text|>'
print_info: EOG token        = 128008 '<|eom_id|>'
print_info: EOG token        = 128009 '<|eot_id|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: layer   0 assigned to device Vulkan0, is_swa = 0
load_tensors: layer   1 assigned to device Vulkan0, is_swa = 0
load_tensors: layer   2 assigned to device Vulkan0, is_swa = 0
load_tensors: layer   3 assigned to device Vulkan0, is_swa = 0
load_tensors: layer   4 assigned to device Vulkan0, is_swa = 0
load_tensors: layer   5 assigned to device Vulkan0, is_swa = 0
load_tensors: layer   6 assigned to device Vulkan0, is_swa = 0
load_tensors: layer   7 assigned to device Vulkan0, is_swa = 0
load_tensors: layer   8 assigned to device Vulkan0, is_swa = 0
load_tensors: layer   9 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  10 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  11 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  12 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  13 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  14 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  15 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  16 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  17 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  18 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  19 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  20 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  21 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  22 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  23 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  24 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  25 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  26 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  27 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  28 assigned to device Vulkan0, is_swa = 0
load_tensors: tensor 'token_embd.weight' (q6_K) (and 0 others) cannot be used with preferred buffer type Vulkan_
Host, using CPU instead
load_tensors: offloading 28 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 29/29 layers to GPU
load_tensors:      Vulkan0 model buffer size =  1918.35 MiB
load_tensors:   CPU_Mapped model buffer size =   308.23 MiB
...........................................................................
llama_context: constructing llama_context
llama_context: non-unified KV cache requires ggml_set_rows() - forcing unified KV cache
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 2048
llama_context: n_ctx_per_seq = 2048
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 0
llama_context: kv_unified    = true
llama_context: freq_base     = 500000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (2048) < n_ctx_train (131072) -- the full capacity of the model will not be utilize
d
set_abort_callback: call
llama_context: Vulkan_Host  output buffer size =     0.49 MiB
create_memory: n_ctx = 2048 (padded)
llama_kv_cache_unified: layer   0: dev = Vulkan0
llama_kv_cache_unified: layer   1: dev = Vulkan0
llama_kv_cache_unified: layer   2: dev = Vulkan0
llama_kv_cache_unified: layer   3: dev = Vulkan0
llama_kv_cache_unified: layer   4: dev = Vulkan0
llama_kv_cache_unified: layer   5: dev = Vulkan0
llama_kv_cache_unified: layer   6: dev = Vulkan0
llama_kv_cache_unified: layer   7: dev = Vulkan0
llama_kv_cache_unified: layer   8: dev = Vulkan0
llama_kv_cache_unified: layer   9: dev = Vulkan0
llama_kv_cache_unified: layer  10: dev = Vulkan0
llama_kv_cache_unified: layer  11: dev = Vulkan0
llama_kv_cache_unified: layer  12: dev = Vulkan0
llama_kv_cache_unified: layer  13: dev = Vulkan0
llama_kv_cache_unified: layer  14: dev = Vulkan0
llama_kv_cache_unified: layer  15: dev = Vulkan0
llama_kv_cache_unified: layer  16: dev = Vulkan0
llama_kv_cache_unified: layer  17: dev = Vulkan0
llama_kv_cache_unified: layer  18: dev = Vulkan0
llama_kv_cache_unified: layer  19: dev = Vulkan0
llama_kv_cache_unified: layer  20: dev = Vulkan0
llama_kv_cache_unified: layer  21: dev = Vulkan0
llama_kv_cache_unified: layer  22: dev = Vulkan0
llama_kv_cache_unified: layer  23: dev = Vulkan0
llama_kv_cache_unified: layer  24: dev = Vulkan0
llama_kv_cache_unified: layer  25: dev = Vulkan0
llama_kv_cache_unified: layer  26: dev = Vulkan0
llama_kv_cache_unified: layer  27: dev = Vulkan0
llama_kv_cache_unified:    Vulkan0 KV buffer size =   224.00 MiB
llama_kv_cache_unified: size =  224.00 MiB (  2048 cells,  28 layers,  1/ 1 seqs), K (f16):  112.00 MiB, V (f16)
:  112.00 MiB
llama_kv_cache_unified: LLAMA_SET_ROWS=0, using old ggml_cpy() method for backwards compatibility
llama_context: enumerating backends
llama_context: backend_ptrs.size() = 2
llama_context: max_nodes = 2048
llama_context: worst-case: n_tokens = 512, n_seqs = 1, n_outputs = 0
graph_reserve: reserving a graph for ubatch with n_tokens =  512, n_seqs =  1, n_outputs =  512
graph_reserve: reserving a graph for ubatch with n_tokens =    1, n_seqs =  1, n_outputs =    1
graph_reserve: reserving a graph for ubatch with n_tokens =  512, n_seqs =  1, n_outputs =  512
llama_context:    Vulkan0 compute buffer size =   278.50 MiB
llama_context: Vulkan_Host compute buffer size =    10.01 MiB
llama_context: graph nodes  = 1042
llama_context: graph splits = 2
clear_adapter_lora: call
common_init_from_params: added <|end_of_text|> logit bias = -inf
common_init_from_params: added <|eom_id|> logit bias = -inf
common_init_from_params: added <|eot_id|> logit bias = -inf
common_init_from_params: setting dry_penalty_last_n to ctx_size = 2048
common_chat_templates_init: failed to parse chat template (defaulting to chatml): Failed to parse number: '.' ([
json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - invalid lite
ral; last read: '.') at row 5, column 21:

{{ if .System }}{{ .System }}
                    ^
{{- end }}

srv          init: initializing slots, n_slots = 1
slot         init: id  0 | task -1 | new slot n_ctx_slot = 2048
slot        reset: id  0 | task -1 |
main: model loaded
main: chat template, chat_template: {%- for message in messages -%}
  {{- '<|im_start|>' + message.role + '
' + message.content + '<|im_end|>
' -}}
{%- endfor -%}
{%- if add_generation_prompt -%}
  {{- '<|im_start|>assistant
' -}}
{%- endif -%}, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'
main: server is listening on http://0.0.0.0:8080 - starting the main loop
que    start_loop: processing new tasks
que    start_loop: update slots
srv  update_slots: all slots are idle
srv  kv_cache_cle: clearing KV cache
que    start_loop: waiting for new tasks
request: {"stream": true, "messages": [{"role": "user", "content": "hey"}], "model": "llama3.2"}
srv  params_from_: Grammar:
srv  params_from_: Grammar lazy: false
srv  params_from_: Chat format: Content-only
srv  add_waiting_: add task 0 to waiting list. current waiting = 0 (before add)
que          post: new task, id = 0/1, front = 0
que    start_loop: processing new tasks
que    start_loop: processing task, id = 0
slot get_availabl: id  0 | task -1 | selected slot by lru, t_last = -1
slot        reset: id  0 | task -1 |
slot launch_slot_: id  0 | task 0 | launching slot : {"id":0,"id_task":0,"n_ctx":2048,"speculative":false,"is_pr
ocessing":false,"params":{"n_predict":-1,"seed":4294967295,"temperature":0.800000011920929,"dynatemp_range":0.0,
"dynatemp_exponent":1.0,"top_k":40,"top_p":0.949999988079071,"min_p":0.05000000074505806,"top_n_sigma":-1.0,"xtc
_probability":0.0,"xtc_threshold":0.10000000149011612,"typical_p":1.0,"repeat_last_n":64,"repeat_penalty":1.0,"p
resence_penalty":0.0,"frequency_penalty":0.0,"dry_multiplier":0.0,"dry_base":1.75,"dry_allowed_length":2,"dry_pe
nalty_last_n":2048,"dry_sequence_breakers":["\n",":","\"","*"],"mirostat":0,"mirostat_tau":5.0,"mirostat_eta":0.
10000000149011612,"stop":[],"max_tokens":-1,"n_keep":0,"n_discard":0,"ignore_eos":false,"stream":true,"logit_bia
s":[],"n_probs":0,"min_keep":0,"grammar":"","grammar_lazy":false,"grammar_triggers":[],"preserved_tokens":[],"ch
at_format":"Content-only","reasoning_format":"deepseek","reasoning_in_content":false,"thinking_forced_open":fals
e,"samplers":["penalties","dry","top_n_sigma","top_k","typ_p","top_p","min_p","xtc","temperature"],"speculative.
n_max":16,"speculative.n_min":0,"speculative.p_min":0.75,"timings_per_token":false,"post_sampling_probs":false,"
lora":[]},"prompt":"<|begin_of_text|><|im_start|>user\nhey<|im_end|>\n<|im_start|>assistant\n","next_token":{"ha
s_next_token":true,"has_new_line":false,"n_remain":-1,"n_decoded":0,"stopping_word":""}}
slot launch_slot_: id  0 | task 0 | processing task
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 1, front = 0
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 2048, n_keep = 0, n_prompt_tokens = 24
slot update_slots: id  0 | task 0 | trying to reuse chunks with size > 256, slot.n_past = 0
slot update_slots: id  0 | task 0 | after context reuse, new slot.n_past = 0
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 24, n_tokens = 24, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 24, n_tokens = 24
srv  update_slots: decoding batch, n_tokens = 24
clear_adapter_lora: call
set_embeddings: value = 0
srv  update_chat_: Parsing chat message: hey
Parsing input with format Content-only: hey
Parsed message: {"role":"assistant","content":"hey"}
srv          send: sending result for task id = 0
srv          send: task id = 0 pushed to result queue
slot process_toke: id  0 | task 0 | n_decoded = 1, n_remaining = -1, next token: 36661 'hey'
srv  update_slots: run slots completed
que    start_loop: waiting for new tasks
que    start_loop: processing new tasks
que    start_loop: processing task, id = 1
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 2, front = 0
slot update_slots: id  0 | task 0 | slot decode token, n_ctx = 2048, n_past = 25, n_cache_tokens = 25, truncated
 = 0
srv  update_slots: decoding batch, n_tokens = 1
clear_adapter_lora: call
set_embeddings: value = 0
data stream, to_send: data: {"choices":[{"finish_reason":null,"index":0,"delta":{"role":"assistant","content":nu
ll}}],"created":1755235087,"id":"chatcmpl-hq5LBE4p1hGdKTSZkgLevUFoDgPdYPx4","model":"llama3.2","system_fingerpri
nt":"b1-3f4fc97","object":"chat.completion.chunk"}

data stream, to_send: data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"hey"}}],"created":17
55235087,"id":"chatcmpl-hq5LBE4p1hGdKTSZkgLevUFoDgPdYPx4","model":"llama3.2","system_fingerprint":"b1-3f4fc97","
object":"chat.completion.chunk"}

srv  update_chat_: Parsing chat message: heyhey
Parsing input with format Content-only: heyhey
Parsed message: {"role":"assistant","content":"heyhey"}
srv          send: sending result for task id = 0
srv          send: task id = 0 pushed to result queue
slot process_toke: id  0 | task 0 | n_decoded = 2, n_remaining = -1, next token: 36661 'hey'
srv  update_slots: run slots completed
que    start_loop: waiting for new tasks
que    start_loop: processing new tasks
que    start_loop: processing task, id = 2
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 3, front = 0
slot update_slots: id  0 | task 0 | slot decode token, n_ctx = 2048, n_past = 26, n_cache_tokens = 26, truncated
 = 0
srv  update_slots: decoding batch, n_tokens = 1
clear_adapter_lora: call
set_embeddings: value = 0
data stream, to_send: data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"hey"}}],"created":17
55235087,"id":"chatcmpl-hq5LBE4p1hGdKTSZkgLevUFoDgPdYPx4","model":"llama3.2","system_fingerprint":"b1-3f4fc97","
object":"chat.completion.chunk"}

srv  update_chat_: Parsing chat message: heyheyhey
Parsing input with format Content-only: heyheyhey
Parsed message: {"role":"assistant","content":"heyheyhey"}
srv          send: sending result for task id = 0
srv          send: task id = 0 pushed to result queue
slot process_toke: id  0 | task 0 | n_decoded = 3, n_remaining = -1, next token: 36661 'hey'
srv  update_slots: run slots completed
que    start_loop: waiting for new tasks
que    start_loop: processing new tasks
que    start_loop: processing task, id = 3
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 4, front = 0
slot update_slots: id  0 | task 0 | slot decode token, n_ctx = 2048, n_past = 27, n_cache_tokens = 27, truncated
 = 0
srv  update_slots: decoding batch, n_tokens = 1
clear_adapter_lora: call
set_embeddings: value = 0
data stream, to_send: data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"hey"}}],"created":17
55235087,"id":"chatcmpl-hq5LBE4p1hGdKTSZkgLevUFoDgPdYPx4","model":"llama3.2","system_fingerprint":"b1-3f4fc97","
object":"chat.completion.chunk"}

srv  update_chat_: Parsing chat message: heyheyheyhey
Parsing input with format Content-only: heyheyheyhey
Parsed message: {"role":"assistant","content":"heyheyheyhey"}
srv          send: sending result for task id = 0
srv          send: task id = 0 pushed to result queue
slot process_toke: id  0 | task 0 | n_decoded = 4, n_remaining = -1, next token: 36661 'hey'
srv  update_slots: run slots completed
que    start_loop: waiting for new tasks
que    start_loop: processing new tasks
que    start_loop: processing task, id = 4
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 5, front = 0
slot update_slots: id  0 | task 0 | slot decode token, n_ctx = 2048, n_past = 28, n_cache_tokens = 28, truncated
 = 0
srv  update_slots: decoding batch, n_tokens = 1
clear_adapter_lora: call
set_embeddings: value = 0
data stream, to_send: data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"hey"}}],"created":17
55235087,"id":"chatcmpl-hq5LBE4p1hGdKTSZkgLevUFoDgPdYPx4","model":"llama3.2","system_fingerprint":"b1-3f4fc97","
object":"chat.completion.chunk"}

srv  update_chat_: Parsing chat message: heyheyheyheyhey
Parsing input with format Content-only: heyheyheyheyhey
Parsed message: {"role":"assistant","content":"heyheyheyheyhey"}
srv          send: sending result for task id = 0
srv          send: task id = 0 pushed to result queue
slot process_toke: id  0 | task 0 | n_decoded = 5, n_remaining = -1, next token: 36661 'hey'
srv  update_slots: run slots completed
que    start_loop: waiting for new tasks
que    start_loop: processing new tasks
que    start_loop: processing task, id = 5
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 6, front = 0
slot update_slots: id  0 | task 0 | slot decode token, n_ctx = 2048, n_past = 29, n_cache_tokens = 29, truncated
 = 0
srv  update_slots: decoding batch, n_tokens = 1
clear_adapter_lora: call
set_embeddings: value = 0
data stream, to_send: data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"hey"}}],"created":17
55235088,"id":"chatcmpl-hq5LBE4p1hGdKTSZkgLevUFoDgPdYPx4","model":"llama3.2","system_fingerprint":"b1-3f4fc97","
object":"chat.completion.chunk"}

srv  update_chat_: Parsing chat message: heyheyheyheyheyhey
Parsing input with format Content-only: heyheyheyheyheyhey
Parsed message: {"role":"assistant","content":"heyheyheyheyheyhey"}
srv          send: sending result for task id = 0
srv          send: task id = 0 pushed to result queue
slot process_toke: id  0 | task 0 | n_decoded = 6, n_remaining = -1, next token: 36661 'hey'
srv  update_slots: run slots completed
que    start_loop: waiting for new tasks
que    start_loop: processing new tasks
que    start_loop: processing task, id = 6
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 7, front = 0
slot update_slots: id  0 | task 0 | slot decode token, n_ctx = 2048, n_past = 30, n_cache_tokens = 30, truncated
 = 0
srv  update_slots: decoding batch, n_tokens = 1
clear_adapter_lora: call
set_embeddings: value = 0
data stream, to_send: data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"hey"}}],"created":17
55235088,"id":"chatcmpl-hq5LBE4p1hGdKTSZkgLevUFoDgPdYPx4","model":"llama3.2","system_fingerprint":"b1-3f4fc97","
object":"chat.completion.chunk"}

srv  update_chat_: Parsing chat message: heyheyheyheyheyheyhey
Parsing input with format Content-only: heyheyheyheyheyheyhey
Parsed message: {"role":"assistant","content":"heyheyheyheyheyheyhey"}
srv          send: sending result for task id = 0
srv          send: task id = 0 pushed to result queue
slot process_toke: id  0 | task 0 | n_decoded = 7, n_remaining = -1, next token: 36661 'hey'
srv  update_slots: run slots completed
que    start_loop: waiting for new tasks
que    start_loop: processing new tasks
que    start_loop: processing task, id = 7
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 8, front = 0
slot update_slots: id  0 | task 0 | slot decode token, n_ctx = 2048, n_past = 31, n_cache_tokens = 31, truncated
 = 0
srv  update_slots: decoding batch, n_tokens = 1
clear_adapter_lora: call
set_embeddings: value = 0
data stream, to_send: data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"hey"}}],"created":17
55235088,"id":"chatcmpl-hq5LBE4p1hGdKTSZkgLevUFoDgPdYPx4","model":"llama3.2","system_fingerprint":"b1-3f4fc97","
object":"chat.completion.chunk"}

srv  update_chat_: Parsing chat message: heyheyheyheyheyheyheyhey
Parsing input with format Content-only: heyheyheyheyheyheyheyhey
Parsed message: {"role":"assistant","content":"heyheyheyheyheyheyheyhey"}
srv          send: sending result for task id = 0
srv          send: task id = 0 pushed to result queue
slot process_toke: id  0 | task 0 | n_decoded = 8, n_remaining = -1, next token: 36661 'hey'
srv  update_slots: run slots completed
que    start_loop: waiting for new tasks
que    start_loop: processing new tasks
que    start_loop: processing task, id = 8
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 9, front = 0
slot update_slots: id  0 | task 0 | slot decode token, n_ctx = 2048, n_past = 32, n_cache_tokens = 32, truncated
 = 0
srv  update_slots: decoding batch, n_tokens = 1
clear_adapter_lora: call
set_embeddings: value = 0
data stream, to_send: data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"hey"}}],"created":17
55235088,"id":"chatcmpl-hq5LBE4p1hGdKTSZkgLevUFoDgPdYPx4","model":"llama3.2","system_fingerprint":"b1-3f4fc97","
object":"chat.completion.chunk"}

srv  update_chat_: Parsing chat message: heyheyheyheyheyheyheyheyhey
Parsing input with format Content-only: heyheyheyheyheyheyheyheyhey
Parsed message: {"role":"assistant","content":"heyheyheyheyheyheyheyheyhey"}
srv          send: sending result for task id = 0
srv          send: task id = 0 pushed to result queue
slot process_toke: id  0 | task 0 | n_decoded = 9, n_remaining = -1, next token: 36661 'hey'
srv  update_slots: run slots completed
que    start_loop: waiting for new tasks
que    start_loop: processing new tasks
que    start_loop: processing task, id = 9
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 10, front = 0
slot update_slots: id  0 | task 0 | slot decode token, n_ctx = 2048, n_past = 33, n_cache_tokens = 33, truncated
 = 0
srv  update_slots: decoding batch, n_tokens = 1
clear_adapter_lora: call
set_embeddings: value = 0
data stream, to_send: data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"hey"}}],"created":17
55235088,"id":"chatcmpl-hq5LBE4p1hGdKTSZkgLevUFoDgPdYPx4","model":"llama3.2","system_fingerprint":"b1-3f4fc97","
object":"chat.completion.chunk"}

[...snip...]

srv          send: sending result for task id = 0
slot process_toke: id  0 | task 0 | n_decoded = 548, n_remaining = -1, next token: 36661 'hey'
srv  update_slots: run slots completed
que    start_loop: waiting for new tasks
que    start_loop: processing new tasks
que    start_loop: processing task, id = 549
slot      release: id  0 | task 0 | stop processing: n_past = 571, truncated = 0
que    start_loop: processing task, id = 548
que    start_loop: update slots
srv  update_slots: all slots are idle
que    start_loop: waiting for new tasks

0 replies

booxter · 2025-08-15T14:40:36Z

booxter
Aug 15, 2025
Author

Is there an easy way to try out a newer Mesa? I see Rawhide has 25.1.x: https://packages.fedoraproject.org/pkgs/mesa/mesa-libEGL/ but not 25.2.x that is presumably better. Is there a way to pull it into the container? (Would we need to backport the slp COPR patches or are they already present in 25.1.x branch?)

I wonder if it would be too hard to build a replacement container built from Nix packages that already contain 25.2.x+...

0 replies

kpouget · 2025-08-15T16:42:06Z

kpouget
Aug 15, 2025
Collaborator

the important part of the logs is:

common_chat_templates_init: failed to parse chat template (defaulting to chatml): Failed to parse number: '.' ([
json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - invalid lite
ral; last read: '.') at row 5, column 21:

{{ if .System }}{{ .System }}
                    ^
{{- end }}

so it's a template issue, nothing linked with the Vulkan backend, most likely

Is there an easy way to try out a newer Mesa?

the latest Ramalama image should use Mesa 25.0.7-101 (see there). Currently MESA has to be patched by SLP to work on MacOS with krunkit, so other versions can't be easily tested (merged there)

0 replies

booxter · 2025-08-15T18:30:49Z

booxter
Aug 15, 2025
Author

@kpouget when I run with an image that uses KOMPUTE, I see the same error about the template, but the model replies in a reasonable way.

0 replies

booxter · 2025-08-15T19:40:23Z

booxter
Aug 15, 2025
Author

And to be on safe side, I rebuilt the image with the latest main that includes the .101 version bump and confirmed it's installed:

$ podman exec -it ramalama_PYpkXwoIBd dnf info mesa-vulkan-drivers
Updating and loading repositories:
 Copr repo for mesa-libkrun-vulkan owned by slp                         100% |   4.4 KiB/s |   8.9 KiB |  00m02s
 Fedora 42 - aarch64 - Updates                                          100% |   8.3 MiB/s |  19.5 MiB |  00m02s
 Fedora 42 openh264 (From Cisco) - aarch64                              100% |   2.4 KiB/s |   5.8 KiB |  00m02s
 Fedora 42 - aarch64                                                    100% |   5.6 MiB/s |  34.2 MiB |  00m06s
Repositories loaded.
Installed packages
Name            : mesa-vulkan-drivers
Epoch           : 0
Version         : 25.0.7
Release         : 101.fc42
Architecture    : aarch64
Installed size  : 77.5 MiB
Source          : mesa-25.0.7-101.fc42.src.rpm
From repository : copr:copr.fedorainfracloud.org:slp:mesa-libkrun-vulkan
Summary         : Mesa Vulkan drivers
URL             : http://www.mesa3d.org
License         : MIT AND BSD-3-Clause AND SGI-B-2.0
Description     : The drivers with support for the Vulkan API.
Vendor          : Fedora Copr - user slp
[...]

The behaviour is still bad.

0 replies

kpouget · 2025-08-15T20:18:27Z

kpouget
Aug 15, 2025
Collaborator

@kpouget when I run with an image that uses KOMPUTE, I see the same error about the template, but the model replies in a reasonable way.

but it is an old image, or a recent one? (if it's an old one, things may have change since then)

could be interesting to see the behavior when running natively. Metal vs Vulkan vs Kompute. Does Vulkan fail, and everything else succeeds ?
(I'll try to have a look next week)

0 replies

booxter · 2025-08-15T20:25:10Z

booxter
Aug 15, 2025
Author

@kpouget when I test KOMPUTE (instead of VULKAN), I'm testing with main + reverting defines used when building llama from VULKAN to KOMPUTE. So it's the latest image, but built with KOMPUTE.

0 replies

booxter · 2025-08-15T22:00:08Z

booxter
Aug 15, 2025
Author

Some more testing.

With llama-cpp in ramalama closure: ramalama --nocontainer run llama3.2 works fine, and it loads GPU (as per nvtop).

With llama-cpp-vulkan instead of llama-cpp: ramalama --nocontainer run llama3.2 generates garbage. You can trace the difference between two builds of the package here. Looks like it's a matter of GGML_VULKAN plus some additional headers during the build process. And this patch.

With mlx-lm, ramalama --nocontainer --runtime=mlx run llama3.2 doesn't work because mlx-lm assumes a model is a directory with config.json, so it fails with: NotADirectoryError: [Errno 20] Not a directory: '/Users/ihrachyshka/.local/share/ramalama/store/ollama/library/llama3.2/blobs/sha256-dde5aa3fc5ffc17176b5e8bdc82f587b24b2678c6c66101bf7da77af9f7ccdff/config.json'

So I tried with a different model - as listed in README: ramalama --nocontainer --runtime=mlx run hf://mlx-community/Unsloth-Phi-4-4bit and it crashes with:

Traceback (most recent call last):
  File "/nix/store/ks5wzr7dxakmhhdzrrbas33khq3mn0fl-ramalama-0.12.0/lib/python3.13/site-packages/ramalama/cli.py", line 1275, in main
    args.func(args)
    ~~~~~~~~~^^^^^^
  File "/nix/store/ks5wzr7dxakmhhdzrrbas33khq3mn0fl-ramalama-0.12.0/lib/python3.13/site-packages/ramalama/cli.py", line 1010, in run_cli
    model.serve(args, quiet=True) if args.rag else model.run(args)
                                                   ~~~~~~~~~^^^^^^
  File "/nix/store/ks5wzr7dxakmhhdzrrbas33khq3mn0fl-ramalama-0.12.0/lib/python3.13/site-packages/ramalama/model.py", line 400, in run
    self._start_server(args)
    ~~~~~~~~~~~~~~~~~~^^^^^^
  File "/nix/store/ks5wzr7dxakmhhdzrrbas33khq3mn0fl-ramalama-0.12.0/lib/python3.13/site-packages/ramalama/model.py", line 411, in _start_server
    self.serve(args, True)
    ~~~~~~~~~~^^^^^^^^^^^^
  File "/nix/store/ks5wzr7dxakmhhdzrrbas33khq3mn0fl-ramalama-0.12.0/lib/python3.13/site-packages/ramalama/model.py", line 794, in serve
    exec_args = self.build_exec_args_serve(args)
  File "/nix/store/ks5wzr7dxakmhhdzrrbas33khq3mn0fl-ramalama-0.12.0/lib/python3.13/site-packages/ramalama/model.py", line 694, in build_exec_args_serve
    exec_args = self.mlx_serve(args)
  File "/nix/store/ks5wzr7dxakmhhdzrrbas33khq3mn0fl-ramalama-0.12.0/lib/python3.13/site-packages/ramalama/model.py", line 688, in mlx_serve
    return self._build_mlx_exec_args("server", args, extra)
           ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/ks5wzr7dxakmhhdzrrbas33khq3mn0fl-ramalama-0.12.0/lib/python3.13/site-packages/ramalama/model.py", line 522, in _build_mlx_exec_args
    shlex.quote(self._get_entry_model_path(args.container, args.generate, args.dryrun)),
                ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/ks5wzr7dxakmhhdzrrbas33khq3mn0fl-ramalama-0.12.0/lib/python3.13/site-packages/ramalama/model.py", line 216, in _get_entry_model_path
    raise SafetensorModelNotSupported()
ramalama.model.SafetensorModelNotSupported

I found a model that is listed in mlx-lm README that is not in safetensors format. Result:

  File "/nix/store/ks5wzr7dxakmhhdzrrbas33khq3mn0fl-ramalama-0.12.0/lib/python3.13/site-packages/ramalama/model.py", line 688, in mlx_serve
    return self._build_mlx_exec_args("server", args, extra)
           ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/ks5wzr7dxakmhhdzrrbas33khq3mn0fl-ramalama-0.12.0/lib/python3.13/site-packages/ramalama/model.py", line 522, in _build_mlx_exec_args
    shlex.quote(self._get_entry_model_path(args.container, args.generate, args.dryrun)),
                ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/ks5wzr7dxakmhhdzrrbas33khq3mn0fl-ramalama-0.12.0/lib/python3.13/site-packages/ramalama/model.py", line 217, in _get_entry_model_path
    raise NoGGUFModelFileFound()
ramalama.model.NoGGUFModelFileFound

So not sure which model would even work with mlx-lm... Ideas?

0 replies

booxter · 2025-08-16T00:05:19Z

booxter
Aug 16, 2025
Author

The fact that I was able to reproduce it on llama vulkan with --nocontainer allowed me to test some more. I decided I will bump versions of relevant packages, starting from llama.cpp to vulkan-loader to moltenvk to shaderc... I bumped to latest releases of all components and found that the bump of moltenvk from 1.2.11 to 1.3.0 fixes the issue.

I did git bisect old/new for these versions and the PR that fixes the problem for --nocontainer with llama.cpp compiled with Vulkan is: KhronosGroup/MoltenVK#2441 (I backported two patches from the PR, skipping submodule update because this dependency is delivered separately in Nixpkgs, but one would probably need the submodule update in a different environment).

I still don't know a fix for podman scenario. ~~Not sure how to debug it.~~ Actually, moltenVK linked to libkrun is still the culprit. But the fix is something else (not the PR linked) - upgrading to 1.3.0 solves the podman scenario too. Will bisect it.

UPD: I probably haven't restarted podman machine properly the previous time I checked. I can now confirm that the PR linked also fixes the issue for podman scenario.

0 replies

booxter · 2025-08-17T01:09:29Z

booxter
Aug 17, 2025
Author

Not sure if this bug needs anything done in ramalama. At best, README could mention that a newer moltenvk is needed when vulkan is used (krunkit and llama server with vulkan enabled). Not sure if this project wants to track this explicitly though.

0 replies

rhatdan · 2025-08-17T11:00:31Z

rhatdan
Aug 17, 2025
Maintainer

Where is moltenvk installed, locally on the MAC? Is there a way in the pypi to force an upgrade?

Or is this yet another reason to add a dmg installer.

0 replies

kpouget · 2025-08-18T08:07:43Z

kpouget
Aug 18, 2025
Collaborator

Where is moltenvk installed, locally on the MAC? Is there a way in the pypi to force an upgrade?

/opt/homebrew/Cellar/molten-vk-krunkit/1.2.12/lib/libMoltenVK.dylib

or

/opt/podman/lib/libMoltenVK.dylib

On MacOS, molten-vk can come from brew, from @slp's repo

==> Downloading https://raw.githubusercontent.com/slp/homebrew-krunkit/master/bottles/molten-vk-krunkit-1.2.12.arm64_sequoia.bottle.tar.gz

It can also come from Podman installer, which contains the krunkit release binaries, so must be v1.2.12 as well

0 replies

rhatdan · 2025-08-18T11:05:44Z

rhatdan
Aug 18, 2025
Maintainer

Can you open an issue with those packagers krunkit to automatically pull release binaries of 1.2.12 or later.

5 replies

kpouget Aug 18, 2025
Collaborator

containers/krunkit#67

Hatzen Sep 29, 2025

Are there any further news to this discussion/issue?
I ran into a similar issue with Asahi Linux on Mac Studio M1 with ramalama 0.12.2
The krunkit issue doesnt seem to be a solution nor it relates to Linux but to homebrew?

kpouget Sep 29, 2025
Collaborator

I was going to ask

can you check the version of MotenVK installed in your system?

but you're using Asahi Linux, so you don't need the krunkit machinery

which library provides the Vulkan implementation in your system?

Hatzen Sep 29, 2025

thanks for the very fast response!
i am not familiar with these things chatgpt suggested:

vulkaninfo | grep "driverName"
'DISPLAY' environment variable not set... skipping surface info
        driverName                                           = Honeykrisp
        driverName                                           = llvmpipe

Sorry i should have tested a little bit more. The issue seems to be llm related
following works fine:
ramalama --debug serve ollama://gemma3/gemma3:1b
ramalama --debug serve hf://ggml-org/gemma-3-4b-it-GGUF
following doesnt:
ollama://library/alfred
ollama://library/dolphin-mixtral:8x7b

so probably my issue is offtopic and might be corrupted files at least i cannot tell so far..

kpouget Sep 29, 2025
Collaborator

nice,
yes there are issues with ollama models and their template in particular
see this one for instance #1882

Garbage generated when ramalama image is compiled with Vulkan (MacOS, libkrun) #1853

Uh oh!

Uh oh!

booxter Aug 15, 2025

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

ramalama info output

Upstream Latest Release

Additional environment details

Additional information

Replies: 17 comments · 5 replies

Uh oh!

booxter Aug 15, 2025 Author

Uh oh!

booxter Aug 15, 2025 Author

Uh oh!

Uh oh!

kpouget Aug 15, 2025 Collaborator

Uh oh!

joelpurra Aug 15, 2025

Uh oh!

booxter Aug 15, 2025 Author

Uh oh!

booxter Aug 15, 2025 Author

Uh oh!

kpouget Aug 15, 2025 Collaborator

Uh oh!

booxter Aug 15, 2025 Author

Uh oh!

booxter Aug 15, 2025 Author

Uh oh!

kpouget Aug 15, 2025 Collaborator

Uh oh!

booxter Aug 15, 2025 Author

Uh oh!

booxter Aug 15, 2025 Author

Uh oh!

Uh oh!

booxter Aug 16, 2025 Author

Uh oh!

booxter Aug 17, 2025 Author

Uh oh!

rhatdan Aug 17, 2025 Maintainer

Uh oh!

kpouget Aug 18, 2025 Collaborator

Uh oh!

rhatdan Aug 18, 2025 Maintainer

Uh oh!

kpouget Aug 18, 2025 Collaborator

Uh oh!

Hatzen Sep 29, 2025

Uh oh!

kpouget Sep 29, 2025 Collaborator

Uh oh!

Hatzen Sep 29, 2025

Uh oh!

Uh oh!

kpouget Sep 29, 2025 Collaborator

booxter
Aug 15, 2025

Replies: 17 comments 5 replies

booxter
Aug 15, 2025
Author

booxter
Aug 15, 2025
Author

kpouget
Aug 15, 2025
Collaborator

joelpurra
Aug 15, 2025

booxter
Aug 15, 2025
Author

booxter
Aug 15, 2025
Author

kpouget
Aug 15, 2025
Collaborator

booxter
Aug 15, 2025
Author

booxter
Aug 15, 2025
Author

kpouget
Aug 15, 2025
Collaborator

booxter
Aug 15, 2025
Author

booxter
Aug 15, 2025
Author

booxter
Aug 16, 2025
Author

booxter
Aug 17, 2025
Author

rhatdan
Aug 17, 2025
Maintainer

kpouget
Aug 18, 2025
Collaborator

rhatdan
Aug 18, 2025
Maintainer

kpouget Aug 18, 2025
Collaborator

kpouget Sep 29, 2025
Collaborator

kpouget Sep 29, 2025
Collaborator