How can this run MLX

[TomLucidor]

Referring to the following models

• https://huggingface.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit
• https://huggingface.co/mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit
• https://huggingface.co/mlx-community/gpt-oss-20b-MXFP4-Q4

Got the following

 ~ % ramalama --runtime=mlx serve https://hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit
INFO:ramalama:MLX runtime automatically uses --nocontainer mode
Downloading Kimi-Linear-48B-A3B-Instruct-4bit
100% |████████████████████████████|   98.71 KB/  98.71 KB   3.78 MB/s        0s
Traceback (most recent call last):
  File "/Users/user/.local/bin/mlx_lm.server", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/user.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/server.py", line 1083, in main
    run(args.host, args.port, ModelProvider(args))
                              ^^^^^^^^^^^^^^^^^^^
  File "/Users/user/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/server.py", line 168, in __init__
    self.load(self.cli_args.model, draft_model_path="default_model")
  File "/Users/user/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/server.py", line 196, in load
    model, tokenizer = load(
                       ^^^^^
  File "/Users/user/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/utils.py", line 266, in load
    model, config = load_model(model_path, lazy, model_config=model_config)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/utils.py", line 165, in load_model
    config = load_config(model_path)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/utils.py", line 127, in load_config
    with open(model_path / "config.json", "r") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
NotADirectoryError: [Errno 20] Not a directory: '/Users/user/.local/share/ramalama/store/https/hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit/blobs/sha256-2a8496563bad6a7f2a3d9537b514826a4fd474e41ea88c2e0a5f51c58420964f/config.json'

[rhatdan]

@engelmi PTAL, looks like we are handing MLX the wrong file?

[engelmi]

@TomLucidor Can you provide the debug output?
ramalama --runtime=mlx serve https://hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit

It seems we pass each file individually to the mlx CLI and the mlx library expects a directory containing all files to be passed - and has expectations on the naming. I can't test it, unfortunately. Maybe @kush-gupt can help? I'll still have a look at it.

[kush-gupt]

The URL pull fails super quick:

❯ ramalama --debug --runtime=mlx serve https://hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit
INFO:ramalama:MLX runtime automatically uses --nocontainer mode
2025-11-04 14:35:44 - DEBUG - Checking if 8080 is available
DEBUG:ramalama:Checking if 8080 is available
2025-11-04 14:35:44 - DEBUG - run_cmd: npu-smi info
DEBUG:ramalama:run_cmd: npu-smi info
2025-11-04 14:35:44 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:35:44 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:35:44 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:35:44 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:35:44 - DEBUG - run_cmd: mthreads-gmi
DEBUG:ramalama:run_cmd: mthreads-gmi
2025-11-04 14:35:44 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:35:44 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:35:44 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:35:44 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:35:44 - DEBUG - run_cmd: podman inspect quay.io/ramalama/ramalama:0.14
DEBUG:ramalama:run_cmd: podman inspect quay.io/ramalama/ramalama:0.14
2025-11-04 14:35:44 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:35:44 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:35:44 - DEBUG - Ignore all: True
DEBUG:ramalama:Ignore all: True
2025-11-04 14:35:44 - DEBUG - env: None
DEBUG:ramalama:env: None
Downloading Kimi-Linear-48B-A3B-Instruct-4bit
2025-11-04 14:35:44 - DEBUG - Running urlopen https://hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit with headers: {'Range': 'bytes=0-'}
DEBUG:ramalama:Running urlopen https://hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit with headers: {'Range': 'bytes=0-'}
100% |██████████████████████████████████████████████████████████████████████████████████████████████████████████████|   98.72 KB/  98.72 KB  11.91 MB/s        0s
2025-11-04 14:35:45 - DEBUG - run_cmd: npu-smi info
DEBUG:ramalama:run_cmd: npu-smi info
2025-11-04 14:35:45 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:35:45 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:35:45 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:35:45 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:35:45 - DEBUG - run_cmd: mthreads-gmi
DEBUG:ramalama:run_cmd: mthreads-gmi
2025-11-04 14:35:45 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:35:45 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:35:45 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:35:45 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:35:45 - DEBUG - run_cmd: podman inspect quay.io/ramalama/ramalama:0.14
DEBUG:ramalama:run_cmd: podman inspect quay.io/ramalama/ramalama:0.14
2025-11-04 14:35:45 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:35:45 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:35:45 - DEBUG - Ignore all: True
DEBUG:ramalama:Ignore all: True
2025-11-04 14:35:45 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:35:45 - DEBUG - run_cmd: npu-smi info
DEBUG:ramalama:run_cmd: npu-smi info
2025-11-04 14:35:45 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:35:45 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:35:45 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:35:45 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:35:45 - DEBUG - run_cmd: mthreads-gmi
DEBUG:ramalama:run_cmd: mthreads-gmi
2025-11-04 14:35:45 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:35:45 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:35:45 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:35:45 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:35:45 - DEBUG - exec_cmd: mlx_lm.server --model /Users/kugupta/.local/share/ramalama/store/https/hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit/blobs/sha256-2a8496563bad6a7f2a3d9537b514826a4fd474e41ea88c2e0a5f51c58420964f --temp 0.8 --host 0.0.0.0 --port 8080
DEBUG:ramalama:exec_cmd: mlx_lm.server --model /Users/kugupta/.local/share/ramalama/store/https/hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit/blobs/sha256-2a8496563bad6a7f2a3d9537b514826a4fd474e41ea88c2e0a5f51c58420964f --temp 0.8 --host 0.0.0.0 --port 8080
Traceback (most recent call last):
  File "/Users/kugupta/.local/bin/mlx_lm.server", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/server.py", line 1083, in main
    run(args.host, args.port, ModelProvider(args))
                              ^^^^^^^^^^^^^^^^^^^
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/server.py", line 168, in __init__
    self.load(self.cli_args.model, draft_model_path="default_model")
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/server.py", line 196, in load
    model, tokenizer = load(
                       ^^^^^
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/utils.py", line 266, in load
    model, config = load_model(model_path, lazy, model_config=model_config)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/utils.py", line 165, in load_model
    config = load_config(model_path)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/utils.py", line 127, in load_config
    with open(model_path / "config.json", "r") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
NotADirectoryError: [Errno 20] Not a directory: '/Users/kugupta/.local/share/ramalama/store/https/hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit/blobs/sha256-2a8496563bad6a7f2a3d9537b514826a4fd474e41ea88c2e0a5f51c58420964f/config.json'

[engelmi]

Thanks! @kush-gupt
When pulled via URL, then the file type is not set to be a safetensor. In case of safetensors, the whole snapshot directory is passed to mlx. In contrast, for normal files the specific blob is passed. I'll have a look at this, fix should be easy to enable this for URL pulls as well.

[engelmi]

Rechecking again, the issue is that only one file is being pulled by RamaLama: the .html of the hf website.
Which makes sense. We could do a bit of preprocessing for the model input (similar to the shortnames) and detect the host (huggingface.co) and change the model input accordingly - so it wouldn't use the http pull, but the hf one (which would fall back to the hf cli).

[engelmi]

@TomLucidor @kush-gupt Opened a PR to map certain https:// usages internally to huggingface/ollama and pull the whole model: #2117

[kush-gupt]

It looks like the URL pull https://hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit versus the hf pull ramalama --debug --runtime=mlx run hf://mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit causes the issue here. Have not tested URL pull for MLX before so that's not unexpected, but pulling down the model successfully on the latest released Ramalama.

The next crash I ran into was the following, seems to be an issue at the MLX layer:

❯ ramalama --debug --runtime=mlx run hf://mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit
INFO:ramalama:MLX runtime automatically uses --nocontainer mode
2025-11-04 14:20:10 - DEBUG - Checking if 8080 is available
DEBUG:ramalama:Checking if 8080 is available
2025-11-04 14:20:10 - DEBUG - run_cmd: npu-smi info
DEBUG:ramalama:run_cmd: npu-smi info
2025-11-04 14:20:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:20:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:20:10 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:20:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:20:10 - DEBUG - run_cmd: mthreads-gmi
DEBUG:ramalama:run_cmd: mthreads-gmi
2025-11-04 14:20:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:20:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:20:10 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:20:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:20:10 - DEBUG - run_cmd: podman inspect quay.io/ramalama/ramalama:0.14
DEBUG:ramalama:run_cmd: podman inspect quay.io/ramalama/ramalama:0.14
2025-11-04 14:20:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:20:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:20:10 - DEBUG - Ignore all: True
DEBUG:ramalama:Ignore all: True
2025-11-04 14:20:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:20:10 - DEBUG - run_cmd: npu-smi info
DEBUG:ramalama:run_cmd: npu-smi info
2025-11-04 14:20:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:20:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:20:10 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:20:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:20:10 - DEBUG - run_cmd: mthreads-gmi
DEBUG:ramalama:run_cmd: mthreads-gmi
2025-11-04 14:20:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:20:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:20:10 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:20:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:20:10 - DEBUG - run_cmd: podman inspect quay.io/ramalama/ramalama:0.14
DEBUG:ramalama:run_cmd: podman inspect quay.io/ramalama/ramalama:0.14
2025-11-04 14:20:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:20:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:20:10 - DEBUG - Ignore all: True
DEBUG:ramalama:Ignore all: True
2025-11-04 14:20:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:20:10 - DEBUG - MLX server not ready, waiting... (attempt 1/10)
DEBUG:ramalama:MLX server not ready, waiting... (attempt 1/10)
2025-11-04 14:20:10 - DEBUG - run_cmd: npu-smi info
DEBUG:ramalama:run_cmd: npu-smi info
2025-11-04 14:20:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:20:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:20:10 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:20:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:20:10 - DEBUG - run_cmd: mthreads-gmi
DEBUG:ramalama:run_cmd: mthreads-gmi
2025-11-04 14:20:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:20:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:20:10 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:20:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:20:10 - DEBUG - exec_cmd: mlx_lm.server --model /Users/kugupta/.local/share/ramalama/store/huggingface/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit/snapshots/sha256-e0f9962de48aca5482b3f9308b9ef4719df02f51 --temp 0.8 --port 8080
DEBUG:ramalama:exec_cmd: mlx_lm.server --model /Users/kugupta/.local/share/ramalama/store/huggingface/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit/snapshots/sha256-e0f9962de48aca5482b3f9308b9ef4719df02f51 --temp 0.8 --port 8080
2025-11-04 14:20:11,477 - ERROR - Model type kimi_linear not supported.
Traceback (most recent call last):
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/utils.py", line 68, in _get_classes
    arch = importlib.import_module(f"mlx_lm.models.{model_type}")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kugupta/.local/share/uv/python/cpython-3.12.8-macos-aarch64-none/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1324, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'mlx_lm.models.kimi_linear'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/kugupta/.local/bin/mlx_lm.server", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/server.py", line 1083, in main
    run(args.host, args.port, ModelProvider(args))
                              ^^^^^^^^^^^^^^^^^^^
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/server.py", line 168, in __init__
    self.load(self.cli_args.model, draft_model_path="default_model")
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/server.py", line 196, in load
    model, tokenizer = load(
                       ^^^^^
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/utils.py", line 266, in load
    model, config = load_model(model_path, lazy, model_config=model_config)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/utils.py", line 178, in load_model
    model_class, model_args_class = get_model_classes(config=config)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/utils.py", line 72, in _get_classes
    raise ValueError(msg)
ValueError: Model type kimi_linear not supported.

From uv tool list:

mlx-lm v0.28.3
- mlx_lm.awq
- mlx_lm.benchmark
- mlx_lm.cache_prompt
- mlx_lm.chat
- mlx_lm.convert
- mlx_lm.dwq
- mlx_lm.dynamic_quant
- mlx_lm.evaluate
- mlx_lm.fuse
- mlx_lm.generate
- mlx_lm.gptq
- mlx_lm.lora
- mlx_lm.manage
- mlx_lm.perplexity
- mlx_lm.server
- mlx_lm.upload
ramalama v0.14.0
- ramalama

[kush-gupt]

Seems to be tracking at this PR here: ml-explore/mlx-lm#577

[kush-gupt]

Seems OSS works perfectly fine since it has been implemented within MLX for a while, example:

❯ ramalama --debug --runtime=mlx run hf://mlx-community/gpt-oss-20b-MXFP4-Q4
INFO:ramalama:MLX runtime automatically uses --nocontainer mode
2025-11-04 14:29:10 - DEBUG - Checking if 8080 is available
DEBUG:ramalama:Checking if 8080 is available
2025-11-04 14:29:10 - DEBUG - Checking if 8082 is available
DEBUG:ramalama:Checking if 8082 is available
2025-11-04 14:29:10 - DEBUG - run_cmd: npu-smi info
DEBUG:ramalama:run_cmd: npu-smi info
2025-11-04 14:29:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:29:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:29:10 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:29:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:29:10 - DEBUG - run_cmd: mthreads-gmi
DEBUG:ramalama:run_cmd: mthreads-gmi
2025-11-04 14:29:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:29:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:29:10 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:29:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:29:10 - DEBUG - run_cmd: podman inspect quay.io/ramalama/ramalama:0.14
DEBUG:ramalama:run_cmd: podman inspect quay.io/ramalama/ramalama:0.14
2025-11-04 14:29:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:29:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:29:10 - DEBUG - Ignore all: True
DEBUG:ramalama:Ignore all: True
2025-11-04 14:29:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:29:10 - DEBUG - run_cmd: npu-smi info
DEBUG:ramalama:run_cmd: npu-smi info
2025-11-04 14:29:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:29:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:29:10 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:29:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:29:10 - DEBUG - run_cmd: mthreads-gmi
DEBUG:ramalama:run_cmd: mthreads-gmi
2025-11-04 14:29:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:29:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:29:10 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:29:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:29:10 - DEBUG - run_cmd: podman inspect quay.io/ramalama/ramalama:0.14
DEBUG:ramalama:run_cmd: podman inspect quay.io/ramalama/ramalama:0.14
2025-11-04 14:29:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:29:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:29:10 - DEBUG - Ignore all: True
DEBUG:ramalama:Ignore all: True
2025-11-04 14:29:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:29:10 - DEBUG - MLX server not ready, waiting... (attempt 1/10)
DEBUG:ramalama:MLX server not ready, waiting... (attempt 1/10)
2025-11-04 14:29:10 - DEBUG - run_cmd: npu-smi info
DEBUG:ramalama:run_cmd: npu-smi info
2025-11-04 14:29:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:29:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:29:10 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:29:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:29:10 - DEBUG - run_cmd: mthreads-gmi
DEBUG:ramalama:run_cmd: mthreads-gmi
2025-11-04 14:29:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:29:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:29:10 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:29:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:29:10 - DEBUG - exec_cmd: mlx_lm.server --model /Users/kugupta/.local/share/ramalama/store/huggingface/mlx-community/gpt-oss-20b-MXFP4-Q4/snapshots/sha256-a35c077138d6f155efd51134df24d477c0b1fc19 --temp 0.8 --port 8082
DEBUG:ramalama:exec_cmd: mlx_lm.server --model /Users/kugupta/.local/share/ramalama/store/huggingface/mlx-community/gpt-oss-20b-MXFP4-Q4/snapshots/sha256-a35c077138d6f155efd51134df24d477c0b1fc19 --temp 0.8 --port 8082
2025-11-04 14:29:13 - DEBUG - MLX server not ready, waiting... (attempt 2/10)
DEBUG:ramalama:MLX server not ready, waiting... (attempt 2/10)
/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/server.py:973: UserWarning: mlx_lm.server is not recommended for production as it only implements basic security checks.
  warnings.warn(
2025-11-04 14:29:13,909 - INFO - Starting httpd at 127.0.0.1 on port 8082...
🍏 > Hello!
2025-11-04 14:29:22 - DEBUG - Request: URL=http://127.0.0.1:8082/v1/chat/completions, Data=b'{"stream": true, "messages": [{"role": "user", "content": "Hello!"}], "temperature": 0.8}', Headers={'Content-Type': 'application/json'}
DEBUG:ramalama:Request: URL=http://127.0.0.1:8082/v1/chat/completions, Data=b'{"stream": true, "messages": [{"role": "user", "content": "Hello!"}], "temperature": 0.8}', Headers={'Content-Type': 'application/json'}
127.0.0.1 - - [04/Nov/2025 14:29:22] "POST /v1/chat/completions HTTP/1.1" 200 -
2025-11-04 14:29:22,474 - INFO - Prompt processing progress: 0/69
2025-11-04 14:29:23,633 - INFO - Prompt processing progress: 68/69
2025-11-04 14:29:23,671 - INFO - Prompt processing progress: 69/69
<|channel|>analysis<|message|>We need to respond. The user just says "Hello!" It's a greeting. We need to respond politely. Then maybe ask how can help. So produce friendly answer.<|end|><|start|>assistant<|channel|>final<|message|>Hello! 😊 How can I help you today?
🍏 > /bye

[TomLucidor]

Could you check Qwen3 and Kimi Linear just in case?

[kush-gupt]

@TomLucidor So Kimi Linear just isn't in MLX yet unless you build off that PR because of how new it is. Once it's in MLX, you should be able to run uv tool upgrade mlx-lm and then the ramalama commands.

Here is Qwen3 Coder working just fine though:

❯ ramalama --runtime=mlx run hf://mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit
INFO:ramalama:MLX runtime automatically uses --nocontainer mode
🍏 > Create an infinite python loop
Here are several ways to create infinite loops in Python:

## Method 1: Using `while True` (Most Common)
```python
while True:
    print("This loop will run forever")
    # Add your code here
    # Use break condition when needed

Method 2: Using while 1 (Same as True)

while 1:
    print("This loop will run forever")
    # Add your code here

Method 3: Using for loop with `itertools.count()

import itertools

for i in itertools.count():
    print(f"Loop iteration: {i}")
    # Add your code here
    # Use break condition when needed

Method 4: Using while True with break condition

counter = 0
while True:
    print(f"Loop iteration: {counter}")
    counter += 1

    # Break condition - remove or modify as needed
    if counter >= 5:
        break

Method 5: Using a function with recursion (be careful!)

def infinite_loop():
    print("Infinite loop running")
    infinite_loop()  # Recursive call

# This will cause a RecursionError in Python
# infinite_loop()

Example with user input and break condition:

while True:
    user_input = input("Enter 'quit' to stop: ")

    if user_input.lower() == 'quit':
        print("Goodbye!")
        break
    else:
        print(f"You entered: {user_input}")

Example with a simple counter:

import time

count = 0
while True:
    print(f"Count: {count}")
    count += 1
    time.sleep(1)  # Wait 1 second between iterations

Important Notes:

• Use Ctrl+C (or Ctrl+Z on Windows) to stop an infinite loop in the terminal
• Always include a break condition in production code
• Be careful with infinite loops in production applications as they can consume system resources
• Use break statement to exit the loop when needed
• Use continue to skip the current iteration and move to the next one

The most common and recommended approach is while True with proper break conditions.
🍏 > /bye

[TomLucidor]

Merged 3 days ago (hype!) Please test that again, so that I can see if can work with coding agents. Could Be The Move.