Llama Deck

Llama Deck is a self-hosted operations console for local llama.cpp model containers. It manages an explicit allow-list of Docker deployments, streams OpenAI-compatible model output, measures native llama.cpp timings, and exposes host telemetry without giving the browser a shell.

Highlights

• Stable, fixed-size model cards with start, stop, restart, logs, process RSS, CPU utilization, DRM GTT/VRAM, and GPU engine utilization.
• Structured, allow-listed llama.cpp launch parameter editing with persistent overrides and rollback-protected container recreation.
• Streaming chat, embedding inspection, and reranker testing through the same UI.
• Benchmark profiles and profile matrices for prefill, decode, warmup, prompt length, output length, and measured rounds.
• Per-model benchmark history plus complete JSON logs containing configuration, every measured round, aggregate metrics, deviations, outputs, and failures.
• CPU, GPU, RAM, swap, disk, network, load average, hwmon temperature, and fan telemetry.
• Automatic display of Linux distribution, kernel, CPU, Docker/API, AMDGPU, ROCm, and Llama Deck versions.
• External JSON model registry. No model names, paths, ports, or GPU architecture are compiled into the application.

Quick Start

Requirements: Linux, Docker Engine with Compose, and one or more existing or declaratively configured model containers.

git clone https://github.com/mango8853/llama-deck.git
cd llama-deck
cp config/models.example.json config/models.json
# Edit config/models.json for your images, model paths, ports, and devices.
docker compose up -d --build

Open http://HOST_IP:8090 on your LAN.

Llama Deck uses host networking because local inference servers commonly bind to host loopback addresses. If you use a different networking layout, change each model's endpoint and the Compose networking settings together.

Model Registry

Each entry in config/models.json describes one allow-listed deployment:

{
  "id": "my-model-rocm",
  "container": "my-model-rocm",
  "name": "My Model · ROCm",
  "kind": "chat",
  "endpoint": "http://127.0.0.1:8081",
  "port": 8081,
  "context": 32768,
  "editable_parameters": ["context", "gpu_layers", "batch_size", "threads"],
  "launch": {
    "image": "ghcr.io/ggml-org/llama.cpp:server-rocm",
    "entrypoint": ["/app/llama-server"],
    "command": ["-m", "/models/model.gguf", "-c", "32768", "-ngl", "999"],
    "volumes": {"/host/models": {"bind": "/models", "mode": "ro"}},
    "devices": ["/dev/kfd:/dev/kfd:rwm", "/dev/dri:/dev/dri:rwm"]
  }
}

The launch block is used when the container does not exist. Existing containers can be adopted: Llama Deck reads their image, entrypoint, arguments, mounts, devices, ports, network, capabilities, and restart policy. Applying an edited parameter renames the old container as a rollback candidate, creates and starts the replacement, and only then removes the old one.

Models sharing the same declared port are treated as mutually exclusive. A start or restart automatically stops another registered model on that port.

Editable Parameters

The UI never accepts a raw shell command. A registry entry may expose these structured llama.cpp parameters:

context, gpu_layers, batch_size, ubatch_size, threads, parallel, flash_attention, cache_type_k, cache_type_v, temperature, top_p, top_k, and min_p.

Model paths, host bindings, ports, entrypoints, and Docker security settings are not editable from the browser. Change those in the registry.

Benchmark Logs

Summary records are stored in data/benchmarks.json. Complete logs are stored as individual files under data/benchmark-logs/. Both are persistent Compose volumes and are intentionally excluded from Git.

The built-in profiles are:

Profile	Prompt	Output	Purpose
Quick	512	128	Fast smoke test
Standard	2,048	256	Balanced comparison
Long prefill	8,192	256	Prompt processing throughput
Decode	512	1,024	Sustained token generation

Custom tests support up to 262,144 prompt tokens and 65,536 output tokens, subject to the model server's configured context and memory capacity.

Security

Llama Deck mounts the Docker socket and read-only host telemetry paths. Docker socket access is effectively host-root access. Run it only on a trusted machine, keep it behind a firewall or authenticated reverse proxy, and do not expose it directly to the public internet. See SECURITY.md.

Development

python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
python -m unittest discover -s tests -v
uvicorn app.main:app --reload --port 8090

License

MIT