exl3

Kyojin: open-source engine that runs 100 GB-class open models (GLM-5.3, Qwen3.8, DeepSeek V4.1) on one desktop machine. AMD Strix Halo, NVIDIA RTX and Apple Mac. OpenAI API, built on ExLlamaV3.

DeepSeek-V4.1-Flash on two DGX Sparks: EXL3 routed experts, 262k context, 2M-token KV, vision, tools, MTP, bundled with Pi

DeepSeek-V4.1-Flash-1017 EXL3 2.9bpw on 2x NVIDIA DGX Spark (GB10) with TensorFold: 2.1-2.3x the vLLM kit's decode speed.

An environment where open models work as agents: a first-level harness with routing, skills, memory and sandboxed tools, on an engine that drafts and verifies its own tokens.

Fast, capable GLM-5.3-Flash FP4-Local on two DGX Sparks: opinionated Atlas serving and reproducible inference research.

GLM-5.3-Flash EXL3 (320B MoE) on 2x NVIDIA DGX Spark — production serving kit, 1M context, 97%+ multi-session prefix caching, DFlash2 spec decode

Validated GLM-5.3 Flash recipe for 2x NVIDIA RTX PRO 6000 Blackwell 96GB: 262K context, EXL3/TR3, adaptive MTP, tools, and vision.

DeepSeek-V4.1-Flash on two NVIDIA DGX Sparks: EXL3 weights from the official checkpoint, upstream vLLM with our patches and decode kernels

A tiered-memory system design for workloads that don't fit in RAM: measure the working set, pin the hot tier, stream the cold tier from flash. Ships the residency calculator, measurement harnesses, and the build recipes behind it. Predictions validated against public benchmarks.

Serve EXL3 (ExLlamaV3 trellis) quantized models on vLLM fork runtimes — any architecture, mixed per-layer bitrates, composable with source-format non-routed weights

vLLM for NVIDIA CMP 170HX (Ampere sm_80): fast, repeatable model serving on one card or several

vLLM recipe: GLM-5.3-Flash EXL3 K2 on one DGX Spark GB10. Native MTP k=2. Measured tok/s.

Qwen3.8-27B EXL3 (4.00 bpw) + DFlash2 speculative decoding for ExLlamaV3, validated at 262k context on a 24 GB RTX 3090

DeepSeek-V4.1-Flash on 4× NVIDIA CMP 170HX with vLLM: up to 297 tok/s for one user, up to 868 tok/s at eight, 262K context, one-command container install

DeepSeek 1017 EXL3 on 3-4 DGX Sparks. TP4 4.0.4: 4,277 tok/s cold prefill; C16 code 331.7 / prose 287.5 native decode sum. Smart CUDA Graph, KV Cache and Adaptive Speculation controllers. C16/R48, 9M KV, 3.5 GiB checkpoints. TP3 1.2.8: up to +58% cold prefill. Exact recipes, source and benchmarks.

Run Qwen3.8-Flash-Next on one AMD Strix Halo (Ryzen AI Max+ 395, gfx1151). EXL3 4.05 and 3.05 bpw packs, own ROCm/HIP kernels, OpenAI-compatible server, 4 sessions of 262K context, setup guide included.

GLM-5.3-Flash EXL3 on 4x RTX PRO 6000 with TensorFold: one-command recipe and engine patches. By Aevonix Research in collaboration with Mia's AI Lab.

DeepSeek-V4.1-Flash-1017 on two NVIDIA DGX Sparks (GB10) with TensorFold TP2: scripts, chat-template fix, speed and quality results

Qwen3.8-27B (EXL3) on an AMD RX 9070 XT: ~150 tok/s for code, 96k context, one-line install (container, any Linux distro)

Reproducible 262K GLM-5.3 Flash Uncensored EXL3+DFlash2 deployment for 2x RTX PRO 6000 Blackwell