Quantization Variant Metadata and Selection for Edge Agent Deployment

[Shehrozkashif]

Quantization Variant Metadata and Selection for Edge Agent Deployment

Problem

Building autonomous GUI agent (GSoC Intel/OpenVINO) on Intel Core Ultra 7 258V with 27GB VRAM.

Currently hardcoded in config.py:

• qwen3-8b-int4-ov: 5.5 GB, 7-15s latency
• ui-tars-1.5-7b-int8-ov: 7.5 GB, 200ms latency

Tested but rejected:

• 14B LLM: latency jumped to 45-101s (too slow)
• FP16 VLM: +8GB VRAM, no accuracy gain

Supporting multiple hardware targets (laptop 4GB, desktop 8GB, cloud 64GB) requires separate ModelKits or complex config logic.

Solution

One ModelKit per model with quantization variants + metadata:

ModelKit: qwen3-8b

• int4: 5.5 GB, min 4GB VRAM, 7-15s latency
• int8: 7.2 GB, min 6GB VRAM, 5-10s latency
• fp16: 13 GB, min 14GB VRAM, 3-7s latency

Query at deployment:
kit_query("qwen3-8b", max_size=8, target_accelerator="Intel GPU")
→ returns: qwen3-8b:int4

Metadata Needed

• Quantization scheme (int4, int8, fp16, fp32)
• Artifact size
• Min VRAM required
• Target accelerator (CPU, GPU, Intel GPU, NPU)
• Measured latency
• Accuracy delta (optional)

Questions

1. Variants as separate manifests in one ModelKit?
2. Metadata in Kitfile, manifest, or separate file?
3. Query/matching provided by KitOps or runtime?
4. How to express "optimized for Intel GPU, also works on generic GPU"?

Impact

Without variant standardization: 10 models × 4 variants = 40 separate ModelKits
With variant support: 10 models × 1 ModelKit each = clean, queryable

Artifacts available (actual quantized models, config.py, latency logs).

[Schnitzel]

Data point from the opposite end of the hardware range — datacentre GPUs rather than edge — where the same metadata gaps show up. Strong +1 on the shape of this; three things we measured suggest the metadata list needs extending.

1. "Min VRAM" does not rank variants

We benchmarked four models across 32 concurrency points each on H200 NVL. Inkling-Small (NVFP4, 159 GiB of weights) on two GPUs beat GLM-5.3-Flash (FP8, 306 GiB) on four at every single point — 2291 vs 1610 tok/s aggregate at 32 concurrent requests. A selector that knows only artifact size and min-VRAM picks the wrong one here.

What did predict the ordering was active parameters per token:

model	active	total	c=1 tok/s	agg @ c=32
Inkling-Small	~12 B (6 of 256 experts)	266 B	140.1	2291
MiMo-V2.5	15 B (8 of 256)	310 B	156.4	1802
Hy3	21 B (8 of 192)	295 B	92.3	1395

Three vendors, three architectures, one ordering — and it is active parameters, not total parameters and not artifact size. For MoE variants that seems worth carrying as an explicit field.

2. Measured latency is not a scalar

The same engine and variant gave 140.1 tok/s at concurrency 1 and 71.6 tok/s at concurrency 32 — a 2x spread with nothing changed but load. A latency: 200ms field needs a concurrency qualifier, otherwise it is only meaningful for the single-user case.

The shape of that decay also varies a lot between variants (1.77x to 2.78x from c=1 to c=32 across our four), and it is what actually determines how many users a variant serves. If the goal is kit_query(...) returning a variant, decay-under-load is arguably more decision-relevant than a single latency number.

3. Accelerator identity needs a count, and the engine build matters

target_accelerator: "Intel GPU" is not sufficient for the datacentre case: the correct tuning differs between 2 and 4 of the same GPU. We have one runtime field whose correct value is a function of GPU count and speculative-decoding depth, worth ~9 % when wrong (details in modelpack/model-spec#208).

Separately, runnability is not a property of the weights alone. We have a checkpoint that appears in a stable vLLM release's model registry and still fails during weight load on that release, requiring a specific prebuilt image. So the tuple is really (weights x accelerator count x engine build) — and a variant selector that ignores the engine build can return a variant that cannot load at all.

On your open questions

• Q2 (Kitfile vs manifest vs separate file): Should ModelPack standardize a Modelfile format? modelpack/model-spec#208 is actively debating exactly this for runtime/serving profiles. The argument there — that a profile is mutable and plural per model, so it should not force a new model digest — applies to variant metadata too, and points at attached artifacts rather than the packaging manifest.
• Q4 ("optimized for Intel GPU, also works on generic GPU"): we ended up needing the same expressiveness for "tuned for 4x H200, runs on 2x with a different profile", which suggests this wants to be a set of profiles with a match/score function rather than a single target string.

[Shehrozkashif]

Thanks for the detailed breakdown, this is really useful. Agreed that min-VRAM and a single latency number don't capture enough, especially for MoE models. I'll add active parameters, concurrency-tagged latency, and accelerator count + engine build to the metadata list.

Also good pointer to model-spec#208, makes sense to treat this as a separate, attachable profile rather than baking it into the manifest. Will look into a match/score function instead of a single target string, similar to what you suggested for the 2x vs 4x GPU case.