[presentyourlove]

Description

This PR implements Frontier Dimension 1.1: Built-in One-Click Model Quantization CLI (exo quantize), providing an end-to-end local quantization pipeline without external tooling dependencies.

Background & Motivation

Deploying extreme low-bit models on edge devices typically relies on pre-quantized community weights (e.g. GGUF or MLX weights on Hugging Face). When users wish to serve newly released foundation models or proprietary fine-tuned checkpoints, obtaining high-quality ultra-low-bit representations is a major friction point.
With exo quantize <model-id> --bits [1|1.58|2] --output <dir>, users can quantize any model locally. The pipeline automatically analyzes architectural sensitivity (protecting critical attention output and gate layers), calibrates activation outliers via SmoothQuant, and auto-generates registered Exo .toml model cards for immediate cluster inference.

Key Changes

1. Quantization CLI Types & Reports (src/exo/shared/types/quantize_cli.py):

   • QuantizationConfig: Encapsulates model ID, bit depth (1-bit, 1.58-bit, 2-bit), calibration samples, and layer protection toggles.
   • LayerSensitivityRecord & QuantizationReport: Records per-layer sensitivity classification and final compression metrics.

2. Quantization Compiler & Calibration Engine (src/exo/worker/engines/quantization/quantize_compiler.py):

   • ModelSensitivityAnalyzer: Automatically detects critical backbone layers (embed_tokens, lm_head, norm) and sensitive projection heads (o_proj, gate_proj), preserving them at FP16.
   • SmoothQuantCalibrator: Implements $s = \frac{\max(|X|)^\alpha}{\max(|W|)^{1-\alpha}}$ to migrate activation outlier difficulty into weights.
   • ExoModelCardGenerator: Automatically generates and writes .toml model cards into the target directory.
   • QuantizeCompiler: Coordinates the end-to-end compilation workflow.

3. CLI Command & Exo Main Dispatch (src/exo/cli/commands/quantize.py, src/exo/main.py, src/exo/__main__.py):

   • Implemented exo quantize CLI subcommand with progress and summary output.
   • Wired dispatch in exo.main and exo.__main__.

4. Unit Tests (src/exo/shared/tests/test_quantize_cli.py):

   • Verified layer sensitivity detection and precision assignments.
   • Verified SmoothQuant migration factor computation.
   • Verified valid Exo .toml model card generation.
   • Verified 1-bit, 1.58-bit, and 2-bit end-to-end compilation pipelines and CLI command argument validation.

Verification

• Ran unit tests:

  .venv/Scripts/pytest.exe src/exo/shared/tests/test_quantize_cli.py
  # 6 passed in 0.09s

• Code styling and linting:

  .venv/Scripts/ruff.exe check src/exo/shared/types/quantize_cli.py src/exo/worker/engines/quantization/quantize_compiler.py src/exo/worker/engines/quantization/__init__.py src/exo/cli/commands/quantize.py src/exo/cli/__init__.py src/exo/shared/tests/test_quantize_cli.py
  # All checks passed!

• Tested CLI invocation directly:

  python -m exo quantize meta-llama/Llama-3.2-1B --bits 1.58 --output ./test_quant_out
  # Successfully compiled and generated model_card.toml

Related Issues

• Resolves: Frontier Dimension 1.1 in 人人都能跑llm.md (Built-in One-Click Model Quantization CLI)