[presentyourlove]

Description

This PR implements AirLLM-Style Sliding-Window Layer Streaming (SlidingWindowLayerStreamer) using an asynchronous double-buffered ring pipeline and zero-dirty-page clean eviction, enabling 70B parameter models to run on memory-constrained edge hardware with 2GB–4GB RAM.

Background & Motivation

In extreme edge hardware environments (e.g. legacy laptops, mini PCs with 2GB–4GB RAM), storing even 2-bit quantized 70B model weights (7GB–18GB) exceeds physical memory capacity, while OS swap causes catastrophic thrashing.
Because Transformer forward inference proceeds strictly layer-by-layer sequentially, retaining only the currently active layer in RAM and prefetching the subsequent layer over a double-buffered ring decouples memory capacity from total model parameter count.

Key Changes

1. Streaming Data Models (src/exo/shared/types/streaming_layer.py):

   • LayerStreamingConfig: Configuration defining streaming mode, buffer size limit, and clean page eviction toggle.
   • LayerDiskMetadata: Location descriptor tracking layer offsets, file paths, and byte sizes.
   • StreamedLayerBuffer: Status descriptor representing double-buffering slots (slot_a, slot_b).
   • StreamingPipelineTelemetry: Tracks peak resident RAM, swap latency, and I/O overlap ratio.

2. Sliding Window Layer Streamer Engine (src/exo/worker/engines/streaming/layer_streamer.py):

   • Double-Buffered Ring Pipeline: Maintains alternating active and standby slots. While layer $L$ computes in slot A, layer $L+1$ prefetches asynchronously into slot B.
   • Pointer Swap & Clean Page Eviction: Upon completing a layer, active pointers swap and the previous layer is immediately evicted from RAM, guaranteeing peak RAM usage remains strictly capped at the size of at most 2 layers (< 500MB).

3. Module Exports (src/exo/worker/engines/streaming/__init__.py):

   • Exported SlidingWindowLayerStreamer.

4. Unit Tests (src/exo/shared/tests/test_sliding_window_streamer.py):

   • Verified pipeline initialization and initial layer load.
   • Verified sequential layer advancement and eviction across multiple layers.
   • Verified peak memory residency strictly capped at 2 layers across a simulated 32-layer model (total weights 320MB, peak RAM <= 20MB).
   • Verified error handling for missing/unregistered layers.

Verification

• Ran unit tests:

  .venv/Scripts/pytest.exe src/exo/shared/tests/test_sliding_window_streamer.py
  # 4 passed in 0.15s

• Linting and type checking:

  .venv/Scripts/ruff.exe check src/exo/shared/types/streaming_layer.py src/exo/worker/engines/streaming/layer_streamer.py src/exo/worker/engines/streaming/__init__.py src/exo/shared/tests/test_sliding_window_streamer.py
  # All checks passed!

Related Issues

• Resolves: Frontier Dimension 5.2 in 人人都能跑llm.md (AirLLM-Style Sliding Window Layer Streaming)