Add evaluation

.gitignore

@@ -28,4 +28,7 @@ Cargo.lock
 *.pdb
 
 # Results log files from analysis script
-*.log
+*.log
+
+# Python virtual environment
+venv/

README.md

@@ -6,7 +6,7 @@
 [![GitHub CI][github-ci-shield]][github-ci]
 
 This repository contains the artifact for the paper "Meta-monomorphizing Specializations".
-The artifact consists of two main components: a prototype implementation and an instrumentation tool for empirical analysis.
+The artifact consists of three main components: a prototype implementation, an instrumentation tool for empirical analysis, and a benchmark evaluation suite.
 
 ---
 
@@ -18,6 +18,7 @@ The artifact consists of two main components: a prototype implementation and an
 - [Getting Started](#getting-started)
   - [Prototype Implementation (`spec-trait-impl`)](#prototype-implementation-spec-trait-impl)
   - [Instrumentation Tool (`spec-trait-inst`)](#instrumentation-tool-spec-trait-inst)
+  - [Evaluation Benchmarks (`spec-trait-evaluation`)](#evaluation-benchmarks-spec-trait-evaluation)
 - [Reproducing the Empirical Evaluation](#reproducing-the-empirical-evaluation)
 - [Pre-computed Results](#pre-computed-results)
 - [Contact](#contact)
@@ -32,6 +33,8 @@ This artifact provides:
 
 2. **`spec-trait-inst/`** — An instrumentation tool built as a `cargo`/`rustc` wrapper that statically analyzes Rust projects to identify trait specialization opportunities.
 
+3. **`spec-trait-evaluation/`** — A benchmark suite that compares `spec!`/`#[when]` compile-time specialization against alternative dispatch strategies (naive, enum, runtime `TypeId`), measuring runtime performance, build time, and binary size. See the [evaluation README](spec-trait-evaluation/README.md) for details.
+
 ---
 
 ## Repository Structure
@@ -45,9 +48,15 @@ spec-trait/
 │       ├── spec-trait-order/     # Compile-time trait ordering
 │       └── spec-trait-utils/     # Shared utilities
 │
-└── spec-trait-inst/              # Instrumentation tool
-    ├── src/                      # Source code
-    └── tests/
+├── spec-trait-inst/              # Instrumentation tool
+│   ├── src/                      # Source code
+│   └── tests/
+│
+└── spec-trait-evaluation/        # Benchmark evaluation suite
+    ├── benches-crate/            # Criterion benchmarks
+    ├── notebooks/                # Analysis notebooks
+    ├── results/                  # Pre-computed CSV results
+    └── scripts/                  # Metric collection scripts
         ├── scripts/              # Analysis scripts and results
         │   ├── analysis.sh       # Main analysis script
         │   ├── repos.txt         # List of repositories analyzed

spec-trait-evaluation/.vscode/settings.json

@@ -0,0 +1,3 @@
+{
+    "python-envs.defaultEnvManager": "ms-python.python:system"
+}

spec-trait-evaluation/README.md

@@ -0,0 +1,95 @@
+# spec-trait Evaluation
+
+Benchmark suite that compares `spec!`/`#[when]` compile-time specialization against three alternative dispatch strategies:
+
+| Approach | Description |
+|---|---|
+| **spec** | Compile-time specialization via `spec!`/`#[when]` macros |
+| **naive** | Plain generic fallback (no specialization) |
+| **enum** | Enum-based match dispatch |
+| **runtime_typeid** | `TypeId`-based runtime branching |
+
+## Examples
+
+Each example implements the same operation with 4 dispatch approaches.
+The `spec` variant exploits type-specific knowledge (e.g. memcpy, SIMD byte search, counting sort) that a naive generic fallback cannot use.
+
+| # | Operation | `when` condition | spec strategy | Naive fallback |
+|---|-----------|-----------------|---------------|----------------|
+| ex01 | extend_from_slice | `T: Copy` | `ptr::copy_nonoverlapping` (memcpy) | clone loop |
+| ex02 | to_bytes | `T: Copy` | raw `transmute` (zero-copy) | `format!()` per element |
+| ex03 | contains | `T = u8` | `memchr` SIMD | `iter().any()` |
+| ex04 | sort | `T = u8` | counting sort `[u32; 256]` | `sort_unstable()` |
+| ex05 | histogram (max freq) | `T = u8` | `[u32; 256]` direct index | `HashMap<&T, u32>` |
+| ex06 | dedup (keep first) | `T = u8` | `[bool; 256]` bitmap | `HashSet` |
+| ex07 | hash_key | `K = u64` | identity XOR (no hashing) | `DefaultHasher` (SipHash) |
+| ex08 | count_value | `T = u8` | `memchr_iter().count()` | `filter().count()` |
+
+## Project structure
+
+```
+spec-trait-evaluation/
+├── benches-crate/             # Rust crate with benchmarks
+│   ├── Cargo.toml
+│   ├── build.rs
+│   ├── benches/
+│   │   └── bench.rs           # Criterion harness (8 benchmark groups)
+│   └── src/
+│       ├── lib.rs
+│       ├── data.rs            # Input generators (make_u8, make_i32, ...)
+│       └── examples/          # 32 source files (8 examples × 4 variants)
+├── scripts/
+│   ├── run_all.sh             # Run benchmarks + collect all metrics
+│   ├── collect_metrics.py     # Parse Criterion JSON → runtime.csv
+│   ├── measure_build_time.sh  # Per-variant cargo check timing → build_time.csv
+│   └── measure_binary_size.sh # Per-variant release rlib sizes → binary_size.csv
+├── notebooks/
+│   ├── eval.ipynb             # Analysis notebook (plots + summary tables)
+│   └── requirements.txt       # Python dependencies
+└── results/
+    ├── runtime.csv
+    ├── build_time.csv
+    └── binary_size.csv
+```
+
+## Reproducing
+
+### Prerequisites
+
+- Rust nightly (edition 2024)
+- Python 3 with `pandas`, `matplotlib`, `seaborn`
+
+### Run individual measurements
+
+```bash
+# Runtime benchmarks (Criterion)
+cargo bench --manifest-path benches-crate/Cargo.toml --bench bench
+cargo bench --manifest-path benches-crate/Cargo.toml --bench bench -- --warm-up-time 1 --measurement-time 3 --sample-size 10
+cargo bench --manifest-path benches-crate/Cargo.toml --bench bench -- --warm-up-time 1 --measurement-time 3 --sample-size 10 <name_filter>
+
+# Collect runtime CSV from Criterion output
+python3 scripts/collect_metrics.py \
+  --criterion-dir benches-crate/target/criterion \
+  --output results/runtime.csv
+
+# Compile time (3 iterations, 8 examples)
+bash scripts/measure_build_time.sh 3 8
+
+# Binary size
+bash scripts/measure_binary_size.sh 8
+```
+
+### Notebook
+
+```bash
+pip install -r notebooks/requirements.txt
+jupyter notebook notebooks/eval_pretty.ipynb
+```
+
+## Benchmark parameters
+
+- **Input sizes**: 256, 1024, 4096, 16384, 65536 elements
+- **Data type**: `u8` (most examples), `u64` (ex07 hash_key)
+- **Criterion config**: warm-up 3s, measurement 5s, 100 samples (default); `--quick` uses 1s/3s/10
+- **Build time**: `cargo check` from clean local crate (dependencies pre-cached), 3 iterations
+- **Binary size**: release-mode `rlib` artifact (`opt-level=3`, `lto=false`, `codegen-units=16`)

spec-trait-evaluation/benches-crate/Cargo.toml

@@ -0,0 +1,23 @@
+[package]
+name = "spec-trait-evaluation"
+version = "0.1.0"
+edition = "2024"
+
+[dependencies]
+spec-trait-macro = { path = "../../spec-trait-impl/crates/spec-trait-macro" }
+memchr = "2"
+
+[dev-dependencies]
+criterion = { version = "0.5", features = ["html_reports", "csv_output"] }
+
+[build-dependencies]
+spec-trait-order = { path = "../../spec-trait-impl/crates/spec-trait-order" }
+
+[profile.release]
+opt-level = 0
+lto = false
+codegen-units = 16
+
+[[bench]]
+name = "bench"
+harness = false