Support explicit Ollama model runs

.mex/ROUTER.md

@@ -0,0 +1,79 @@
+---
+name: router
+description: Session bootstrap and navigation hub. Read at the start of every session before any task. Contains project state, routing table, and behavioural contract.
+edges:
+  - target: context/architecture.md
+    condition: when working on system design, integrations, or understanding how evaluation and paper components connect
+  - target: context/stack.md
+    condition: when working with Python, LaTeX, evaluation libraries, or making technology decisions
+  - target: context/conventions.md
+    condition: when writing or reviewing Python or LaTeX changes
+  - target: context/decisions.md
+    condition: when making an architectural or technology choice, or understanding why one was made
+  - target: context/setup.md
+    condition: when setting up the environment, compiling the paper, or running project commands
+  - target: context/evaluation.md
+    condition: when working on datasets, models, multi-turn protocols, metrics, or result artifacts
+  - target: patterns/INDEX.md
+    condition: when starting any task and looking for a repeatable workflow
+last_updated: "2026-08-08"
+---
+
+# Session Bootstrap
+
+If you haven't already read `.mex/AGENTS.md`, read it now — it contains the project identity, non-negotiables, and commands.
+
+Then read this file fully before doing anything else in this session.
+
+## Current Project State
+
+**Working:**
+- The repository contains a modular LaTeX paper workflow with `main.tex`, sections, figures, tables, macros, and bibliography.
+- `Makefile` and `build.ps1` define paper compilation and related PDF workflows.
+- The project goal is defined: evaluate whether LLM performance degrades in multi-turn conversations.
+- `runs/run_experiment.py` evaluates 160 creative-writing tasks across six
+  domains using full and sharded multi-turn conditions.
+- The runner supports both LM Studio and Ollama. Collaborators select explicit
+  model IDs with `--backend {lmstudio,ollama} --models …`; every run is capped
+  at 16,384 context tokens and omits the reasoning field.
+  It appends every raw response immediately to a readable per-model JSONL file
+  and an independent combined JSONL dataset; `results/index.json` records
+  provenance and progress. `--resume` is reserved for a known interrupted run
+  in the active results directory.
+  Previous runs are isolated by reasoning mode under `runs/results/`.
+- Do not use `runs/context error/` as run history.
+
+**Not yet built:**
+- The workflow that transfers verified results into paper text, figures, or tables.
+
+**Known issues:**
+- LM Studio model IDs must be obtained from the running server; UI display names may be truncated or differ from API IDs.
+- The project must remain an experiment-and-paper repository rather than a product.
+
+## Routing Table
+
+| Task type | Load |
+|-----------|------|
+| Understanding how the evaluation and paper workflow works | `context/architecture.md` |
+| Working with Python, LaTeX, or evaluation technologies | `context/stack.md` |
+| Writing or reviewing code or paper files | `context/conventions.md` |
+| Making a design or technology decision | `context/decisions.md` |
+| Setting up, compiling, or running the project | `context/setup.md` |
+| Working on datasets, protocols, metrics, or result artifacts | `context/evaluation.md` |
+| Running a dataset-based multi-turn evaluation | `patterns/run-evaluation.md` |
+| Updating the paper with verified results | `patterns/update-paper-results.md` |
+| Any specific task | Check `patterns/INDEX.md` for a matching pattern |
+
+## Behavioural Contract
+
+For every task, follow this loop:
+
+1. **CONTEXT** — Load the relevant context file(s) from the routing table above. Check `patterns/INDEX.md` for a matching pattern. If one exists, follow it. Narrate what you load: "Loading architecture context..."
+2. **BUILD** — Do the work. If a pattern exists, follow its Steps. If you are about to deviate from an established pattern, say so before writing code — state the deviation and why.
+3. **VERIFY** — Load `context/conventions.md` and run the Verify Checklist item by item. State each item and whether the output passes. Do not summarise — enumerate explicitly.
+4. **DEBUG** — If verification fails or something breaks, check `patterns/INDEX.md` for a debug pattern. Follow it. Fix the issue and re-run VERIFY.
+5. **GROW** — After meaningful work, run this binary checklist:
+   - **Ground:** What changed in reality? Name the changed behavior, system, command, dependency, or workflow.
+   - **Record:** If project state changed, update the "Current Project State" section above. If documented facts changed, update the relevant `context/` file surgically.
+   - **Orient:** If this task can recur and no pattern exists, create one in `patterns/` using `patterns/README.md`, then add it to `patterns/INDEX.md`. If a pattern exists but you learned a gotcha, update it.
+   - **Write:** Update the `last_updated` marker in every scaffold file you changed when a real project date is available.

.mex/context/architecture.md

@@ -0,0 +1,62 @@
+---
+name: architecture
+description: How the evaluation workflow and collaborative paper workspace connect. Load when working on system design, integrations, or understanding how components interact.
+triggers:
+  - "architecture"
+  - "system design"
+  - "evaluation flow"
+  - "how does an experiment connect to the paper"
+edges:
+  - target: context/stack.md
+    condition: when specific Python, evaluation, or LaTeX technology details are needed
+  - target: context/evaluation.md
+    condition: when the dataset, multi-turn protocol, metrics, or result contract is involved
+  - target: context/conventions.md
+    condition: when implementing a new evaluation or editing paper-supporting code
+  - target: patterns/run-evaluation.md
+    condition: when executing or reproducing an evaluation run
+  - target: patterns/update-paper-results.md
+    condition: when verified evaluation results are being transferred into the paper
+grounds_to: []
+last_updated: "2026-08-07"
+---
+
+# Architecture
+
+## System Overview
+
+A dataset is supplied to a Python evaluation workflow.
+The workflow evaluates whether LLM performance degrades across multi-turn conversations.
+The evaluator entry point is `runs/run_experiment.py`; it uses LM Studio's
+model interface and a tracked `runs/models.json` sequence with per-model
+context lengths. Every generated raw response is written immediately to a
+self-contained `results/<run-id>/outputs.json` document and to the matching
+run group in `results/all.json`.
+`results/index.json` connects the run datasets and records proof metadata for
+review and exact resume.
+Collaborators use verified results while editing the modular LaTeX paper files.
+The existing Makefile or `build.ps1` compiles `main.tex` and its sections into the paper PDF.
+
+## Key Components
+
+- **Python evaluation workflow** — `runs/run_experiment.py` runs dataset-based multi-turn LLM evaluations.
+- **Evaluation dataset** — `runs/benchmark_data.json` supplies 40 creative-writing tasks across romance and mystery.
+- **Result artifacts** — one run directory per model/configuration,
+  `results/all.json` for independent analysis across all runs, and
+  `results/index.json` for run provenance, progress, compact record digests,
+  and hashes.
+- **LaTeX paper workspace** — `main.tex`, `sections/`, `figures/`, `tables/`, and `macros/` hold the collaborative manuscript.
+- **Paper build pipeline** — `Makefile` and `build.ps1` compile the LaTeX source into a PDF.
+
+## External Dependencies
+
+- **Python runtime** — runs the evaluator; the supported version is [TO BE DETERMINED].
+- **LLM inference access** — LM Studio provides model listing and chat completion over its OpenAI-compatible HTTP API; the endpoint and optional bearer token are configurable.
+- **LaTeX distribution with `latexmk`** — compiles the paper; the supported distribution and version are [TO BE DETERMINED].
+- **Git repository** — supports collaboration and review of Python, results, and `.tex` changes.
+
+## What Does NOT Exist Here
+
+- No product UI, production service, or user-facing application.
+- No database or persistent service layer has been selected; this is [TO BE DETERMINED].
+- No production deployment or over-engineered platform architecture.

.mex/context/evaluation.md

@@ -0,0 +1,90 @@
+---
+name: evaluation
+description: Experimental protocol and result contract for measuring LLM performance across multi-turn conversations. Load when changing datasets, prompts, models, metrics, or evaluation outputs.
+triggers:
+  - "evaluation"
+  - "experiment"
+  - "multi-turn"
+  - "dataset"
+  - "metric"
+  - "benchmark"
+edges:
+  - target: context/architecture.md
+    condition: when the evaluation flow or its connection to the paper changes
+  - target: context/stack.md
+    condition: when choosing Python libraries, model clients, or analysis tools
+  - target: context/conventions.md
+    condition: when implementing or reviewing evaluation code and result artifacts
+  - target: patterns/run-evaluation.md
+    condition: when executing an evaluation or reproducing a result
+  - target: patterns/update-paper-results.md
+    condition: when transferring verified evaluation results into the paper
+grounds_to: []
+last_updated: "2026-08-08"
+---
+
+# Evaluation
+
+## Purpose
+
+Measure whether LLM performance degrades in multi-turn conversations for the research paper.
+
+## Protocol
+
+- Dataset source and schema: `runs/benchmark_data.json`, containing 160 items
+  across comedy, fantasy, historical fiction, mystery, romance, and science
+  fiction; each item has one full instruction and 5–9 ordered shards.
+- Models and inference interface: collaborators select exact model IDs with
+  `--backend lmstudio|ollama --models MODEL …`. Every run uses a fixed 16,384
+  context-token cap and omits the reasoning field. LM Studio loads one model at
+  a time through native `POST /api/v1/models/load`, sends completions through
+  `POST /v1/chat/completions`, and unloads the returned instance. Ollama uses
+  `GET /api/tags` for model discovery and `POST /api/chat` with `num_ctx` set
+  to the same cap. The explicit selection is recorded in `index.json` so the
+  dashboard can project a batch ETA from the active model's observed rate.
+- Context policy: use at most 32K for models under 20B parameters and 16K for
+  models at or above 20B. Mistral 7B Q8 also uses 16K: at 32K its GPU KV cache
+  required an additional 4 GiB and failed to allocate on the available GPU.
+- Active run policy: every explicit model selection uses 16,384 context tokens
+  and sends no `reasoning_effort` field. Keep result sets from different
+  reasoning modes separate: the earlier partial high-reasoning attempt is
+  archived in `runs/results/reasoning_on/`, while non-reasoning baselines are
+  archived in `runs/results/reasoning_off/`; the brief unsupported `none`
+  attempt is isolated in `runs/results/reasoning_none/`. The fresh
+  non-reasoning sequence writes to `runs/results/`.
+- Conversation prompts, turn count, and state handling: The `full` condition
+  sends the complete instruction in one user turn. The `sharded` condition
+  sends each shard as a separate user turn, retaining every assistant reply in
+  the conversation and scoring the final reply.
+- Baselines and comparison conditions: archived or error-context artifacts are
+  not experiment evidence and must not be treated as saved runs. A normal
+  invocation starts a fresh, versioned raw-result run; it does not mix or
+  silently resume legacy files. Use `--resume` only when deliberately
+  recovering a known interrupted run in the active results directory.
+
+## Metrics and Results
+
+- Primary performance metric: [TO BE DETERMINED — populate after first implementation].
+- Supporting metrics and aggregation: [TO BE DETERMINED — populate after first implementation].
+- Result artifacts and storage location: the runner appends each raw generated
+  response to `results/results_<model>.jsonl` and to
+  `results/all_results.jsonl`, the independent combined dataset. Every JSONL
+  line contains `model_id`, `model_name`, `parameters`, and `quant_file` as
+  well as the task, condition, turn, text, and metrics. `results/index.json`
+  is the only metadata file; it records model configuration, lifecycle,
+  progress, and integrity checks. Result artifacts contain no absolute local
+  paths. If a process stops between the two writes, `--resume` reconciles the
+  matching record IDs without regenerating output.
+- Minimum metadata needed to reproduce a result: the raw record plus its
+  matching `results/index.json` entry, which supplies the exact run ID, model
+  identity, context, inference settings, dataset/protocol/runner hashes,
+  lifecycle, and file hashes needed for independent review.
+
+## Boundaries
+
+- This domain supports research experiments and paper reporting, not a product or production service.
+- Do not change the dataset, model, prompt, turn count, or metric silently between runs.
+- Keep experiment-specific complexity in this context and its patterns rather than spreading it across general architecture.
+- Treat raw results as immutable evidence. Future scoring or annotation writes
+  separate derived artifacts and never overwrites the per-model JSONL file or
+  `all_results.jsonl`.

.mex/context/setup.md

@@ -0,0 +1,74 @@
+---
+name: setup
+description: Development environment setup and commands for running evaluations and compiling the paper.
+triggers:
+  - "setup"
+  - "install"
+  - "environment"
+  - "getting started"
+  - "how do I run"
+edges:
+  - target: context/stack.md
+    condition: when specific technology versions or libraries are needed
+  - target: context/architecture.md
+    condition: when understanding how setup commands fit the evaluation-to-paper flow
+  - target: context/evaluation.md
+    condition: when configuring a dataset, model access, evaluation protocol, or result output
+  - target: patterns/run-evaluation.md
+    condition: when setup is complete and an evaluation run is being executed
+grounds_to: []
+last_updated: "2026-08-08"
+---
+
+# Setup
+
+## Prerequisites
+
+- Python: Python 3.10+ is recommended; the runner uses only the standard library.
+- A LaTeX distribution with `latexmk`: supported distribution/version is [TO BE DETERMINED].
+- Git for collaborative repository work.
+
+## First-time Setup
+
+1. Clone or open the repository and confirm the Python environment: [TO BE DETERMINED].
+2. No Python package installation is required for the runner.
+3. Configure either LM Studio's local server or Ollama with the desired model.
+4. Compile the paper with `make` to verify the LaTeX toolchain.
+
+## Environment Variables
+
+- LM Studio endpoint: `LM_STUDIO_BASE_URL` (defaults to
+  `http://localhost:1234/v1`). Optional bearer token:
+  `LM_STUDIO_API_KEY` (defaults to `lm-studio`).
+- Ollama endpoint: `OLLAMA_BASE_URL` (defaults to `http://localhost:11434`).
+- Dataset location or identifier: [TO BE DETERMINED].
+- Evaluation configuration/output location: [TO BE DETERMINED].
+
+## Common Commands
+
+- `make` — compile the paper PDF.
+- `make watch` — continuously recompile the paper while editing.
+- `make clean` — remove LaTeX build artifacts and the generated PDF.
+- `make diff REF=HEAD~1` — generate a LaTeX diff against a Git revision.
+- `python3 runs/run_experiment.py --backend lmstudio --list-models` — list
+  exact LM Studio model IDs.
+- `python3 runs/run_experiment.py --backend ollama --list-models` — list exact
+  locally available Ollama tags.
+- `python3 runs/run_experiment.py --backend lmstudio --models MODEL_ID` — run
+  one or more explicit LM Studio model IDs.
+- `python3 runs/run_experiment.py --backend ollama --models MODEL_TAG` — run
+  one or more explicit Ollama tags. Every run uses a fixed 16,384-token cap.
+- Add `--resume` only to recover a known interrupted run in the current
+  `results/` directory. Do not infer resumable work from archived, legacy, or
+  error-context artifacts.
+- Add `--limit 3` for a smoke test. The default `--output results` appends to
+  `results/results_<model>.jsonl` and `results/all_results.jsonl`; every line
+  identifies its model, parameters, and quant file. `results/index.json` is
+  the single status and provenance index.
+
+## Common Issues
+
+- Evaluation setup or execution failures: Confirm the selected backend is
+  running, use its `--list-models` output to obtain exact IDs or tags, and
+  ensure each requested model is available before starting a long run.
+- LaTeX compilation failures: [TO BE DETERMINED after the first collaborative build issue].