learn_quant: SALT 35 paper reproduction layer, data, manuscript, and fixes - #74
Open
haberchr wants to merge 8 commits into
Open
learn_quant: SALT 35 paper reproduction layer, data, manuscript, and fixes#74haberchr wants to merge 8 commits into
haberchr wants to merge 8 commits into
Conversation
The README documents 'model=mvlstm' / 'model=transformer' overrides and learn.yaml declares 'defaults: - model: null', but the conf/model config group was silently excluded by the blanket 'model/' .gitignore pattern (intended for saved model artifacts), so the documented invocations fail on a fresh clone. Un-ignore conf/model/ and commit the two configs used for the SALT 35 experiments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
determine_start_index fell through to an implicit None when cfg.training had no 'resume' key (or when the resume expression was not found), and define_index_bounds returned (None, None + 1) -> TypeError when cfg.expressions.index was present but null. Return 0 as the explicit start-index fallback and require a non-null index. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scripts, docs, and notebooks to regenerate every figure and statistical result in the SALT 35 monotonicity manuscript from the committed runs CSV (outputs/combined_runs_AOC_monotonicity_updated.csv) — no database, MLflow store, or training runs required: - scripts/reproduce_figures.py: paper Figure 1 (exact plotnine reproduction) + length/function-count twins + complexity-vs-learning - scripts/deeper_analysis.py: partial correlations, directional monotonicity, per-operator Ridge, nested models (tables to analysis/tables/) - scripts/review_extensions.py: 2026-07 review figures and diagnostics - scripts/reproduce_from_postgres.py, verify_postgres_matches_csv.py: optional provenance checks; the Postgres DSN is read from the MLFLOW_PG_DSN env var (no credentials in the repo) - REPRODUCE.md: full walkthrough with headline results - FIGURES_TECHNICAL.ipynb: cell-by-cell companion for every number, including the mixed models reproducing published Table 6 - notebooks/reproduce_figures_and_analysis.ipynb: one-shot Run All - notebooks/get_AOC.ipynb, get_experiment_data.ipynb: the queries that originally exported the runs CSV (provenance) Scripts log paths relative to the package root so tee'd tables and notebook outputs are machine-independent. README gains a rewritten introduction and a pointer to REPRODUCE.md. All paths point at outputs/ where this repo keeps the runs CSV. Verified: all three scripts run clean from src/examples/learn_quant/. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- outputs/monotonicity_values_redl.csv: degree + four directional monotonicity senses recomputed for all 9550 generated expressions. Strict superset of outputs/monotonicity_values.csv (54 expressions); values agree exactly on shared expressions and on the 2000 trained expressions in combined_runs_AOC_monotonicity_updated.csv. - Expression_to_Run_Hashes.csv: expression -> MLflow run-hash map used to join training runs to expressions (provenance for the runs CSV). - ignore_hashes.txt: run hashes excluded from the published analysis. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Camera-ready of 'Quantifiers that are more monotone are easier to learn' (Proceedings of SALT 35: 286-309, 2025), with the regenerated figures. The README's placeholder links used reversed markdown syntax and pointed at a paper.pdf that was never committed; add the full citation and point to REPRODUCE.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
analysis/tables/*.txt are the full OLS/Ridge/nested-model results that REPRODUCE.md documents as committed artifacts. The figures/ directory is fully regenerated by scripts/reproduce_figures.py + deeper_analysis.py + review_extensions.py (~24 MB of PNGs), so it stays untracked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reproduces the appendix figure 'First Step vs. AUC, Shaded by Model' (learning speed vs learning difficulty, per-model linear fits with SE bands) from the verified runs CSV, and recomputes its caption statistic: partial Spearman correlation of first_step and val_loss_step_AOC controlling for model (binary covariate), via the first-order partial-correlation formula on pairwise Spearman coefficients (no pingouin dependency). Reproduced r = 0.893 on n = 4068 converged runs vs the published caption's r = 0.892; the statistic is printed and saved to analysis/tables/10_first_step_auc_partial_spearman.txt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
haberchr
force-pushed
the
learn-quant-paper-reproduction
branch
from
July 26, 2026 18:49
c3ef5c7 to
a8e6add
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds everything needed to reproduce the SALT 35 paper Quantifiers that are more monotone are easier to learn (Proceedings of SALT 35: 286–309, 2025) from data committed in this repo, plus two small fixes uncovered along the way.
What's included
conf/model/Hydra configs were missing — the README documentsmodel=mvlstm/model=transformer, but the blanketmodel/.gitignore pattern (meant for saved model artifacts) had silently excluded the config group, so those invocations fail on a fresh clone. Un-ignored and committed (mvlstm.yaml,transformer.yaml).util.pyindex helpers could returnNone(determine_start_indexwithout aresumekey;define_index_boundswith a present-but-null index).REPRODUCE.md(full walkthrough),scripts/reproduce_figures.py/deeper_analysis.py/review_extensions.py(all figures + stat tables from the committed runs CSV),FIGURES_TECHNICAL.ipynb(cell-by-cell companion incl. mixed models reproducing Table 6), one-shot notebook, and the provenance notebooks that originally exported the CSV. No database, MLflow store, or training runs required — verified end-to-end from a fresh clone.outputs/monotonicity_values_redl.csv(degree + 4 directional senses for all 9550 generated expressions; strict superset of the existing 54-row file, identical values on shared expressions), expression↔run-hash map, excluded-hash list, and the committed stat tables.manuscript/, README links fixed (were reversed markdown pointing at a missing file).MLFLOW_PG_DSN; no credentials in the repo. Scripts log repo-relative paths so committed tables are machine-independent.Key reproduced numbers
AUC ~ degree * model + (1|expression)reproduces published Table 6 to ~0.2%🤖 Generated with Claude Code