Config overhaul: one YAML file, vocabularies declared once

[dmarx]

Draft — the engine is converted and working, the test corpus is not. Status and remaining work at the bottom; please read that before reviewing depth-first.

Closes #254 and #256 when finished.

What is done

One luria.yaml. TOML is gone, not deprecated. tomllib is out of config.py, upgrade.py and remotes.py.

A central vocabularies: table, referenced by name — this is the part with a measured bug behind it:

vocabularies:
  record-statuses:
    Active: {label: Current, blurb: in force}

schemes:
  ADR:
    statuses: record-statuses
  DP:
    statuses: record-statuses      # the same words, said once

Scheme.tags_yaml/.statuses_yaml (paths) become .tags/.statuses (values); Vocabulary.file becomes .values_by_name. Nothing downstream opens a vocabulary file, because there is no longer one to open.

omegaconf underneath the validation, not in place of it. OmegaConf.merge replaces the hand-rolled _merge over DEFAULTS and types the result. The 44 raise ValueErrors stay — they are cross-field semantic rules, and a structured config validates the shape of a value, not a relationship between two. Trading luria's errors for a schema complaint about a key of the wrong type would make the tool worse exactly when a user is confused.

Reference gains label/blurb (#254) — the two keys a vocabulary value already carried.

label has one fallback instead of three. It was tag.title() in the tag pages, "" in the status legend, and the raw value in the vocabulary pages, so a scheme declaring no label got a title-cased heading and an empty legend cell.

The evidence for centralizing

Four byte-identical copies of one statuses.yaml across the two records — ADR and DP here, the same pair again in anthology-of-the-sota.

And the one that matters: that record's ADR-026 decided its practice registry and reading list share one topic vocabulary, and says so in its CLAUDE.md. On disk, two files, same thirteen keys, ten of thirteen blurbs different. Two edits to one vocabulary, months apart, that nothing could reconcile because nothing knew they were the same vocabulary.

luria already knew this failure mode from the other direction — primary_tags' docstring records "seven terms across four places, and the blurbs for the same tag already disagreed between two of them (ADR-060)". ADR-060 fixed the copies within a scheme; the copies between schemes were unreachable.

This repo's own config is converted, and ADR/DP now share one vocabulary declared once.

Verification so far

• luria lint runs end to end on this repository under the new config. Its only findings are three stale links in generated views, which regenerate.
• 562 tests pass, 556 fail. Every failure I have inspected is a fixture writing a TOML luria.toml; the lint run above is why I believe the library itself is sound, but I have not audited all 556.

What is NOT done — do not merge

1. The test corpus. 562/556 as above. The shapes: ~16 module-level TOML constants, ~18 f-string configs, and ~17 sites that concatenate a base config with an extra fragment. The last group is the real work — TOML tables concatenate, YAML mappings do not, so BASE + extra needs restructuring rather than a codemod. 8 sites are converted; the rest are not.
2. init.py still writes a commented luria.toml plus per-scheme vocabulary files. Its generator is a rewrite, not a port.
3. migrate.py's spec reader still uses tomllib (unreached by the lint run).
4. config_doc.py — the generated configuration reference — is unreviewed against the new shape.
5. docs/*.md still say luria.toml in ten files.
6. No converter for downstream records. anthology-of-the-sota cannot upgrade without one, and the hazard it must own is escaping: uid = "(\\d{4})[.:](\\d{4,5})" does not survive TOML → YAML by copying bytes. Round-tripping every uid/title_re in a real record is the test, not a fixture. The throwaway script I used for this repo is not that converter.
7. A changelog fragment and devlog entry. ADR-tmp8hp25 is here; the other two are not.

Roughly: the design is settled and demonstrated, and the long tail of mechanical conversion is ahead. I stopped here rather than push a half-migrated tree without saying so.

ADR-tmp8hp25 carries the decision and the rejected alternatives — keeping vocabularies local (withdrawn on @dmarx's evidence), central-with-an-override (rejected: the override is what permits the drift), dual-reading both formats (rejected on instruction), and letting structured configs replace the semantic checks.

---

Generated by Claude Code

[dmarx]

Status: 1106 passing, 11 failing (was 345 / 773 at the low point)

Pushed. The tail is nearly closed and the remainder is a short, characterised list — including one item I deliberately stopped on rather than hack.

Done since the draft

The test corpus, via four AST codemods plus hand work where the shape resisted:

• merged() in tests/_config.py replaces textual composition. TOML tables concatenate and YAML mappings do not — two fragments both declaring schemes: is a duplicate key, not a merge — so BASE + extra became merged(BASE, extra) with a deep merge. That is the better shape anyway: a fixture that says which keys it adds says something a reader can check.
• Several fixtures changed from string surgery to dicts where the old TOML was being built by interpolation (f'requires = [{terms}]' → {"lint": {"narrow_terms": [...]}}).

examples/ — all seven projects converted, 46/46 green. Three had a vocabulary two schemes shared as two files, now one declaration each: constitution (PRACTICE and BOUNDARY on VALUE's statuses), knowledge-base (SOTA on LIT's tags), rfcs-and-specs (SPEC on RFC's statuses).

init.py scaffolds YAML and no longer writes a statuses.yaml per scheme. Appending a scheme now inserts inside the schemes: mapping — an indented block at the end of a YAML document attaches to whatever the last top-level key happens to be, silently.

migrate.py reads YAML specs; record/migrations.d/0001-promote.toml converted. upgrade.py, new.py, remotes.py and the prose across the library now say luria.yaml.

luria/migrate_config.py converts a downstream record. It parses with tomllib and writes with yaml rather than moving bytes — which is what makes a regex in a uid survive — and it leaves the TOML in place, because deleting what you just converted before anyone has read the result is not a migration anybody should trust.

A codemod bug worth knowing about

A fragment whose first lines were bare keys — 'requires = ["source"]\n' '[luria.schemes.SOTA.references]\n…' — put those keys at the TOML document root, where parsed.get("luria", parsed) discarded them. So several fixtures silently lost their requires and the tests asserting on it failed for a reason that looked like a behaviour change. Found by diffing against the original; eight sites affected, most now restored.

What is still failing (11)

test_contract.py          4   fixtures still missing a dropped `requires`
test_statuses.py          3   `luria upgrade statuses` — see below
test_record_doc.py        2   assertions on generated config text
test_doc_refs.py          1   repo docs still say luria.toml in prose
test_migrations.py        1   rename_scheme — the real one

test_migrations.py::test_rename_scheme_end_to_end is a design question, not a fixture. rename_scheme edits the config by string-replacing TOML section headers:

plan.section_renames.append((config_file, f"[luria.schemes.{old}]", f"[luria.schemes.{new}]"))

A fully-qualified TOML header is unambiguous. A YAML key is not: FXL: appears under schemes: and again under each remotes.<R>.schemes:, and the test correctly asserts that renaming this project's scheme leaves another project's mirror alone. Doing that properly means path-aware editing — walk to schemes and remotes.<R>.schemes, rename those keys only. Round-tripping through the parser would lose every comment in a user's config, which I don't think is acceptable for a migration. I stopped rather than guess at which trade you want.

luria upgrade statuses may simply be obsolete now. It exists to write a statuses.yaml per scheme into a record that predates the declaration — and there is no such file any more. Its own lint output already says "has nothing left to do here — remove it at 1.0.0". I repointed it at the central vocabulary, but deleting it is probably the honest answer, and that is your call.

Still not done

• config_doc.py — the generated configuration reference — is unreviewed against the new shape.
• docs/*.md still say luria.toml in prose.
• A changelog fragment and devlog entry (ADR-tmp8hp25 is written).
• luria index has not been run, so the views in this branch are stale by design.

One environment note

omegaconf is a new hard dependency and it does not install on a Debian-patched setuptools — antlr4-python3-runtime, its dependency, fails to build with AttributeError: install_layout. It installs fine under modern setuptools in a clean venv, which is what I used. Worth deciding whether that is acceptable before merging, since it will bite some contributor.

---

Generated by Claude Code

[dmarx]

Green, and out of draft: 1143 passed, 33 skipped, 0 failed

luria lint is clean on this record, with 0 unacknowledged references. Merged origin/main on the way — #251, #252 and #255 all landed while this was in progress, and the one conflict was in _checked_converses, where #255 made the converse resolve against the far scheme and this branch had renamed the file in the same message. Both, obviously; the lookup is #255's.

The three real bugs, and they were one fact

A TOML table header is fully qualified. A YAML key is not.

[luria.remotes.SG.schemes.FXL]      # says exactly where it is

      FXL:                          # says only "one level in from here"

Everything that edited a config as text was built on the first fact.

rename_scheme renamed nothing. It string-replaced [luria.schemes.FXL], a string that does not occur in a YAML config. Renaming the bare key would have been worse: FXL: appears under schemes: and under every remotes.<R>.schemes:, and the whole point of the section-aware pass is that renaming our scheme leaves another project's mirror alone. rename_key_at(text, path, old, new) now tracks the open-mapping stack by indentation and renames only where the stack equals path — line editing, so a project keeps its comments.

config_paths_pass rewrote paths belonging to other projects. It found the enclosing section with a TOML-header regex, so under YAML section stayed "", nothing was ever frozen, and an unclaimed remote's own document: path got swept with everyone else's. This one is worse than the rename: a no-op is visible the moment anyone looks; this wrote a wrong value and looked like success.

luria upgrade statuses wrote dotted keys. schemes.VP.statuses: statuses appended to a YAML document is a key literally named schemes.VP.statuses. It now writes the declaration into each scheme's block and declares the vocabulary it names.

Same fact broke the fixtures (BASE + extra had to become a merge) and luria init (an indented block appended to a document attaches to whatever the last top-level key happens to be).

The last thing to go wrong, which said nothing about paths

The shared vocabulary was called record-statuses — the better name. A vocabulary's per-value pages render at <view>/<name>/, so the rename moved docs/decisions/statuses/ to docs/decisions/record-statuses/ and orphaned the old directory.

What that broke: the bare-reference check skips generated views, and recognises them by generated-output path. An orphan is not one — so pages generated for a year started being scanned as hand-written prose, and every code in them was reported bare. A vocabulary's name is a published path. It is called statuses.

I only found this because test_repo_docs_have_no_bare_references passed on origin/main and failed here against a byte-identical view file. Worth knowing before anyone renames a vocabulary in a real record.

Docs

Every config example in docs/ is YAML, and the ones in config.py's docstrings — which docs/configuration.md is generated from — with them. The inline [luria.x.y] spelling is now the path it names, x.y. Prose says luria.yaml. Nothing generated is committed here.

Also fixed

A codemod bug worth recording: a fragment whose first lines were bare keys put them at the TOML document root, where parsed.get("luria", parsed) discarded them. Several contract fixtures silently lost their requires and then failed in a way that looked like a behaviour change. Found by diffing against the pre-migration file; all restored.

Two things still for you

omegaconf does not install on a Debian-patched setuptools — antlr4-python3-runtime fails with AttributeError: install_layout. Fine under modern setuptools in a clean venv, which is what I used throughout, but it is a new hard dependency and it will bite someone.

luria upgrade statuses may be dead. It exists to write a per-scheme statuses.yaml into a record that predates the declaration, and there is no such file any more. Its own lint output already says "has nothing left to do here — remove it at 1.0.0". I repointed it rather than delete it, but deleting is probably the honest answer and that is your call.

ADR-tmp8hp25, a changelog fragment and a devlog entry are filed.

---

Generated by Claude Code

[dmarx]

luria upgrade yaml — you were right, and it was worse than missing

The converter existed as luria/migrate_config.py and was not reachable: not imported anywhere, not in COMMANDS, and — after my blanket luria.toml → luria.yaml rename swept through it — reading luria.yaml and parsing it as TOML. Dead code that would have been broken on arrival, for the one job a record on the old format actually needs.

It belongs in upgrade, whose own module docstring already says it:

Nothing here goes through config.load(). The config an upgrade repairs is the config the new version refuses to load, so a command that needed it would be unrunnable in exactly the situation it exists for.

That is precisely this case. So it gets the mechanism that comes with that home: a SUNSET entry saying what has to be true before it is deleted, --dry-run, and the spent-upgrade lint. And luria upgrade statuses now refuses on a record still on TOML — pointing at yaml — instead of reporting "nothing to upgrade", which was true and useless.

Verified against the real thing, not a fixture

Converting anthology-of-the-sota's actual config:

SOTA.tags -> sota-tags          LIT.tags   -> lit-tags
SOTA.statuses -> sota-statuses  ADR.tags   -> adr-tags
NOTE.statuses -> note-statuses  ADR.statuses -> adr-statuses
DP.statuses -> adr-statuses (the same words adr-statuses already holds)
SOTA.fields vocabulary consensus -> consensus

It collapsed DP.statuses into ADR's — a genuine duplicate — and left the two drifted tag vocabularies as two. That is the line I want it to hold: it collapses byte-identical copies, never opinions. Merging lit-tags and sota-tags would be the converter deciding something the record never said, and their blurbs differ in ten of thirteen entries.

The result loads, all five schemes, and ARXIV's uid still matches 1904.10509 — the escaping hazard, checked on the regex that motivated worrying about it.

Ten new tests, including the round-trip and an end-to-end "the config it writes actually loads". It leaves the TOML and the now-unread vocabulary files on disk and prints the git rm — deleting what you just converted, before anyone has read the result, is not a migration anybody should trust.

One more thing it surfaced, and it is the subtler half

Converting the anthology reproduced the orphaned-view trap at scale — every scheme's status pages moved, because pages rendered at <view>/<vocabulary-name>/. So sharing a vocabulary between two schemes — the entire point of this change — moved published pages.

A vocabulary's name is a config detail; a published path is not. Pages now render under the field's name, so no config change can move one. This moves <view>/statuses/ to <view>/status/ once, here, at a boundary a record crosses deliberately, rather than in a later release where it would surprise someone. The orphans are removed in this branch (CI regenerates views but cannot delete a directory nothing claims any more).

I considered keeping the path keyed on the vocabulary name and having the converter preserve the old stem — it moves nothing in the common case, and fails exactly where this decision aims: a record whose five schemes each have a different statuses.yaml cannot keep all five on the name statuses, so four move anyway. "Your pages move unless your vocabularies happen not to collide" is not a rule anybody can hold.

1153 passed, 33 skipped, 0 failed. luria lint clean, 0 unacknowledged references. ADR-tmp8hp25 carries both decisions and the rejected alternative.

---

Generated by Claude Code

[dmarx]

tags stops being built in too — b704bf6

The two reasons it held out are stated right there in the code, next to BUILT_IN_AXES:

tags stays: it is OPEN, and a vocabulary is closed by construction (ADR-054 deferred even a closed flag), and its tag_groups constrain a subset of values, which a vocabulary cannot express.

Both true. Both missing features rather than facts — which is why it was worth taking as a scope rather than a note.

What the declaration can say now

• closed on a vocabulary (default true). tags sets it false: the declared values supply order, label and blurb, and reaching for a new one stays an edit to a document. That is ADR-054's deferred flag, written.
• fields.<field>.groups replaces schemes.X.tag_groups, so a group constrains the field it is declared under.

Then one key is left:

schemes:
  SCENE:
    axis: worlds        # which field heads this scheme's index
    fields:
      worlds: {vocabulary: worlds, many: true, closed: false}

axis is not tags renamed. It names which of this scheme's own fields gets the categories block and a page per value — a rendering choice a world-bible answers differently from an anthology — validated eagerly against the fields the scheme declares. A scheme naming none has no taxonomy and renders none, which the old code could not express: every scheme had tags whether it wanted one or not, and every document was told to carry them.

A real bug fell out of moving the groups

group_violations read meta["tags"] whatever the group was declared about. With tag_groups scheme-level that was invisible, because a group could only ever be about tags. The moment a group can name another field, it's a rule that looks enforced and constrains nothing — which is the exact failure tag groups exist to remove. There's a test for it now (test_a_group_reads_its_own_field_not_a_key_called_tags).

Gone

BUILT_IN_AXES; tags from BUILT_IN_CONDITION_FIELDS (now status alone, and only because it has a default vocabulary no scheme has to declare); the "tags" literal in the plural set; the carve-out refusing fields.tags; and lint's "no tags: in frontmatter" branch, which is required: true on the axis field, checked in check_contracts with every other obligation.

Two consequences worth flagging

many: true now types a field on its own. It says the field holds a list, which is what makes it nameable in a chain or a derivation and gives the record page something to print — exactly what being built in used to say about tags. Without this, fields: {tags: {many: true}} was "declares no type".

Axis values are no longer lower-cased. Adr.tags did str(t).strip().lower(), a tags convention applied to every value — and it disagreed with the vocabulary check beside it, which has always compared the value as written. An axis of worlds with values A/B is the case it breaks. I checked the corpus first: 43 distinct tag values across record/ and examples/, none mixed-case, so nothing here moves. Worth knowing before someone adopts a capitalised vocabulary.

What did not change

No published path moves. Every scheme's axis is tags and the pages render under the axis's own name, so <view>/tags/ stays <view>/tags/. luria upgrade yaml writes the axis, the field and the moved groups, so a record crosses without hand-editing.

I deliberately did not collapse render_categories/render_tag_page into vocabularies.index_blocks/pages. They are close enough to look like duplication, and with tags a declared field they would now write into the same directory — so the axis is skipped in the generic path. But the categories block lists the documents under each value and the chip row does not, and choosing between them changes every project's index. That is its own decision, not this one.

1177 passing, luria lint clean with 0 unacknowledged references.

---

Generated by Claude Code

[dmarx]

The two renderers are one — 2463b71

tag_order, render_categories and render_tag_page are gone. vocabularies.py renders the axis with everything else. adr_index.py loses 58 lines net.

Once tags became a declared field they wrote into the same directory, so this wasn't really optional any more.

They had drifted — which is the argument for collapsing rather than picking one

• The label fallback. One answer since vocabularies.label_of, but the tag page reached it and the value page reached it differently.
• The blurb. The tag page sentence-cased it with a hand-rolled raw[:1].upper() — str.capitalize() having silently destroyed a blurb running to more than one sentence — while the value page rendered **Label** — blurb.
• A value the vocabulary does not declare. The axis showed it; every other field dropped it. Neither was reasoning about it. closed is now, and a field naming no vocabulary is open for the plainer reason that there's no closed set to fall outside of.

What's left of the difference is a shape the scheme chooses

Not one the code knows: the axis lists the documents under each value because it's the browsing surface; every other field is a row of chips, because the value's own page already holds the table. That follows from axis:, which the project declares.

Scheme.grouped_fields is the one answer to which fields those are, axis first. Three places needed it — which directories the generator owns, which are exempt from the docs index, which paths count as generated — and a fourth disagreeing with them is how a page becomes an orphan. (That third one had the axis and the vocabularies as two separate clauses; now it's one.)

Axis first because it heads the index. Worth saying because I got it wrong first: grouped_fields initially preserved the fields: table order, which put **By status:** above the tag categories on every index. The fields: table is written in whatever order reads best; that's not an answer about which taxonomy comes first.

Output changes, once — and no path moves

-# ADRs tagged `record`
+# ADRs with `tags` `record`

-What the four layers hold, and the rules between them.
+**The record** — what the four layers hold, and the rules between them.

And a declared value nobody uses now gets a row reading (0) and a page of its own, where the axis omitted it and every other field already showed it. Four such pages appear across the examples (constitution's refusal, knowledge-base's stability/systems). The row carries no colon when there's no list to introduce:

**[Refusal](tags/refusal.md)** (0) — declining, and how.

I went with "show it" because that's what the generic path already did, and (0) against a declared tag is a real signal — either aspirational or dead.

1183 passing (six new in tests/test_axis.py for the collapse itself), luria lint clean with 0 unacknowledged references. docs/project-memory.md, the vocabularies.py module docstring and the adr_index.py header are rewritten against the one renderer; generated views reverted before committing.

---

Generated by Claude Code