Skip to content

Fix/reference list round trip - #72

Merged
dschwartznyc merged 6 commits into
mainfrom
fix/reference-list-round-trip
Nov 11, 2026
Merged

dschwartznyc merged 6 commits into
mainfrom
fix/reference-list-round-trip

Conversation

@plamen-neykov

@plamen-neykov plamen-neykov commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Resolve references stored directly in list fields and preserve their per-occurrence metadata through validation, serialization, mutation, copying, sorting, and model construction.

Callers receive actual target objects while the runtime retains whether each occurrence was supplied inline or by reference. The same target can therefore appear both ways in one list and serialize correctly.

Fixes #43.

Builds on #71.

Compatibility and release

This change is intended for runtime 3.0.0 because the runtime and generated-code contracts require a coordinated update.

  • The package version will be generated from the 3.0.0 Git tag.
  • The corresponding generator update should require rune.runtime>=3.0.0,<4.0.0.
  • Older generated packages that require <3.0.0 remain on runtime 2.x.
  • Generator support for emitting rune_sorted is a separate follow-up. Adding the runtime helper does not change existing generated code.

Reference-list behavior

  • Resolve list-element references, including forward and nested references, while checking target types and reference permissions.
  • Retain reference keys and types separately for each occurrence.
  • Preserve metadata through supported list mutations, slices, copies, concatenation, repetition, reversal, and in-place sorting.
  • Keep the mutable-list API. Whole-field assignment owns its list storage while sharing target objects.
  • Serialize reference occurrences as @ref entries without traversing their targets as containment children.
  • Reject unresolved references that would be serialized.
  • Respect serialization filtering, aliases, and subtype information.

Equality follows ordinary list value equality. A reference occurrence and an equivalent inline value compare equal. remove(value) removes the first equal occurrence; callers should use an index when selecting a particular occurrence.

Serialization

Use PlainSerializer(..., when_used='unless-none') to handle whole-value None without explicit guards in the metadata serializers.

Delegate resolved scalar-reference metadata serialization to Reference.serialise_meta. Preserve null list entries where supported by the serializer.

Sorting and proxy handling

Add rune_sorted for stable, shallow sorted copies that preserve reference-list metadata:

ordered = rune_sorted(
    model.refs,
    key=lambda party: party.name,
    skip_none=True,
)
  • skip_none defaults to False.
  • With skip_none=True, remove only None entries before evaluating the key.
  • Preserve COW item proxies and leave the source collection's order unchanged.
  • Rebuild retained values and their reference metadata together when filtering null entries.
  • Ordinary iterables produce plain lists.

The built-in sorted() and explicit conversion to a plain list discard occurrence metadata.

Add rune_unwrap_proxy in utils.py and reuse it where a single callable _unwrap hook should be followed. This helper does not depend on cow.py; recursive COW unwrapping remains separate.

ObjectBuilder integration

Preserve reference-list metadata during ObjectBuilder._normalize while recursively normalizing its elements.

Previously, the generic list branch converted copied or sorted reference lists into plain lists. Building a model from those values then serialized references as full objects.

The reference-list branch now retains occurrence metadata, including when inline elements contain nested builders.

Performance

  • Cache reference fields, containment fields, and reference constraints by model class.
  • Invalidate cached results when models are rebuilt.
  • Use the plain serialization path for models without reference-capable fields.
  • Skip primitive-only fields during containment traversal.
  • Refresh parent links at the public reference-resolution entry point; validation-time resolution reuses existing parent wiring.

Add benchmarks/benchmark_list_references.py to measure reading, writing, and validation using the same models and dependencies across revisions. PR #71 provides the comparison baseline because main cannot serialize these lists.

Ownership and related work

When revalidation creates a new model that shares existing children, those children retain their original parent. The new model receives separate key-registry dictionaries whose entries still point to the shared targets.

These semantics are relevant to #49 and #66. This PR does not implement detached subtree copying or a broader ownership redesign.

Utility cleanup

Retain the accompanying utils.py cleanup:

  • Organize imports and __all__.
  • Use a module logger.
  • Raise TypeError rather than ValueError for an invalid rune_add_to_list destination.

Also ignore local .tmp/ scratch files.

Validation

The latest targeted run passed 132 tests, covering ObjectBuilder, reference-list serialization, sorting, COW behavior, utility helpers, and operations.

New builder regressions cover copied, sliced, sorted, and COW-backed reference lists, plus recursive normalization of nested builders. These cases failed before the normalization fix.

git diff --check passed.

Earlier broader runs identified an existing Windows path/module-resolution failure in test_init_invalid_directory_or_module; that issue is outside this change.

@plamen-neykov plamen-neykov self-assigned this Nov 11, 2026
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Test Results

563 tests  +112   543 ✅ +112   2s ⏱️ -1s
  1 suites ±  0    20 💤 ±  0 
  1 files   ±  0     0 ❌ ±  0 

Results for commit 5ba613b. ± Comparison against base commit 6cd39ce.

♻️ This comment has been updated with latest results.

Cache reference fields, containment fields and reference constraints,
invalidating the caches when models are rebuilt. Skip unnecessary
serialization work and separate parent refresh from reference resolution.

Add reproducible benchmarks and regression coverage. Document ownership,
list semantics and utility compatibility changes, and ignore local scratch files.
Preserve reference metadata and COW item proxies in stable sorted copies.
Centralize one-level _unwrap hook handling in utils.py and reuse it in
metadata, validation, comparisons, condition checks and sorting.

Add regression coverage and document reference-list semantics.
@plamen-neykov
plamen-neykov marked this pull request as ready for review November 11, 2026 19:49
@plamen-neykov
plamen-neykov marked this pull request as draft November 11, 2026 20:08
Add skip_none=False to optionally filter None entries before sorting
and key evaluation.

Rebuild values and reference metadata together in remove_none,
preserving occurrence order, object identity, and COW behavior.

Validation: 124 targeted tests passed.
Normalize reference-list values without converting them to plain lists,
retaining per-occurrence metadata through model construction.

Add regression coverage for copies, slices, sorted and COW inputs,
and nested builders.

Validation: 132 targeted tests passed.
@plamen-neykov
plamen-neykov marked this pull request as ready for review November 11, 2026 21:18
@dschwartznyc
dschwartznyc merged commit ba623bb into main Nov 11, 2026
6 checks passed
@plamen-neykov
plamen-neykov deleted the fix/reference-list-round-trip branch November 11, 2026 21:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: resolve_references does not resolve references held in list fields

3 participants