Silent type coercion in SHAPE normalization masks malformed-spec type errors

[amc-corey-cox]

Malformed-type values in a transformation spec are silently coerced to strings during the SHAPE phase (ReferenceValidator.normalize()) instead of surfacing as validation errors. Pydantic alone rejects all of these; ReferenceValidator coerces them before the model ever sees them:

Spec input	After SHAPE	Pydantic-alone result
sources: 5	["5"]	ValidationError
sources: {a: 1}	["1"]	ValidationError
object_derivations: "oops"	["oops"] (then flattened to [])	ValidationError

ensure_list is doing two jobs at once: the desirable scalar→list convenience ("lr" → ["lr"], which the multivalued populated_from feature relies on) and an undesirable coercion of genuinely wrong types that hides the error.

This is a SHAPE/ReferenceValidator layer-wide behavior affecting every multivalued field, not specific to sources or object_derivations. Low priority — it only bites on actively malformed YAML, the common scalar/number-string case coerces to something sensible, and it predates the current normalization work.

Possible directions:

• Add a pre-SHAPE type check that rejects values that are neither a scalar of the expected type nor a list of them, before ensure_list coerces them.
• Or a strict mode that distinguishes "scalar → one-element list" convenience from "wrong type" and fails loudly on the latter.

Surfaced during review of #250 (Copilot flagged the _flatten_ods_in_class_deriv and _migrate_pv_sources_to_populated_from MIGRATE-phase helpers, but the coercion actually happens upstream in SHAPE).

[github-actions]

Summary

Bug: The SHAPE normalization layer silently coerces wrong-typed spec values into plausible-looking strings, bypassing what would otherwise be Pydantic ValidationErrors. The result is that malformed YAML specs are silently accepted and may produce unexpected (but non-crashing) behavior at transform time.

---

Root Cause: Two jobs in one function

The coercion originates in ReferenceValidator.ensure_list() (in the upstream linkml_runtime library, at linkml_runtime/processing/referencevalidator.py:555):

def ensure_list(self, input_object, parent_slot, pk_slot_name, report):
    if not isinstance(input_object, list | dict):
        # Atom -> List   ← catches int, float, bool, str — all atoms
        return [input_object]
    if isinstance(input_object, dict):
        # Dict -> List
        return [_obj_from_item(k, v) for k, v in input_object.items()]
    return input_object

This function correctly handles the desired case ("lr" → ["lr"]), which the multivalued populated_from feature relies on. But it also silently handles undesired cases where the value is the wrong type entirely (5, {a: 1}, "oops" for object_derivations).

The coercion path for sources: 5:

1. Transformer._normalize_spec_dict() → normalize_transform_spec() → ReferenceValidator.normalize() (SHAPE phase)
2. normalize_slot_value() calls normalize_to_collection_form() with CollectionForm.List for the multivalued sources field
3. ensure_list(5, ...) → [5] (integer wrapped in a list)
4. normalize_instance(5, ...) → type-coerces 5 to "5" (string)
5. Final result: ["5"] — passes Pydantic validation since sources: list[str]

For sources: {a: 1}:

• ensure_list({a: 1}, ...) treats the dict as an expanded-dict collection form → ["1"] (value from dict entry)

For object_derivations: "oops":

• ensure_list("oops", ...) → ["oops"]
• Then _flatten_ods_in_class_deriv (MIGRATE helper) flattens it to [] — the malformed string disappears entirely

---

How it was surfaced

Discovered during review of PR #250 (multivalued populated_from on PermissibleValueDerivation — closes out the sources deprecation from PR #194 / issue #193). Copilot flagged the _flatten_ods_in_class_deriv and _migrate_pv_sources_to_populated_from MIGRATE-phase helpers as potential coercion concerns, but the actual coercion happens upstream in the SHAPE layer, not in those helpers.

---

Scope

This affects every multivalued field in the transformer spec — not just sources or object_derivations. Any field that ReferenceValidator normalizes via ensure_list with CollectionForm.List will silently accept wrong-typed values.

---

Proposed fix directions

Option A — Pre-SHAPE type guard in linkml-map (preferred for localization):
Before Transformer.normalize_transform_spec() calls the ReferenceValidator, add a type-checking pass that inspects known multivalued fields and rejects values that are neither a scalar of the expected type nor a list of them. This keeps the fix inside linkml-map without forking upstream.

Option B — Strict mode flag on ReferenceValidator (upstream change):
Add a flag like strict_collection_coercion=True to ReferenceValidator that, when set, raises a validation error when ensure_list receives a value that isn't a valid atom of the expected range type. This is cleaner architecturally but requires a change in linkml_runtime.

---

Priority

Low — as noted in the issue. This only bites on actively malformed YAML that no correct tooling would produce. The common scalar-to-string case (sources: "lr" → ["lr"]) is actually the desirable convenience behavior that populated_from depends on. The bug predates current normalization work and there are no known production incidents from it.

A reasonable sequencing would be to tackle this after PR #250 merges, as a follow-on hardening pass on the normalization layer.

---

Related issues

Issue/PR	Title	Status
#250	Refactor deprecation scanning; allow list-form populated_from on PV	Open PR
#194	Deprecate sources and derived_from, clarify populated_from	Merged
#193	Support multi-source class derivations via sources	Open