Related PR
#23914
Problem
cast_list_column and cast_list_view_column preserve parent offsets/sizes and recursively cast the full backing child array. A sliced or limited List can therefore fail because an unreachable child value cannot be cast.
Example:
CREATE TABLE lt AS SELECT * FROM (VALUES
(1, [struct('1')]), (2, [struct('2')]), (3, [struct('bad')])
) AS t(i, l);
SELECT arrow_cast(l, 'List(Struct("c0": Int32))') FROM lt LIMIT 2;
-- currently fails while casting the unreachable `bad` value
Invariant / desired behavior
Recursive nested casts must process only child values reachable from visible, non-null parent rows. Hidden backing children must not produce cast errors.
Proposed direction
Establish one consistent child-selection strategy for List, LargeList, and ListView casts, analogous to Map entry compaction. Preserve visible offsets, nulls, and schema while avoiding recursive casts of unreachable children.
Scope
In
- List, LargeList, and ListView recursive cast paths in
datafusion/common/src/nested_struct.rs
- Slice and null-parent regression coverage
Out
- Map behavior (already compacts entries)
- FixedSizeList behavior, unless the shared solution naturally applies
Acceptance criteria
Tests / verification
- Unit regressions for List, LargeList, and ListView sliced/null-parent inputs.
- Regression based on the SQL example above.
Related PR
#23914
Problem
cast_list_columnandcast_list_view_columnpreserve parent offsets/sizes and recursively cast the full backing child array. A sliced or limited List can therefore fail because an unreachable child value cannot be cast.Example:
Invariant / desired behavior
Recursive nested casts must process only child values reachable from visible, non-null parent rows. Hidden backing children must not produce cast errors.
Proposed direction
Establish one consistent child-selection strategy for List, LargeList, and ListView casts, analogous to Map entry compaction. Preserve visible offsets, nulls, and schema while avoiding recursive casts of unreachable children.
Scope
In
datafusion/common/src/nested_struct.rsOut
Acceptance criteria
Tests / verification