Skip to content

Avoid casting unreachable child values in nested List casts #24506

Description

@kosiew

Related PR

#23914

Problem

cast_list_column and cast_list_view_column preserve parent offsets/sizes and recursively cast the full backing child array. A sliced or limited List can therefore fail because an unreachable child value cannot be cast.

Example:

CREATE TABLE lt AS SELECT * FROM (VALUES
  (1, [struct('1')]), (2, [struct('2')]), (3, [struct('bad')])
) AS t(i, l);
SELECT arrow_cast(l, 'List(Struct("c0": Int32))') FROM lt LIMIT 2;
-- currently fails while casting the unreachable `bad` value

Invariant / desired behavior

Recursive nested casts must process only child values reachable from visible, non-null parent rows. Hidden backing children must not produce cast errors.

Proposed direction

Establish one consistent child-selection strategy for List, LargeList, and ListView casts, analogous to Map entry compaction. Preserve visible offsets, nulls, and schema while avoiding recursive casts of unreachable children.

Scope

In

  • List, LargeList, and ListView recursive cast paths in datafusion/common/src/nested_struct.rs
  • Slice and null-parent regression coverage

Out

  • Map behavior (already compacts entries)
  • FixedSizeList behavior, unless the shared solution naturally applies

Acceptance criteria

  • A cast succeeds when invalid nested child data is reachable only through sliced-out or null parent rows.
  • Visible invalid child data still returns a cast error.
  • Output offsets, nulls, and data type are preserved.

Tests / verification

  • Unit regressions for List, LargeList, and ListView sliced/null-parent inputs.
  • Regression based on the SQL example above.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions