Skip to content

feat: prune unused columns from the body of a materialized CTE - #26274

Merged
adriangb merged 11 commits into
apache:mainfrom
adriangb:mcte-column-pruning
Nov 20, 2026
Merged

adriangb merged 11 commits into
apache:mainfrom
adriangb:mcte-column-pruning

Conversation

@adriangb

@adriangb adriangb commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Stacked on #26269. Only the last commit is new.

Rationale for this change

The body of a materialized CTE kept all its columns, also when no reference read them.

CREATE TABLE wide(a INT, b INT, c INT, d INT, e INT) AS VALUES (1, 10, 100, 1000, 10000);

EXPLAIN WITH w AS MATERIALIZED (SELECT * FROM wide)
SELECT a FROM w
UNION ALL
SELECT b FROM w;

Before, all five columns were read and buffered:

MaterializedCte: name=w
  SubqueryAlias: w
    TableScan: wide projection=[a, b, c, d, e]
  Union
    Projection: w.a
      MaterializedCteScan: name=w
    Projection: w.b AS a
      MaterializedCteScan: name=w

After, only a and b are read and buffered:

MaterializedCte: name=w
  SubqueryAlias: w
    TableScan: wide projection=[a, b]
  Union
    MaterializedCteScan: name=w projection=[a]
    Projection: w.b AS a
      MaterializedCteScan: name=w projection=[b]

What changes are included in this PR?

  • MaterializedCteScan has a new field projection: Option<Vec<usize>>: the indexes of the columns of the CTE body that the scan outputs.
  • OptimizeProjections narrows each MaterializedCteScan to the columns that its parent needs, as it does for a TableScan.
  • At a MaterializedCte, OptimizeProjections optimizes the continuation first. It then computes the union of the columns of all the scans of the CTE, projects the body to this union, and gives each scan the new positions of its columns. Scans in subquery expressions and in the bodies of other CTEs are counted.
  • MaterializedCteScanExec applies the projection to the shared buffered batches.
  • The physical planner checks that each scan agrees with the physical schema of its body when it binds the scan, and returns an internal error if it does not.

The projection in the scan makes the intermediate state (a narrow scan and a wide body) a valid plan, so no separate rule is necessary.

Notes:

  • A reference that reads no column (SELECT count(*) FROM c) gets projection=[]. If no reference reads a column, the body keeps its first column.
  • The projection is put above the body. A volatile column can be removed, but the number of rows does not change. Columns below a DISTINCT or a recursive query are not removed.
  • A scan in a correlated subquery that is not decorrelated is not narrowed. The CTE then keeps all its columns.
  • The pgjson display does not show the projection.

Comparison with PostgreSQL and DuckDB. This is what DuckDB does: RemoveUnusedColumns prunes the body to the columns that its references read (remove_unused_columns.cpp). PostgreSQL plans a CTE for full retrieval and does not prune (subselect.c).

Are these changes tested?

Yes. cte_materialized.slt: EXPLAIN and result checks for the example above, a column that only a filter reads, count(*), a CTE that another CTE reads, references in subqueries, a volatile body, a DISTINCT body, aliased columns, a self join, a recursive body and a CTE that nothing reads. Unit tests for the scan projection, nested CTEs, idempotence and the zero-column case.

Are there any user-facing changes?

EXPLAIN shows projection=[...] on MaterializedCteScan and MaterializedCteScanExec when the scan does not read all the columns. MaterializedCteScan has the new public field projection. The upgrade guide for 56.0.0 says this.

@github-actions github-actions Bot added documentation Improvements or additions to documentation sql SQL Planner logical-expr Logical plan and expressions optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) substrait Changes to the substrait crate proto Related to proto crate physical-plan Changes to the physical-plan crate auto detected api change Auto detected API change labels Oct 13, 2026
@codecov-commenter

codecov-commenter commented Oct 13, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.05850% with 136 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.88%. Comparing base (484183d) to head (108276b).
⚠️ Report is 161 commits behind head on main.

Files with missing lines Patch % Lines
...tafusion/expr/src/logical_plan/materialized_cte.rs 83.50% 20 Missing and 28 partials ⚠️
...tafusion/optimizer/src/optimize_projections/mod.rs 78.35% 6 Missing and 23 partials ⚠️
datafusion/physical-plan/src/materialized_cte.rs 85.41% 4 Missing and 10 partials ⚠️
datafusion/expr/src/logical_plan/plan.rs 65.78% 11 Missing and 2 partials ⚠️
datafusion/expr/src/logical_plan/display.rs 0.00% 8 Missing ⚠️
datafusion/optimizer/src/analyzer/type_coercion.rs 83.72% 4 Missing and 3 partials ⚠️
datafusion/core/src/physical_planner.rs 86.36% 2 Missing and 4 partials ⚠️
...ion/substrait/src/logical_plan/producer/rel/mod.rs 0.00% 4 Missing ⚠️
datafusion/optimizer/src/decorrelate.rs 76.92% 1 Missing and 2 partials ⚠️
datafusion/optimizer/src/optimizer.rs 60.00% 0 Missing and 2 partials ⚠️
... and 2 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #26274      +/-   ##
==========================================
- Coverage   82.89%   82.88%   -0.02%     
==========================================
  Files        1154     1154              
  Lines      457630   458101     +471     
  Branches   457630   458101     +471     
==========================================
+ Hits       379370   379712     +342     
- Misses      55256    55310      +54     
- Partials    23004    23079      +75     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@adriangb
adriangb force-pushed the mcte-column-pruning branch 4 times, most recently from 00200fa to 8acc60b Compare October 16, 2026 16:57
@adriangb
adriangb force-pushed the mcte-column-pruning branch from 8acc60b to 108276b Compare October 17, 2026 21:59
@adriangb
adriangb force-pushed the mcte-column-pruning branch from 108276b to 92da7a4 Compare October 28, 2026 00:55
@github-actions github-actions Bot removed sql SQL Planner substrait Changes to the substrait crate proto Related to proto crate labels Oct 28, 2026
@adriangb
adriangb marked this pull request as ready for review October 28, 2026 01:03
@adriangb
adriangb force-pushed the mcte-column-pruning branch from 92da7a4 to e2a2838 Compare October 28, 2026 05:11

@kosiew kosiew left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adriangb,

Thanks for this change. The implementation addresses unnecessary column retention in materialized CTEs, and the approach looks appropriate. The added regression coverage exercises the main projection and remapping behavior across nested references, subqueries, and row-count-only scans.

The key architectural boundary is preserved: the shared CTE body retains the union of columns required by its references, while each scan independently selects its output. Per-reference filters remain outside the materialization fence. Optimizing the continuation first and remapping scan positions at the shared-body owner is a sound way to maintain this separation.

I found no blocking correctness issues. I've left two optional suggestions for strengthening regression coverage around schema attributes and spilled-batch projections.

}

#[test]
fn scan_projection_refers_to_the_body() -> Result<()> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: could we extend this test with field/schema metadata, mixed nullability, and a functional dependency whose indices change under projection? Please verify these attributes survive projection composition and that body-column remapping leaves the scan schema unchanged, since the current fixture has no metadata or dependencies and all fields are nullable.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, done in 146a575

/// A scan with a projection outputs the selected columns of the buffered
/// batches. A scan that selects no column keeps the row count.
#[tokio::test]
async fn scan_projection() -> Result<()> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional follow-up: could we test different nonempty projections, including reordered columns, over a forced-spill, multi-column body? Please assert that spilling occurs and verify selected values, exact batch schemas, and row counts after disk replay. Existing spill tests cover zero-column projection, but not this combination of spill serialization and scan-local projection.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 146a575

@github-actions
github-actions Bot marked this pull request as draft November 18, 2026 01:09
@github-actions

Copy link
Copy Markdown

Marked as draft after 14 days without activity: merge conflicts.

Please address these items and mark Ready for review when ready.

@adriangb
adriangb force-pushed the mcte-column-pruning branch from e2a2838 to 908c137 Compare November 18, 2026 08:29
@github-actions github-actions Bot removed the auto detected api change Auto detected API change label Nov 18, 2026
@adriangb
adriangb marked this pull request as ready for review November 18, 2026 19:28

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adriangb, thanks for the update. Pruning at the CTE owner correctly narrows the shared buffer while preserving each reference's projection.

I'm requesting changes for two issues: projected scans overestimate byte sizes, affecting join selection, and schema refresh rejects valid type changes.

statistics: Some(statistics),
// The body statistics have a column for each column of the body.
statistics: Some(Arc::new(
Arc::unwrap_or_clone(statistics).project(self.projection.as_ref()),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Projected scans retain the body's total byte size, which can change join selection. Please call calculate_total_byte_size(&self.schema) after project and assert the expected bytes directly in the test.

pub fn with_body_schema(&self, body: &DFSchema) -> Result<Self> {
let body = match &self.projection {
Some(projection) => {
self.check_body_schema(body)?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with_body_schema rejects valid projected-field type changes because check_body_schema compares old types. Please validate indices and names without comparing types, and add tests for type changes and name mismatches.

.all(|(index, field)| {
body.fields()
.get(*index)
.is_some_and(|b| b.data_type() == field.data_type())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we also validate field names to prevent manually constructed scans from mislabeling columns? First confirm that valid physical and logical field names always match.

}
Ok(())
}
DisplayFormatType::TreeRender => write!(f, "name={}", self.name),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we show the projection in physical TreeRender? Currently, projected and unprojected scans appear identical, making plans harder to diagnose.

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adriangb, thanks for addressing the previous findings. The follow-up fixes resolve both blockers, and the logical and physical projection handling remains consistent.

I found no remaining correctness issues.

@adriangb

Copy link
Copy Markdown
Contributor Author

Thank you for your help getting this into a usable state!

adriangb and others added 11 commits November 19, 2026 23:04
…Exec

A `MaterializedCteScan` was a leaf with the full schema of the CTE body,
so there was no way to express that a reference to a materialized CTE
reads only some of its columns.

Add `projection: Option<Vec<usize>>` to `MaterializedCteScan`: the indexes
of the columns of the body that the scan outputs (`None` means all of
them). `with_projection` narrows a scan, `with_body_columns` remaps a scan
onto a body that kept a subset of its columns, and `check_body_schema`
checks a scan against the schema of its body. `MaterializedCte::try_new`
uses it to validate the scans of the continuation.

`MaterializedCteScanExec` gets the same projection: it projects the
shared buffered batches, its statistics, and shows the projection in
EXPLAIN. The physical planner passes the projection through and checks
each scan against the body schema when it binds the scan.

Nothing sets a projection yet, so plans do not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add cases to cte_materialized.slt that reference a wide materialized CTE
and read only some of its columns: through a union, an aggregate, a join,
subquery expressions, nested CTEs and a recursive CTE.

The expected output is the current behavior: the body keeps all its
columns and each scan outputs the full schema of the body, with a
projection above it. The next commit prunes the unused columns, and its
diff of this file shows how the plans change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The body of a `MaterializedCte` kept all its columns, because each `MaterializedCteScan` was a leaf with the full schema of the body.

A `MaterializedCteScan` now has a `projection`: the indexes of the columns of the body that it outputs.
`OptimizeProjections` narrows each scan to the columns that its parent needs, as it does for a `TableScan`.
At the `MaterializedCte` node it optimizes the continuation first, computes the union of the columns of all the scans of the CTE (also in subquery expressions and in the bodies of other CTEs), projects the body to this union, and gives each scan the new positions of its columns.
`MaterializedCteScanExec` applies the projection to the shared buffered batches.
The physical planner checks that each scan agrees with the schema of its body when it binds the scan.

Closes apache#26240.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… runs again

CREATE TABLE AS and execute_logical_plan run the analyzer on an optimized
plan. TypeCoercion gave each scan the full body schema, so a pruned scan got
more fields than its projection and planning panicked or failed.
… and the scan projection getter

Mutation testing showed that no test failed when the remapping of scan
positions also entered a nested occurrence of the same CTE.
…pgrade guide

The projection ships in 57, not 56.
…n step

A scan whose schema does not match the body columns it reads gives a panic
or a wrong result at execution. The TypeCoercion bug fixed earlier in this
PR was this kind of mismatch. This commit makes the mismatch harder to
build and catches it during planning.

- Make MaterializedCteScan::schema and ::projection private. The accessor
  methods and the with_* builders keep them in step.
- Replace MaterializedCteScanExec::with_projection with the fallible
  with_body_projection, which rejects a projection whose length does not
  match the scan schema.
- Reject a body of another width in with_body_schema for a scan with no
  projection.
- Check every scan of each MaterializedCte against the body in
  LogicalPlan::check_invariants. The optimizer runs this after each rule in
  debug builds.
…reshes its schema

with_body_schema called check_body_schema, which compares the types of the
body against the old types of the scan. So TypeCoercion failed when it
changed the type of a column that a projected scan reads. with_body_schema
now checks only the indexes and the names. check_body_schema, try_new and
the plan invariant keep the strict check.
- Statistics::project keeps the byte size of the body, so a narrow scan
  looked as large as the body to join selection. The scan now computes the
  byte size again for the columns that it reads.
- check_body_schema also compares the names of the columns, so a scan built
  by hand cannot label a body column with another name.
- TreeRender shows the projection, as the other display formats do.
@adriangb
adriangb force-pushed the mcte-column-pruning branch from e19bb1f to 2808c23 Compare November 20, 2026 05:04
@adriangb
adriangb enabled auto-merge November 20, 2026 05:06
@adriangb
adriangb added this pull request to the merge queue Nov 20, 2026
Merged via the queue into apache:main with commit a72c60b Nov 20, 2026
42 of 43 checks passed
@adriangb
adriangb deleted the mcte-column-pruning branch November 20, 2026 05:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate documentation Improvements or additions to documentation logical-expr Logical plan and expressions optimizer Optimizer rules physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt) v57.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Prune unused columns from the body of a materialized CTE

3 participants