Repository navigation
feat: prune unused columns from the body of a materialized CTE - #26274
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #26274 +/- ##
==========================================
- Coverage 82.89% 82.88% -0.02%
==========================================
Files 1154 1154
Lines 457630 458101 +471
Branches 457630 458101 +471
==========================================
+ Hits 379370 379712 +342
- Misses 55256 55310 +54
- Partials 23004 23079 +75 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
00200fa to
8acc60b
Compare
8acc60b to
108276b
Compare
108276b to
92da7a4
Compare
92da7a4 to
e2a2838
Compare
There was a problem hiding this comment.
Thanks for this change. The implementation addresses unnecessary column retention in materialized CTEs, and the approach looks appropriate. The added regression coverage exercises the main projection and remapping behavior across nested references, subqueries, and row-count-only scans.
The key architectural boundary is preserved: the shared CTE body retains the union of columns required by its references, while each scan independently selects its output. Per-reference filters remain outside the materialization fence. Optimizing the continuation first and remapping scan positions at the shared-body owner is a sound way to maintain this separation.
I found no blocking correctness issues. I've left two optional suggestions for strengthening regression coverage around schema attributes and spilled-batch projections.
| } | ||
|
|
||
| #[test] | ||
| fn scan_projection_refers_to_the_body() -> Result<()> { |
There was a problem hiding this comment.
Optional: could we extend this test with field/schema metadata, mixed nullability, and a functional dependency whose indices change under projection? Please verify these attributes survive projection composition and that body-column remapping leaves the scan schema unchanged, since the current fixture has no metadata or dependencies and all fields are nullable.
| /// A scan with a projection outputs the selected columns of the buffered | ||
| /// batches. A scan that selects no column keeps the row count. | ||
| #[tokio::test] | ||
| async fn scan_projection() -> Result<()> { |
There was a problem hiding this comment.
Optional follow-up: could we test different nonempty projections, including reordered columns, over a forced-spill, multi-column body? Please assert that spilling occurs and verify selected values, exact batch schemas, and row counts after disk replay. Existing spill tests cover zero-column projection, but not this combination of spill serialization and scan-local projection.
|
Marked as draft after 14 days without activity: merge conflicts. Please address these items and mark Ready for review when ready. |
e2a2838 to
908c137
Compare
kosiew
left a comment
There was a problem hiding this comment.
@adriangb, thanks for the update. Pruning at the CTE owner correctly narrows the shared buffer while preserving each reference's projection.
I'm requesting changes for two issues: projected scans overestimate byte sizes, affecting join selection, and schema refresh rejects valid type changes.
| statistics: Some(statistics), | ||
| // The body statistics have a column for each column of the body. | ||
| statistics: Some(Arc::new( | ||
| Arc::unwrap_or_clone(statistics).project(self.projection.as_ref()), |
There was a problem hiding this comment.
Projected scans retain the body's total byte size, which can change join selection. Please call calculate_total_byte_size(&self.schema) after project and assert the expected bytes directly in the test.
| pub fn with_body_schema(&self, body: &DFSchema) -> Result<Self> { | ||
| let body = match &self.projection { | ||
| Some(projection) => { | ||
| self.check_body_schema(body)?; |
There was a problem hiding this comment.
with_body_schema rejects valid projected-field type changes because check_body_schema compares old types. Please validate indices and names without comparing types, and add tests for type changes and name mismatches.
| .all(|(index, field)| { | ||
| body.fields() | ||
| .get(*index) | ||
| .is_some_and(|b| b.data_type() == field.data_type()) |
There was a problem hiding this comment.
Could we also validate field names to prevent manually constructed scans from mislabeling columns? First confirm that valid physical and logical field names always match.
| } | ||
| Ok(()) | ||
| } | ||
| DisplayFormatType::TreeRender => write!(f, "name={}", self.name), |
There was a problem hiding this comment.
Could we show the projection in physical TreeRender? Currently, projected and unprojected scans appear identical, making plans harder to diagnose.
|
Thank you for your help getting this into a usable state! |
…Exec A `MaterializedCteScan` was a leaf with the full schema of the CTE body, so there was no way to express that a reference to a materialized CTE reads only some of its columns. Add `projection: Option<Vec<usize>>` to `MaterializedCteScan`: the indexes of the columns of the body that the scan outputs (`None` means all of them). `with_projection` narrows a scan, `with_body_columns` remaps a scan onto a body that kept a subset of its columns, and `check_body_schema` checks a scan against the schema of its body. `MaterializedCte::try_new` uses it to validate the scans of the continuation. `MaterializedCteScanExec` gets the same projection: it projects the shared buffered batches, its statistics, and shows the projection in EXPLAIN. The physical planner passes the projection through and checks each scan against the body schema when it binds the scan. Nothing sets a projection yet, so plans do not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add cases to cte_materialized.slt that reference a wide materialized CTE and read only some of its columns: through a union, an aggregate, a join, subquery expressions, nested CTEs and a recursive CTE. The expected output is the current behavior: the body keeps all its columns and each scan outputs the full schema of the body, with a projection above it. The next commit prunes the unused columns, and its diff of this file shows how the plans change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The body of a `MaterializedCte` kept all its columns, because each `MaterializedCteScan` was a leaf with the full schema of the body. A `MaterializedCteScan` now has a `projection`: the indexes of the columns of the body that it outputs. `OptimizeProjections` narrows each scan to the columns that its parent needs, as it does for a `TableScan`. At the `MaterializedCte` node it optimizes the continuation first, computes the union of the columns of all the scans of the CTE (also in subquery expressions and in the bodies of other CTEs), projects the body to this union, and gives each scan the new positions of its columns. `MaterializedCteScanExec` applies the projection to the shared buffered batches. The physical planner checks that each scan agrees with the schema of its body when it binds the scan. Closes apache#26240. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… scan projections
… runs again CREATE TABLE AS and execute_logical_plan run the analyzer on an optimized plan. TypeCoercion gave each scan the full body schema, so a pruned scan got more fields than its projection and planning panicked or failed.
… and the scan projection getter Mutation testing showed that no test failed when the remapping of scan positions also entered a nested occurrence of the same CTE.
…pgrade guide The projection ships in 57, not 56.
…57.0.0 upgrade guide
…n step A scan whose schema does not match the body columns it reads gives a panic or a wrong result at execution. The TypeCoercion bug fixed earlier in this PR was this kind of mismatch. This commit makes the mismatch harder to build and catches it during planning. - Make MaterializedCteScan::schema and ::projection private. The accessor methods and the with_* builders keep them in step. - Replace MaterializedCteScanExec::with_projection with the fallible with_body_projection, which rejects a projection whose length does not match the scan schema. - Reject a body of another width in with_body_schema for a scan with no projection. - Check every scan of each MaterializedCte against the body in LogicalPlan::check_invariants. The optimizer runs this after each rule in debug builds.
…reshes its schema with_body_schema called check_body_schema, which compares the types of the body against the old types of the scan. So TypeCoercion failed when it changed the type of a column that a projected scan reads. with_body_schema now checks only the indexes and the names. check_body_schema, try_new and the plan invariant keep the strict check.
- Statistics::project keeps the byte size of the body, so a narrow scan looked as large as the body to join selection. The scan now computes the byte size again for the columns that it reads. - check_body_schema also compares the names of the columns, so a scan built by hand cannot label a body column with another name. - TreeRender shows the projection, as the other display formats do.
e19bb1f to
2808c23
Compare
Which issue does this PR close?
Stacked on #26269. Only the last commit is new.
Rationale for this change
The body of a materialized CTE kept all its columns, also when no reference read them.
Before, all five columns were read and buffered:
After, only
aandbare read and buffered:What changes are included in this PR?
MaterializedCteScanhas a new fieldprojection: Option<Vec<usize>>: the indexes of the columns of the CTE body that the scan outputs.OptimizeProjectionsnarrows eachMaterializedCteScanto the columns that its parent needs, as it does for aTableScan.MaterializedCte,OptimizeProjectionsoptimizes the continuation first. It then computes the union of the columns of all the scans of the CTE, projects the body to this union, and gives each scan the new positions of its columns. Scans in subquery expressions and in the bodies of other CTEs are counted.MaterializedCteScanExecapplies the projection to the shared buffered batches.The projection in the scan makes the intermediate state (a narrow scan and a wide body) a valid plan, so no separate rule is necessary.
Notes:
SELECT count(*) FROM c) getsprojection=[]. If no reference reads a column, the body keeps its first column.DISTINCTor a recursive query are not removed.Comparison with PostgreSQL and DuckDB. This is what DuckDB does:
RemoveUnusedColumnsprunes the body to the columns that its references read (remove_unused_columns.cpp). PostgreSQL plans a CTE for full retrieval and does not prune (subselect.c).Are these changes tested?
Yes.
cte_materialized.slt:EXPLAINand result checks for the example above, a column that only a filter reads,count(*), a CTE that another CTE reads, references in subqueries, a volatile body, aDISTINCTbody, aliased columns, a self join, a recursive body and a CTE that nothing reads. Unit tests for the scan projection, nested CTEs, idempotence and the zero-column case.Are there any user-facing changes?
EXPLAINshowsprojection=[...]onMaterializedCteScanandMaterializedCteScanExecwhen the scan does not read all the columns.MaterializedCteScanhas the new public fieldprojection. The upgrade guide for 56.0.0 says this.