Repository navigation
Conversation
Subodh-17
force-pushed
the
fix/parquet-empty-partition-stats-27456
branch
from
November 20, 2026 07:35
f4b16d3 to
649cb86
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #27456
Rationale for this change
In Apache DataFusion, partitioned tables can have partitions whose data files contain zero rows (e.g. created via
COPY (SELECT ... WHERE 1 = 0)or generated by upstream ETL pipelines).Prior to this change, running aggregate queries on a partitioned table containing an empty partition produced incorrect results when statistics collection was enabled. For example, for a table with partition
q=1(1 row) and partitionq=7(0 rows):returned
(1, 1, 7)with statistics enabled, but(1, 1, 1)when statistics were disabled.Root Cause
PartitionedFile::with_statisticsunconditionally synthesized exact column statistics:for partition columns, regardless of whether the file contained rows. As a result, an empty file with
stats.num_rows == Precision::Exact(0)declared an exact minimum and maximum equal to its partition value (e.g.min=7, max=7).When
Statistics::try_merge_iter_with_ndv_fallbackmerged file statistics into table-level statistics, these spurious extrema were included in the table's bounds. TheAggregateStatisticsphysical optimizer then substituted constant literalsmin(q) -> 1andmax(q) -> 7into the plan without scanning physical data, returning corrupt aggregate results.What changes are included in this PR?
datafusion/datasource/src/mod.rs:PartitionedFile::with_statistics, added an explicit check forstats.num_rows == Precision::Exact(0):distinct_countis assignedPrecision::Exact(0)rather thanPrecision::Absent. This ensures that multi-file NDV merging inStatistics::try_merge_iter_with_ndv_fallbackmatches(Some(&l), Some(&0))and preserves the known table NDV viandv_fallback.merge(l, 0) = max(l, 0) = l, preventing unnecessary degradation toPrecision::Absent.struct PartitionedFileandfn with_statisticsdocumenting the empty-file statistics invariant.test_with_statistics_empty_file_non_null_partition: Verifies an empty file with a non-null partition value (q = 7) yieldsnull_count = Exact(0),min = Absent,max = Absent,distinct_count = Exact(0), andbyte_size = Exact(0).test_with_statistics_empty_file_null_partition: Verifies an empty file with a Hive null partition value (q = NULL) yields identical empty statistics.datafusion/sqllogictest/test_files/listing_table_partitions.slt:q=1with 1 row.q=7with 0 rows.SELECT count(*), min(q), max(q)returns1 1 1.GROUP BY qreturns only active partitions (1 1 1 1).set datafusion.execution.collect_statistics = false.Are these changes tested?
Yes:
cargo test -p datafusion-datasource -- test_with_statisticscargo test -p datafusion-datasourcecargo test -p datafusion-sqllogictest --test sqllogictests -- listing_table_partitionsAre there any user-facing changes?
Yes: Queries evaluating
MINorMAXaggregates over partitioned tables with empty partition files will no longer return incorrect partition extrema derived from the empty files.