Skip to content

fix: do not inject min/max statistics for partition columns on empty files (#27456) - #27561

Open
Subodh-17 wants to merge 1 commit into
apache:mainfrom
Subodh-17:fix/parquet-empty-partition-stats-27456
Open

Subodh-17 wants to merge 1 commit into
apache:mainfrom
Subodh-17:fix/parquet-empty-partition-stats-27456

Conversation

@Subodh-17

Copy link
Copy Markdown

Which issue does this PR close?

Closes #27456

Rationale for this change

In Apache DataFusion, partitioned tables can have partitions whose data files contain zero rows (e.g. created via COPY (SELECT ... WHERE 1 = 0) or generated by upstream ETL pipelines).

Prior to this change, running aggregate queries on a partitioned table containing an empty partition produced incorrect results when statistics collection was enabled. For example, for a table with partition q=1 (1 row) and partition q=7 (0 rows):

SELECT count(*), min(q), max(q) FROM t;

returned (1, 1, 7) with statistics enabled, but (1, 1, 1) when statistics were disabled.

Root Cause

PartitionedFile::with_statistics unconditionally synthesized exact column statistics:

(
    Precision::Exact(0),
    Precision::Exact(partition_value.clone()),
    Precision::Exact(1),
)

for partition columns, regardless of whether the file contained rows. As a result, an empty file with stats.num_rows == Precision::Exact(0) declared an exact minimum and maximum equal to its partition value (e.g. min=7, max=7).

When Statistics::try_merge_iter_with_ndv_fallback merged file statistics into table-level statistics, these spurious extrema were included in the table's bounds. The AggregateStatistics physical optimizer then substituted constant literals min(q) -> 1 and max(q) -> 7 into the plan without scanning physical data, returning corrupt aggregate results.

What changes are included in this PR?

  1. datafusion/datasource/src/mod.rs:

    • In PartitionedFile::with_statistics, added an explicit check for stats.num_rows == Precision::Exact(0):
      let (null_count, min_max, distinct_count) =
          if stats.num_rows == Precision::Exact(0) {
              (Precision::Exact(0), Precision::Absent, Precision::Exact(0))
          } else if partition_value.is_null() {
              (stats.num_rows, Precision::Absent, Precision::Exact(0))
          } else {
              (
                  Precision::Exact(0),
                  Precision::Exact(partition_value.clone()),
                  Precision::Exact(1),
              )
          };
    • An empty file contains 0 rows; therefore, its partition column contains 0 distinct values, 0 nulls, and its min/max extrema are absent ($\min(\emptyset) = \text{undefined}$).
    • distinct_count is assigned Precision::Exact(0) rather than Precision::Absent. This ensures that multi-file NDV merging in Statistics::try_merge_iter_with_ndv_fallback matches (Some(&l), Some(&0)) and preserves the known table NDV via ndv_fallback.merge(l, 0) = max(l, 0) = l, preventing unnecessary degradation to Precision::Absent.
    • Updated docstrings on struct PartitionedFile and fn with_statistics documenting the empty-file statistics invariant.
    • Added unit tests:
      • test_with_statistics_empty_file_non_null_partition: Verifies an empty file with a non-null partition value (q = 7) yields null_count = Exact(0), min = Absent, max = Absent, distinct_count = Exact(0), and byte_size = Exact(0).
      • test_with_statistics_empty_file_null_partition: Verifies an empty file with a Hive null partition value (q = NULL) yields identical empty statistics.
  2. datafusion/sqllogictest/test_files/listing_table_partitions.slt:

    • Added end-to-end sqllogictest reproducing the issue scenario:
      • Partition q=1 with 1 row.
      • Partition q=7 with 0 rows.
      • Verifies SELECT count(*), min(q), max(q) returns 1 1 1.
      • Verifies GROUP BY q returns only active partitions (1 1 1 1).
      • Verifies parity when set datafusion.execution.collect_statistics = false.

Are these changes tested?

Yes:

  • Targeted unit tests:
    cargo test -p datafusion-datasource -- test_with_statistics
    Result: 4 passed, 0 failed.
  • Full crate test suite:
    cargo test -p datafusion-datasource
    Result: 205 passed, 0 failed; 8 doctests passed.
  • Sqllogictests:
    cargo test -p datafusion-sqllogictest --test sqllogictests -- listing_table_partitions
    Result: 1/1 files completed (100%), 0 failures.
  • Code formatting:
    cargo fmt --check
    Result: Clean.
  • Clippy:
    cargo clippy -p datafusion-datasource -- -D warnings
    Result: Clean (0 warnings).

Are there any user-facing changes?

Yes: Queries evaluating MIN or MAX aggregates over partitioned tables with empty partition files will no longer return incorrect partition extrema derived from the empty files.

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) datasource Changes to the datasource crate labels Nov 18, 2026
@Subodh-17
Subodh-17 force-pushed the fix/parquet-empty-partition-stats-27456 branch from f4b16d3 to 649cb86 Compare November 20, 2026 07:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

datasource Changes to the datasource crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MIN/MAX of a partition column includes partitions whose files have 0 rows

1 participant