Repository navigation
fix: estimate GROUP BY rows from integer bounds - #27574
kumarUjjawal wants to merge 3 commits into
Conversation
Generated-by: OpenAI Codex
|
run benchmark sql_planner |
|
run benchmark tpch |
|
run benchmark tpcds |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing kumarUjjawal/issue-25611-review (c5857fb) to 5ea948b (merge-base) diff Run configurationrun benchmark sql_plannerResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing kumarUjjawal/issue-25611-review (c5857fb) to 5ea948b (merge-base) diff Run configurationrun benchmark tpchResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing kumarUjjawal/issue-25611-review (c5857fb) to 5ea948b (merge-base) diff Run configurationrun benchmark tpcdsResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing kumarUjjawal/issue-25611-review (c5857fb) to 5ea948b (merge-base) diff Run configurationrun benchmark tpchCPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing kumarUjjawal/issue-25611-review (c5857fb) to 5ea948b (merge-base) diff Run configurationrun benchmark tpcdsCPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing kumarUjjawal/issue-25611-review (c5857fb) to 5ea948b (merge-base) diff Run configurationrun benchmark sql_plannerCPU Details (lscpu)Details
Resource Usagesql_planner — base (merge-base)
sql_planner — branch
File an issue against this benchmark runner |
asolimando
left a comment
There was a problem hiding this comment.
Thanks for the PR @kumarUjjawal! The radians comment can change plans so I consider it "blocking". The others are minor estimate issues, but they are easy to fix.
| } else if expr.is::<NegativeExpr>() { | ||
| supported_inputs() | ||
| } else { | ||
| expr.is::<ScalarFunctionExpr>() |
There was a problem hiding this comment.
radians_bounds returns a fixed (-pi, pi), which is wrong (radians(720) is about 12.57). Since this PR uses these bounds for statistics, the join below is estimated at 0 rows but returns 301. The merge base estimates 301.
statement ok
COPY (SELECT CAST(value AS DOUBLE) AS a FROM range(0, 721))
TO 'test_files/scratch/parquet_statistics/deg.parquet' STORED AS PARQUET;
statement ok
CREATE EXTERNAL TABLE deg STORED AS PARQUET
LOCATION 'test_files/scratch/parquet_statistics/deg.parquet';
statement ok
COPY (SELECT radians(CAST(value AS DOUBLE)) AS x FROM range(300, 601))
TO 'test_files/scratch/parquet_statistics/rad.parquet' STORED AS PARQUET;
statement ok
CREATE EXTERNAL TABLE rad STORED AS PARQUET
LOCATION 'test_files/scratch/parquet_statistics/rad.parquet';
statement ok
SET datafusion.explain.show_statistics = true;
query TT
EXPLAIN SELECT * FROM (SELECT radians(a) AS r FROM deg) s JOIN rad ON s.r = rad.x;
HashJoinExec: ..., statistics=[Rows=Inexact(0), ...
DataSourceExec: ..., projection=[radians(a@0) as r], ... Min=Inexact(Float64(-3.1415926535897936)) Max=Inexact(Float64(3.1415926535897936))
Computing x * pi / 180 from the input interval in radians_bounds should fix it. The other functions in bounds.rs look fine.
| }); | ||
| column_statistics[idx].max_value = key_stats.max_value; | ||
| column_statistics[idx].min_value = key_stats.min_value; | ||
| column_statistics[idx].distinct_count = key_stats.distinct_count; |
There was a problem hiding this comment.
The Partial output does not keep null_count, so the Final aggregate adds a NULL group that Single mode does not add. A nullable key with no NULLs in [1, 7] gets 8 rows with 2 partitions and 7 rows with 1 partition. The query returns 7.
statement ok
COPY (SELECT CASE WHEN value < 0 THEN NULL ELSE CAST(value % 7 + 1 AS INT) END AS k FROM range(0, 10000))
TO 'test_files/scratch/parquet_statistics/k.parquet' STORED AS PARQUET;
statement ok
CREATE EXTERNAL TABLE kt STORED AS PARQUET
LOCATION 'test_files/scratch/parquet_statistics/k.parquet';
statement ok
SET datafusion.explain.show_statistics = true;
statement ok
SET datafusion.execution.target_partitions = 2;
# FinalPartitioned: Rows=Inexact(8)
query TT
EXPLAIN SELECT k, count(*) FROM kt GROUP BY k;
statement ok
SET datafusion.execution.target_partitions = 1;
# Single: Rows=Inexact(7)
query TT
EXPLAIN SELECT k, count(*) FROM kt GROUP BY k;
Copying null_count for a plain GROUP BY (not for grouping sets) should fix it.
| DROP TABLE literal_ndv_empty; | ||
|
|
||
| # The years of the date bounds limit the groups of the final aggregate. The | ||
| # key is nullable and its NULL count is unknown, so one NULL group is added to |
There was a problem hiding this comment.
d has Null=Exact(0), so the NULL count is known here. The extra group comes from expression_bounds_statistics dropping null_count. Could date_part keep the input NULL count? Otherwise, maybe update this comment.
| stats: &ColumnStatistics, | ||
| input_schema: &Schema, | ||
| ) -> Option<usize> { | ||
| if let Some(&ndv) = stats.distinct_count.get_value() { |
There was a problem hiding this comment.
For an integer key, the range size is an upper bound for the NDV. With bounds [1, 7] and an Inexact NDV of 1000, this returns 1000. Maybe use the smaller of the two?
| match (ndv, limit) { | ||
| match (group_count, limit) { | ||
| (Some(n), Some(l)) => Precision::Inexact(n.min(l)), | ||
| (Some(n), None) => Precision::Inexact(n), |
There was a problem hiding this comment.
When the input row count is unknown, this now returns the range size as the estimate, for example Inexact(2147483647) for bounds [1, i32::MAX]. Before, it returned Absent. The range size is only an upper bound, so I would use it only to cap a known row count.
- Compute radians bounds from the input bounds instead of (-pi, pi). - Keep whether a plain GROUP BY key has NULLs in the aggregate output. - Cap the NDV of an integer key by its range. - Use a range without NDV only to cap a known input row count.
Generated-by: OpenAI Codex
asolimando
left a comment
There was a problem hiding this comment.
Thanks @kumarUjjawal, all my comments have been addressed, LGTM! (cc: @gabotechs)
Which issue does this PR close?
Rationale for this change
In the issue’s TPC-H SF1 example, grouping by shipping year returns seven groups but the planner estimates 6,001,215.
What changes are included in this PR?
Derive year bounds for
Date32inputs, fix theradiansbounds, and propagate inexact bounds through projections and partial aggregates. Partial aggregates keep whether a group key has NULLs. A final aggregate caps the distinct count of an integer key by its range. If the key has no distinct count, the range caps a known input row count.What is the testing strategy for this PR?
The two-partition cases in
parquet_statistics.sltcheck the final estimates, a nullable key without NULLs,radiansbounds, and query results. Rust tests cover bounds propagation, NULL groups, and fallbacks.Are there any user-facing changes?
Tighter group estimates can change join plans.