Skip to content

perf: preserve date_trunc ordering for timezone-aware timestamps at minute and finer granularities - #27547

Merged
xudong963 merged 2 commits into
apache:mainfrom
xudong963:xudong963/date-trunc-fine-granularity-ordering
Nov 20, 2026
Merged

xudong963 merged 2 commits into
apache:mainfrom
xudong963:xudong963/date-trunc-fine-granularity-ordering

Conversation

@xudong963

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

#25409 stopped date_trunc from propagating its input ordering for every timezone-aware timestamp, because truncating in local time can go backwards across a timezone transition (#25353).

That is broader than it needs to be. For minute and finer granularities date_trunc never consults the timezone: it rounds the epoch value down to a multiple of the granularity (truncates_in_input_unit), and rounding down cannot reorder values. So since 56, sorting or grouping timezone-aware timestamps by date_trunc('minute', ts) or date_trunc('second', ts) over input ordered on ts adds a sort, or keeps every group in the hash table until the input ends, where 55 streamed. Time-bucketed aggregation over sorted, timezone-aware data is the common shape for this.

#25464 asks to restore the safe timezone-aware cases and notes that particular granularities can be evaluated separately. #25635 covers one axis (UTC and fixed offsets, every granularity). This PR covers the other (every timezone, minute and finer). They are independent; both change the same match arm, so whichever lands second needs a one-line merge.

What changes are included in this PR?

DateTruncFunc::output_ordering now propagates the input ordering for a timezone-aware timestamp when the precision is a known constant and truncates_in_input_unit holds for it.

It uses the predicate that selects the arithmetic kernel, so ordering is claimed exactly when that kernel runs. If a granularity is ever moved to the local-time path (for example to make minute exact for historical sub-minute offsets, which the kernel ignores today), the ordering claim goes away with it.

Hour and coarser granularities in a timezone, and precisions whose value is not known, stay unordered.

What is the testing strategy for this PR?

  • timestamps.slt extends the existing America/Goose_Bay regression from ORDER BY date_trunc(...) can return incorrectly ordered results for named timezones #25353. Across the same transition, date_trunc('minute', ..) returns correctly ordered rows without a SortExec, and GROUP BY runs with group_clustering_mode=Full. The existing hour case keeps its sort. The two new EXPLAIN cases fail without the change.
  • A unit test covers what is impractical to reach from SQL: each granularity for each time unit, the literal's string type and case, and a constant precision whose value is unknown.

As a check outside the test suite, I ran the kernel over every UTC-offset transition between 1800 and 2037 of all 597 zones in the tz database, with a few hundred instants around each one. minute and finer never reversed. hour reversed in 43 zones and day in 21.

Benchmarks

20M rows, one per second, in a Timestamp(ns, "America/New_York") column, one Parquet file declared WITH ORDER (ts ASC). datafusion-cli release build on an Apple M4 Pro (12 cores), median of 5 runs, wall time and peak RSS of the whole process.

Query main This PR
SELECT date_trunc('minute', ts) AS d FROM t ORDER BY d (EXPLAIN ANALYZE) 0.35 s, 333 MiB 0.24 s, 106 MiB
GROUP BY minute, ORDER BY m LIMIT 1 0.06 s, 144 MiB 0.08 s, 119 MiB
GROUP BY minute (333k groups) 0.06 s, 142 MiB 0.24 s, 131 MiB
GROUP BY second (20M groups) 0.17 s, 1172 MiB 0.51 s, 151 MiB
GROUP BY minute, target_partitions = 1 0.24 s, 66 MiB 0.23 s, 36 MiB
GROUP BY second, target_partitions = 1 0.80 s, 1454 MiB 0.46 s, 37 MiB
GROUP BY hour (plan unchanged) 0.37 s, 120 MiB 0.36 s, 119 MiB

The full GROUP BY queries are the part to look at. Memory no longer grows with the number of groups, and on one partition the clustered aggregation is as fast or faster. With 12 partitions it is slower in wall time, because the order-preserving plan drains the sorted file ranges one after another and so does not scale with cores the way the hash aggregation does. This is the effect described in #25635 (comment).

This is not a new trade-off: it is the plan main already picks for the same queries over a timezone-less timestamp column, with the same numbers (0.24 s, 126 MiB for minute and 0.52 s, 151 MiB for second). The PR makes the timezone-aware case consistent with it. Whether the planner should prefer the clustered plan when it costs parallelism is a separate question, and not one this PR changes.

Benchmark setup
COPY (
  SELECT arrow_cast(
           arrow_cast(arrow_cast(value * 1000000000, 'Timestamp(Nanosecond, None)'),
                      'Timestamp(Nanosecond, Some("UTC"))'),
           'Timestamp(Nanosecond, Some("America/New_York"))') AS ts
  FROM generate_series(0, 19999999) ORDER BY value
) TO 'ny.parquet';

CREATE EXTERNAL TABLE t STORED AS PARQUET LOCATION 'ny.parquet' WITH ORDER (ts ASC);

EXPLAIN ANALYZE SELECT date_trunc('minute', ts) AS d FROM t ORDER BY d;
SELECT date_trunc('minute', ts) AS m, count(*) FROM t GROUP BY m ORDER BY m LIMIT 1;
SELECT count(*), max(c) FROM (SELECT date_trunc('minute', ts) AS m, count(*) AS c FROM t GROUP BY m);
SELECT count(*), max(c) FROM (SELECT date_trunc('second', ts) AS s, count(*) AS c FROM t GROUP BY s);
SELECT count(*), max(c) FROM (SELECT date_trunc('hour', ts) AS h, count(*) AS c FROM t GROUP BY h);

Are there any user-facing changes?

Queries that sort or group timezone-aware timestamps by date_trunc at minute or finer granularity, over input ordered on the timestamp, no longer add a sort and aggregate with clustered aggregation. Results are unchanged. No API changes.

…inute and finer granularities

For minute and finer granularities date_trunc rounds the epoch value down
without consulting the timezone, so it cannot reorder timestamps across a
timezone transition. Propagate the input ordering in that case, using the
predicate that selects the arithmetic kernel.
@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Nov 18, 2026
@xudong963
xudong963 marked this pull request as ready for review November 18, 2026 04:26
@xudong963 xudong963 added the regression Something that used to work no longer does label Nov 18, 2026
xudong963 added a commit to massive-com/arrow-datafusion that referenced this pull request Nov 19, 2026
…ware timestamps at minute and finer granularities (#94)

## Which issue does this PR close?

- Ports apache#27547 (`perf: preserve date_trunc ordering for
timezone-aware timestamps at minute and finer granularities`, part of
apache#25464) to `branch-56`. The upstream PR is still open,
so this copy may need to be updated if review changes it.
- Atlas: with the DF56 upgrade, 32 MV plans lost the time-bucket key
from their aggregate clustering mode, because Atlas timestamps carry a
timezone (`America/New_York`). The aggregate only streams per `date`, so
a full day of `(ticker, bucket)` groups sits in the hash table until the
date changes.

## Rationale for this change

apache#25409 stopped `date_trunc` from propagating input
ordering for every timezone-aware timestamp, because truncating in local
time can go backwards across a timezone transition. For `minute` and
finer granularities, `date_trunc` never consults the timezone. It rounds
the epoch value down, so it cannot reorder values.

## What changes are included in this PR?

`git cherry-pick -x 5a30a39` applied cleanly with no adaptations.
`DateTruncFunc::output_ordering` now keeps the input ordering for a
timezone-aware timestamp when the precision is a known constant and
`truncates_in_input_unit` holds, which is the same predicate that
selects the arithmetic kernel. `hour` and coarser granularities in a
timezone stay unordered.

For Atlas, the 22 minute/second aggregate MVs (crypto, forex, indices,
stocks sip/otc, futures) should get the time-bucket key back. The 10
day-granularity ones stay as they are; their leading `date` key already
provides most of the streaming.

## Are these changes tested?

- `cargo test -p datafusion-functions --lib date_trunc`: 16 passed,
including the new
`output_ordering_with_timezone_depends_on_granularity`.
- The new `timestamps.slt` cases pass, along with every other `.slt`
file that mentions `date_trunc` (14 files: `group_by`, `group`, `order`,
`sort_pushdown`, `range_sorted_time_bin_agg`, `information_schema`,
`clickbench`, `datetime/*`, `pg_compat_timestamptz`,
`spark/datetime/*`).
- `cargo clippy -p datafusion-functions --all-targets --all-features --
-D warnings` and `cargo fmt --check` are clean.

## Are there any user-facing changes?

Queries that sort or group timezone-aware timestamps by `date_trunc` at
`minute` or finer granularity, over input ordered on the timestamp, no
longer add a sort and use clustered aggregation. Results are unchanged.

@gabotechs gabotechs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Looks good!

if range.lower() != range.upper() {
return None;
}
let granularity = range.lower().try_as_str().flatten()?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we really have to convert to a string here?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not compare the relevant rust types?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

precision here is the SQL granularity argument, stored as a string-valued ScalarValue, rather than a DatePart. try_as_str() just borrows the existing string; it doesn't allocate or format it. We need its value because 'minute' and 'hour' have the same type but different ordering guarantees. We then parse it into DatePart and reuse the execution kernel's predicate.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5d8a45d avoids the to_lowercase()

@xudong963

Copy link
Copy Markdown
Member Author

Thanks for your review!

@xudong963
xudong963 added this pull request to the merge queue Nov 20, 2026
Merged via the queue into apache:main with commit 477c7b9 Nov 20, 2026
42 checks passed
@xudong963
xudong963 deleted the xudong963/date-trunc-fine-granularity-ordering branch November 20, 2026 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation regression Something that used to work no longer does sqllogictest SQL Logic Tests (.slt) v57.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants