Repository navigation
perf: preserve date_trunc ordering for timezone-aware timestamps at minute and finer granularities - #27547
Merged
xudong963 merged 2 commits intoNov 20, 2026
Conversation
…inute and finer granularities For minute and finer granularities date_trunc rounds the epoch value down without consulting the timezone, so it cannot reorder timestamps across a timezone transition. Propagate the input ordering in that case, using the predicate that selects the arithmetic kernel.
xudong963
marked this pull request as ready for review
November 18, 2026 04:26
xudong963
added a commit
to massive-com/arrow-datafusion
that referenced
this pull request
Nov 19, 2026
…ware timestamps at minute and finer granularities (#94) ## Which issue does this PR close? - Ports apache#27547 (`perf: preserve date_trunc ordering for timezone-aware timestamps at minute and finer granularities`, part of apache#25464) to `branch-56`. The upstream PR is still open, so this copy may need to be updated if review changes it. - Atlas: with the DF56 upgrade, 32 MV plans lost the time-bucket key from their aggregate clustering mode, because Atlas timestamps carry a timezone (`America/New_York`). The aggregate only streams per `date`, so a full day of `(ticker, bucket)` groups sits in the hash table until the date changes. ## Rationale for this change apache#25409 stopped `date_trunc` from propagating input ordering for every timezone-aware timestamp, because truncating in local time can go backwards across a timezone transition. For `minute` and finer granularities, `date_trunc` never consults the timezone. It rounds the epoch value down, so it cannot reorder values. ## What changes are included in this PR? `git cherry-pick -x 5a30a39` applied cleanly with no adaptations. `DateTruncFunc::output_ordering` now keeps the input ordering for a timezone-aware timestamp when the precision is a known constant and `truncates_in_input_unit` holds, which is the same predicate that selects the arithmetic kernel. `hour` and coarser granularities in a timezone stay unordered. For Atlas, the 22 minute/second aggregate MVs (crypto, forex, indices, stocks sip/otc, futures) should get the time-bucket key back. The 10 day-granularity ones stay as they are; their leading `date` key already provides most of the streaming. ## Are these changes tested? - `cargo test -p datafusion-functions --lib date_trunc`: 16 passed, including the new `output_ordering_with_timezone_depends_on_granularity`. - The new `timestamps.slt` cases pass, along with every other `.slt` file that mentions `date_trunc` (14 files: `group_by`, `group`, `order`, `sort_pushdown`, `range_sorted_time_bin_agg`, `information_schema`, `clickbench`, `datetime/*`, `pg_compat_timestamptz`, `spark/datetime/*`). - `cargo clippy -p datafusion-functions --all-targets --all-features -- -D warnings` and `cargo fmt --check` are clean. ## Are there any user-facing changes? Queries that sort or group timezone-aware timestamps by `date_trunc` at `minute` or finer granularity, over input ordered on the timestamp, no longer add a sort and use clustered aggregation. Results are unchanged.
alamb
approved these changes
Nov 19, 2026
| if range.lower() != range.upper() { | ||
| return None; | ||
| } | ||
| let granularity = range.lower().try_as_str().flatten()?; |
Contributor
There was a problem hiding this comment.
do we really have to convert to a string here?
Contributor
There was a problem hiding this comment.
Why not compare the relevant rust types?
Member
Author
There was a problem hiding this comment.
precision here is the SQL granularity argument, stored as a string-valued ScalarValue, rather than a DatePart. try_as_str() just borrows the existing string; it doesn't allocate or format it. We need its value because 'minute' and 'hour' have the same type but different ordering guarantees. We then parse it into DatePart and reuse the execution kernel's predicate.
15 tasks
zhuqi-lucas
approved these changes
Nov 20, 2026
Member
Author
|
Thanks for your review! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
#25409 stopped
date_truncfrom propagating its input ordering for every timezone-aware timestamp, because truncating in local time can go backwards across a timezone transition (#25353).That is broader than it needs to be. For
minuteand finer granularitiesdate_truncnever consults the timezone: it rounds the epoch value down to a multiple of the granularity (truncates_in_input_unit), and rounding down cannot reorder values. So since 56, sorting or grouping timezone-aware timestamps bydate_trunc('minute', ts)ordate_trunc('second', ts)over input ordered ontsadds a sort, or keeps every group in the hash table until the input ends, where 55 streamed. Time-bucketed aggregation over sorted, timezone-aware data is the common shape for this.#25464 asks to restore the safe timezone-aware cases and notes that particular granularities can be evaluated separately. #25635 covers one axis (UTC and fixed offsets, every granularity). This PR covers the other (every timezone,
minuteand finer). They are independent; both change the samematcharm, so whichever lands second needs a one-line merge.What changes are included in this PR?
DateTruncFunc::output_orderingnow propagates the input ordering for a timezone-aware timestamp when the precision is a known constant andtruncates_in_input_unitholds for it.It uses the predicate that selects the arithmetic kernel, so ordering is claimed exactly when that kernel runs. If a granularity is ever moved to the local-time path (for example to make
minuteexact for historical sub-minute offsets, which the kernel ignores today), the ordering claim goes away with it.Hour and coarser granularities in a timezone, and precisions whose value is not known, stay unordered.
What is the testing strategy for this PR?
timestamps.sltextends the existingAmerica/Goose_Bayregression fromORDER BY date_trunc(...)can return incorrectly ordered results for named timezones #25353. Across the same transition,date_trunc('minute', ..)returns correctly ordered rows without aSortExec, andGROUP BYruns withgroup_clustering_mode=Full. The existinghourcase keeps its sort. The two newEXPLAINcases fail without the change.As a check outside the test suite, I ran the kernel over every UTC-offset transition between 1800 and 2037 of all 597 zones in the tz database, with a few hundred instants around each one.
minuteand finer never reversed.hourreversed in 43 zones anddayin 21.Benchmarks
20M rows, one per second, in a
Timestamp(ns, "America/New_York")column, one Parquet file declaredWITH ORDER (ts ASC).datafusion-clirelease build on an Apple M4 Pro (12 cores), median of 5 runs, wall time and peak RSS of the whole process.mainSELECT date_trunc('minute', ts) AS d FROM t ORDER BY d(EXPLAIN ANALYZE)GROUP BYminute,ORDER BY m LIMIT 1GROUP BYminute (333k groups)GROUP BYsecond (20M groups)GROUP BYminute,target_partitions = 1GROUP BYsecond,target_partitions = 1GROUP BYhour (plan unchanged)The full
GROUP BYqueries are the part to look at. Memory no longer grows with the number of groups, and on one partition the clustered aggregation is as fast or faster. With 12 partitions it is slower in wall time, because the order-preserving plan drains the sorted file ranges one after another and so does not scale with cores the way the hash aggregation does. This is the effect described in #25635 (comment).This is not a new trade-off: it is the plan
mainalready picks for the same queries over a timezone-less timestamp column, with the same numbers (0.24 s, 126 MiB for minute and 0.52 s, 151 MiB for second). The PR makes the timezone-aware case consistent with it. Whether the planner should prefer the clustered plan when it costs parallelism is a separate question, and not one this PR changes.Benchmark setup
Are there any user-facing changes?
Queries that sort or group timezone-aware timestamps by
date_truncatminuteor finer granularity, over input ordered on the timestamp, no longer add a sort and aggregate with clustered aggregation. Results are unchanged. No API changes.